- Notifications
You must be signed in to change notification settings - Fork311
Add partial packet detection and fixup#2714
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
I've added comments to the Packet class as requested. The CI was green apart from some ubuntu legs which timed out, many other ubuntu legs succeeded so I don't see any direct inference on from that. Ready for review@David-Engel@saurabh500@cheenamalhotra |
@Wraith2 We are reviewing this and hope to get faster traction towards EOW. |
cheenamalhotra commentedAug 6, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Pasting test failure for reference: This test should be looked at carefully. It failed on Ubuntu with .NET 6 and 8 , and also hung up on Windows when ran with Managed SNI,link to logs 1link to logs 2. @Wraith2 can you confirm if this is something you're able to repro in Windows with Managed SNI? Please make sure config file is configured to enable Managed SNI on Windows. |
| { | ||
| // Do nothing with callback if closed or broken and error not 0 - callback can occur | ||
| // after connection has been closed. PROBLEM IN NETLIB - DESIGN FLAW. | ||
| return; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Can you add a Debug Assert here and check if this is taking any hit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Test needs to be fixed, before reviewing any further.
Isn't this the set of tests that@David-Engel pointed out in#2608 (comment) ? If so we discussed it at length on the teams call. I don't believe that those tests are reliable. Setup a breakpoint or Debug.WriteLine where an exception is added to the state object and run the test. You should find that an exception isalways added to the state object but that the test will usually succeed. That should not be possible, an exception if added should be thrown. The test is missing failures and if that's the case then the test is unreliable. |
When you work past the terrible code in SNITCPHandle and make the test run for long enough it settles into a steady state where it can't reach the end. There is no indication why yet. those 9 in flight items just don't seem to complete but i don't know why. |
After a few more hours investigation I know what the problem is but I have no clue what change has caused it. In SqlDataReader when an async method is called we use a context object to contain some state and pass that context object to all the async methods that are used to implement the async read machinery. Part of this state is the TaskCompletionSource. What I don't understand is how cancel is supposed to work. I'm unable to run the tests in native sni mode because the native sni can't be initialized (can't find the sni dll). So I can't compare the managed to unmanaged implementations here. I don't believe that I have made any change thatshould affect cancellation. I have verified that there are no partial packets in the state objects when the async tasks get stuck. I don't understand how async cancellation is supposed to work at all. |
88fbb93 to8b57818CompareCan someone with CI access rerun the failed legs? the failures are random or CI resources not being available as far as i can tell. |
Wraith2 commentedAug 10, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
The current failures are interesting. They're in the test that was failing before but they new ones are only detected because i made the test more accurate. The previous version of the test acceptedany exception when it was expecting a cancellation exception. It was passing on netfx with my previous changes because timeout exceptions were being thrown. I judged that accepting a timeout when we were supposed to be testing whether cancellation had occurred was not correct. If we retained the previous version of the test then everything would have passed cleanly. In the current situation since the test completed correctly without hanging the result is equivalent to what we would have experienced in all test runs in the past, all started threads that we expected to be cancelled exited with an exception. [edit] |
@Wraith2 You might be banging your head against an unrelated issue in the driver. IIRC, the test was only introduced to ensure we don't regress "The MARS TDS header contained errors." issue. (The test code came from the repro.) If you isolate your test changes and run them against main code, does it still fail? Yes, the correct exception is probably "Operation cancelled by user." where the exception is being caught. But if it's unrelated to your other changes, I would leave that part of the test as it was and file a new issue with repro code. As it is, it's unclear if and how this behavior is impacting users and I wouldn't hold up your perf improvements for it. |
There was definitely a real problem. The results differed between main and my branch. I've solved that issue now and the current state is that we're seeing a real failure because I've made the test more sensitive. I think it's probably safe to lower the sensitivity of the test again now because the new test that I've added covers the specific scenario in the multiplexer that I had missed and everything else is pass. I'll try that and see how the CI likes it. I think the current state on this branch is that it is as stable as live. We need to have confidence that this set of changes is correct before we can merge it. It's high risk and high complexity code. Even understanding it very deeply it has taken me a week to actively debug a very important behaviour change that I missed. |
Can someone re-run the failed legs? the only failing test is something to do with event counters which I've been no-where near. |
The failing test is EventCounter_ReclaimedConnectionsCounter_Functional. It's doing something with GC specific to net6. It's failing sporadically on net6 managed sni runs but not deterministically. I can't make it fail locally to trace what might be happening. |
Any thoughts? |
David-Engel commentedAug 26, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
I'm not seeing the failures you mentioned in EventCounter_ReclaimedConnectionsCounter_Functional [in the CI results]. I mainly see fairly consistent failures of CancelAsyncConnections on Linux. It seems to pass on Windows managed SNI, so there might be something that is Linux/Ubuntu network specific. Can you run/debug thetest against a local WSL or Docker instance? |
If i click through the failure i get to this pagehttps://sqlclientdrivers.visualstudio.com/public/_build/results?buildId=95784&view=ms.vss-test-web.build-test-results-tab The cancel tests are passing now, those failed in the previous runs but not the current ones. |
I think there is something wrong with the Tests tab. I don't usually reference it. I scroll down the summary tab to see what jobs had failures:
Then drill into the job and the task that failed to see the log:
|
If it's AsyncCancelledConnectionsTest again then there isn't anything further I can do. That test is multithreaded and timing dependent. I've traced the individual packets through the entire call stack. I've run it for 1000 iterations successfully after fixing a reproducible error in it. If someone can isolate a reproducible problem from it then i'll investigate. |
I chatted with@saurabh500 and I just want to add that this is definitely something we all want to see get merged. It'll just take someone finding time (could take a few days dedicated time) to get their head wrapped around the new code and be able to help repro/debug to find the issue. |
I'm happy to make myself available to talk through the code with anyone that needs it. |
@Wraith2 and@David-Engel I was looking at the lifecycle of the snapshots and something that stood out in NetCore vs NetFx is that SqlDataReader for NetCore is storing the cached snapshot with the SqlInternalConnectionTds which is a shared resource among all the SqlDataReader(s) running on a MARS connection. This means that we are saving the reader snapshot on the shared resource, which can be overwritten by any other reader. @Wraith2 have you had a chance to pursue this line of investigation for hanging test? I wonder if the timing is causing the wrong cached snapshot to be provided to a SqlDataReader, causing data corruption and likely causing a hang. |
SqlInternalConnection.cs |
saurabh500 commentedSep 4, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
@Wraith2 I see that you had made the changes in the first place. Can you try another PR where you remove the storage of these contexts and snapshots on SqlInternalConnection and with the multiplexing change, try to see if this solves the problem. Also, I am Happy to be told that my theory is wrong, but I would like to understand how in MARS cases, the shared Cached contexts on InternalConnection is a safe design choice. |
| int remainderLength = partialPacket.CurrentLength - partialPacket.RequiredLength; | ||
| partialPacket.CurrentLength -= remainderLength; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Can thepartialPacket.CurrentLength be changed somewhere else asynchronously, because otherwise this is equivalent of writingpartialPacket.CurrentLength = partialPacket.RequiredLength. Just trying to understand the reasoning here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It can't be changed elsewhere. I wrote it this way because it was logical to me, it documents what I'm doing not just what the result will be.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think this is a valid feedback, can you consider rewriting it topartialPacket.CurrentLength = partialPacket.RequiredLength or do you see any concerns?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It's just a style difference as far as I can see. I feel that my version follows the flow of calculating and using whereas the alternative simply jumps to a resultant fact. It doesn't make enough difference for me to fight about it and with 2:1 votes I've changed it as suggested.
You may also note that I followed up the discussion about making partial packet disposal exception a debug only behaviour in79f9fbc but forgot to mention it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I'm inclined to move forward and see how this performs in a preview release! Looking forward to the next stage 😄
We discussed offline, and our concerns are about things going wrong in production for end users that could require us to patch the driver in future. Because this is an intricate change in the packet processing logic, we're considering enabling this feature by default, but be able to turn it off on demand using an App Context switch. The App Context switch would mean, we will preview this feature for couple of releases, till we have stabilized it and are confident the design is thorough and safe for permanent replacement, after which we can even remove the app context switch to remove old design. @Wraith2 One of our team members will get started on this, and collaborate with you to make that change in this PR, so we can have the App Context switch come in before the next PR you're working on. We're fully supportive of the changes, but also need to ensure we have a fallback plan for customers if anything goes wrong. |
Ok. I've made a start by adding the required appcontext switch, copying the current ProcessSni implementation into the muliplexer file as ProcessSniCompat and updating new tests to be conditional on the appcontext switch. The existing ProcessSni call will check the appcontext switch and call the old one if the compat switch is set. I'm not sure how you need to update the CI legs to account for testing both full sets to make sure we get coverage of both paths. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Packet.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/Packet.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
| /// <summary> | ||
| /// returns a boolean value indicating if the <see cref="DataLength"/> value has been set. | ||
| /// </summary> | ||
| public bool HasDataLength => _dataLength >= 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Consider!= UnknownDataLength. That's the point of having a named constant, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
It's not quite the same thing. The constant is an arbitrary value which is known to produce a false result when this property is evaluated. The property defines ranges, the constant simply represents a single value inside that range that we use to have a specific named meaning. If you really want me to change it then I will but not using it is meaningful to me.
| { | ||
| if (error != 0) | ||
| { | ||
| if ((_parser.State == TdsParserState.Closed) || (_parser.State == TdsParserState.Broken)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Consider_parser.State is TdsParserState.Closed or TdsParserState.Broken
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This falls into the category of maintaining consistent style. The change wouldn't improve functionality and would leave us with files in the solution using different syntax to do the same thing which can cause you not to be able to find them or have to mentally determine that both syntaxes mean the same thing.
If we're going to make this change I'd rather do it in a dedicated style update PR. I don't object to the new syntax I just think we should be consistent.
Any thoughts? |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/TdsParserStateObject.Multiplexer.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
/azp run |
| Azure Pipelines successfully started running 1 pipeline(s). |
I think i've fixed the test build. but the ci isn't running automatically on changes anymore. |
/azp run |
| Azure Pipelines successfully started running 1 pipeline(s). |
Looks like ProcessSniPacketCompat doesn't have visibility to SniNativeWrapper when building for NetFx. |
Should be fixed now. Can you run the CI again please. It would really help development if the project didn't have loads of false errors when working in visual studio. A clean commandline build has lots of these issues in VS which makes it very difficult to see real problems. |
/azp run |
| Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
| Azure Pipelines successfully started running 1 pipeline(s). |
8d5e4f2 intodotnet:mainUh oh!
There was an error while loading.Please reload this page.
Woo! 🚀 |
Split out from#2608 per discussion detailed in#2608 (comment)
Adds packet multiplexer and covering tests.