- Notifications
You must be signed in to change notification settings - Fork311
Fix xUnit Theory serialization warnings#3186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
| <Target Name="RunFunctionalTestsWindows" Condition="'$(IsEnabledWindows)' == 'true'"> | ||
| <PropertyGroup> | ||
| <TestCommand> | ||
| $(DotnetPath)dotnet test "@(FunctionalTestsProj)" |
paulmedynskiFeb 28, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Formatted for readability, and then squished back into a single line since<Exec> treats each line in the Command as a separate command!
| --filter "category!=non$(TargetGroup)tests&category!=failing&category!=nonwindowstests" | ||
| --logger:"trx;LogFilePrefix=Functional-Windows$(TargetGroup)-$(TestSet)" | ||
| </TestCommand> | ||
| <TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Modern dotnet tooling supports String.ReplaceLineEndings(), but we're using the old .NET Framework msbuild.exe tooling to build, so we can't have the nice toys :(
| [MemberData( | ||
| nameof(CEKEncryptionReversalData) | ||
| #if NETFRAMEWORK | ||
| // .NET Framework puts system enums in something called the Global |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I've left a comment once in each file where we're using DisableDiscoveryEnumeration, to make it clear why we're using that flag.
| @@ -505,33 +519,25 @@ public override IEnumerable<Object[]> GetData(MethodInfo testMethod) | |||
| } | |||
| } | |||
| publicclass CEKEncryptionReversalParameters : DataAttribute | |||
| publicstatic IEnumerable<object[]> CEKEncryptionReversalData() | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Since xUnit only supports DisableDiscoveryEnumeration on MemberData attributes, I have converted all of the data sources to be compatible with MemberData.
You will see similar changes throughout the tests, where data source classes become methods.
| [CEKEncryptionReversalParameters] | ||
| [MemberData( | ||
| nameof(CEKEncryptionReversalData) | ||
| #if NETFRAMEWORK |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
.NET doesn't have a GAC, so these tests will enumerate correctly for those targets. We only need to disable for .NET Framework targets.
| @@ -4,6 +4,7 @@ | |||
| using System; | |||
| using System.Runtime.InteropServices; | |||
| using System.Threading; | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I don't think this is necessary.
| #endif | ||
| )] | ||
| [MemberData( | ||
| nameof(ConstructorTextData) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This adds Text to the mix, which seems to have been missing from the InlineData.
| @@ -565,7 +584,20 @@ private void ExecuteAndCheckForError(SqlCommand sqlCmd, bool expectError) | |||
| try | |||
| { | |||
| sqlCmd.ExecuteNonQuery(); | |||
| Assert.Fail("We should have gotten an error but passed instead."); | |||
| StringBuilder builder = new( | |||
paulmedynskiFeb 28, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
With broken (now disabled) enumeration, xUnit doesn't include the test data in any failure output, so we need to do it ourselves.
This test fails for me on main as well, so I wanted to see what data set was causing the problem. Turns out to be the Date data set (line 1450).
| @@ -1,6 +1,13 @@ | |||
| { | |||
| // Typical Windows SQL Server on the local host via TCP. | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
These comments are breaking our CI "Update config.json" task because PowerShell's ConvertFrom-Json fails But as of PowerShell v6, ConvertFrom-Json handles comments:
Are we really using ancient PowerShell 5.1 from 2016? I will have to dig into this further.
- Suppressed xUnit discovery data enumeration warnings in Function and Manual test projects.- Fixed xUnit test data enumeration warnings specific to .NET Framework (enums in the GAC).- Added targets to run Windows and Unix tests.- Cleaned up test run command lines for improved maintenance and runtime logging.
a04396b toae09fccCompare- Added installation of PowerShell Core (pwsh) as a dotnet global tool so we can use the ADO task PowerShell@2 in pwsh mode on all agents/images.
- Added installation of PowerShell Core (pwsh) as a dotnet global tool so we can use the ADO task PowerShell@2 in pwsh mode on all agents/images.
- Removed attempt to install PowerShell Core. Adding it to the 1ES images instead.
- Trying pwsh (PowerShell Core 7.x) again for config file update job.
- Updating pipeline to use my new Win11 1ES image with pwsh.
codecovbot commentedMar 5, 2025 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@## main #3186 +/- ##==========================================- Coverage 72.79% 72.52% -0.27%========================================== Files 285 289 +4 Lines 59156 59543 +387 ==========================================+ Hits 43060 43183 +123- Misses 16096 16360 +264
Flags with carried forward coverage won't be shown.Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Reverting pipeline to use the newly rebuilt ADO-CI-Win11 image.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Pull Request Overview
This PR addresses xUnit Theory serialization warnings by replacing InlineData and ClassData attributes with MemberData attributes that disable discovery enumeration on .NET Framework. It also refactors several test data providers across multiple test files and updates pipeline configuration steps to use pwsh consistently.
- Migrate test data attributes from InlineData/ClassData to MemberData with DisableDiscoveryEnumeration for improved xUnit behavior.
- Refactor test data providers into static methods returning IEnumerable<object[]>.
- Update pipeline YAML steps to use "pwsh" instead of "powershell" for configuration consistency.
Reviewed Changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlParameterTest.cs | Refactored xUnit test attributes for Constructor3, Constructor6, and Constructor7 tests using MemberData. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ConversionTests.cs | Replaced ClassData with MemberData in multiple tests to suppress discovery warnings and enhanced error logging details. |
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlCommandSetTest.cs | Introduced MemberData for CommandType tests with appropriate disable flags on .NET Framework. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ConnectionTestWithSSLCert/CertificateTestWithTdsServer.cs | Updated MemberData usage to disable discovery enumeration for connection tests on different platforms. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/JsonTest/JsonBulkCopyTest.cs | Converted JsonBulkCopyTestData from a class to a static method and updated test attributes accordingly. |
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/AlwaysEncryptedTests/SqlColumnEncryptionCertificateStoreProviderShould.cs | Replaced custom data attribute with MemberData for CEK encryption reversal and valid certificate paths tests. |
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/AmbientTransactionFailureTest.cs | Updated MemberData for exception test data to disable discovery enumeration. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ApiShould.cs | Refactored multiple tests to use MemberData for connection string providers with disable discovery flags. |
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlClientFactoryTest.cs | Updated FactoryMethodTest to use MemberData with disable discovery enumeration for improved stability. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/TestFixtures/DatabaseHelper.cs | Consolidated AE connection string providers into a static DataHelpers class with static test data methods. |
| src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/DataReaderTest/DataReaderStreamsTest.cs | Updated MemberData attributes for various DataReader stream tests with disable discovery flags on .NET Framework. |
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlDataRecordTest.cs | Refactored test data providers for UDT and XXX value tests from ClassData to MemberData. |
| eng/pipelines/common/templates/steps/update-config-file-step.yml | Replaced "powershell:" with "pwsh:" in the YAML steps for updating the config file. |
| src/Microsoft.Data.SqlClient/tests/FunctionalTests/SqlMetaDataTest.cs | Refactored several tests to use MemberData with DisableDiscoveryEnumeration, and added new static test data providers for constructor tests. |
Comments suppressed due to low confidence (2)
src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/JsonTest/JsonBulkCopyTest.cs:27
- The parameter 'enableStraming' appears to be a typo. It is recommended to rename it to 'enableStreaming' for clarity and consistency.
public void TestJsonBulkCopy(CommandBehavior cb, bool enableStraming, int jsonArrayElements, int rows)eng/pipelines/common/templates/steps/update-config-file-step.yml:129
- [nitpick] The task key has been updated from 'powershell:' to 'pwsh:'. Please ensure that all related pipeline configuration files consistently use 'pwsh' as the identifier for PowerShell steps.
- pwsh: || // xUnit can't consistently serialize the data for this test, so we | ||
| // disable enumeration of the test data to avoid warnings on the | ||
| // console. | ||
| DisableDiscoveryEnumeration = true)] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Would a custom serializer help in this case? Not to say we should do it in this PR, just curious if you experimented with it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Yes looked into that, but didn't want to figure out serialization for all of the affected tests as part of this PR. The changes here don't affect how the tests run, and they don't change how they're visualized in IDEs. They just suppress the warnings on the console. I've opened an ADO task to track fixing the serialization itself.
f3ae72a intomainUh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
xUnit makes 2 passes over unit tests:
If the enumeration won't produce the same data in both steps, xUnit emits a warning. We have many test cases whose Theory data doesn't enumerate identically in both passes, for a variety of reasons. These tests will still run correctly, but they won't appear in test runner UIs like Visual Studio. Instead, xUnit collapses them into a single test for these UIs.
The warnings are clutter in the console (where I like to run tests), so xUnit provides a mechanism to disable enumeration completely in the discovery phase. I have updated the offending tests to use the MemberData DisableDiscoveryEnumeration flag. This doesn't affect how the tests run, and it doesn't change how UIs display them (collapsed). All it does is suppress the console warnings.
Further reading here:https://xunit.net/faq/theory-data-stability-in-vs
I also added 4 new targets to
build.projto make running the tests easier. The existing test running targets function as before.Commit notes: