@edwardneal I could use a wee bit of help figuring out the errors - it looks to me that a table doesn't exist, so we can't delete the entries in it before running the test. Since we're not getting any errors creating the tables, I suspect we're dropping the table somewhere between tests. But I can't seem to repro it locally (although that could be an issue with my setup not being the same as test machine).

I've also kicked the failing tests again to see if maybe it's transient. But since it's consistent across all the enclave tests I suspect it isn't transient.

Sorry for the delay here@benrr101, I've just been able to get to look at this.

The fixture here isPlatformSpecificTestContext. This instantiates aSQLSetupStrategyCertStoreProvider, which calls the baseSQLSetupStrategy -CreateTables, followed bySetupDatabase.

The effect ofCreateTables which we care about is that it populates theTrustedMasterKeyPathsTestTable and returns it; the caller adds this to thedatabaseObjects field.

SetupDatabase iterates over each of these objects, calling theirCreate method. Then, it callsInsertSampleData, which wipes the table and adds a new record to it.

The net effect is that we create a new table, then immediately insert sample data into it. This fails, so the test fixture can't be instantiated and every test which uses the fixture fails. I'm not able to reproduce it either.

I'd like to compare the logs of a few CI runs - could you retrigger this please?

/azp run

@edwardneal Ok, that's mostly what I suspected that it was an issue with the database setup steps causing the fixture to fail to be instantiated. Stuff like that working locally but not on the test machine suggests to me some kind of race condition, but that's just wild speculation. As per your request, I've kicked off another CI run. Let me know if you need more - I appreciate you looking into this :)

@edwardneal Any further thoughts on this one?

@edwardneal thanks again for looking into the failures and dealing with the annoying back-n-forth of trying to resolve the issues. Let's see what this round does.

@paulmedynski,@mdaigle do you have anything to add regarding the net8/net9 test behavior? I'm kinda wondering if we need to make sure that the test projects are defined to target the same frameworks as the projects themselves.

/azp run

CI has timed out again, but I was able to see the cause in the realtime logs a few hours ago. It looks like we're not receiving any response from 10.0.0.4 and 10.0.0.5 prior to running the tests. Is the AE Enclave test environment healthy?

@cheenamalhotra does this have to do with the VM decommissioning?

@edwardneal there are test infrastructure issues we are trying to iron out.

10.0.0.5 is the one that was deallocated due to Azure capacity pressure, yes. It let me start it back up so I've re-run failed jobs for this PR.

Thanks - I can see that most of these tests are passing now (with the exception of Azure Key Vault - I'll try to reproduce that locally.)

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.28%. Comparing base(cc23e04) to head(02470f2).
Report is 8 commits behind head on main.

@@            Coverage Diff             @@##             main    #3204      +/-   ##==========================================- Coverage   67.16%   65.28%   -1.89%==========================================  Files         300      300                Lines       65640    65640              ==========================================- Hits        44087    42851    -1236- Misses      21553    22789    +1236

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report?Share it here.

@paulmedynski - I can't review this on myself, so that means we've only got you,@mdaigle,@samsharma2700, and@cheenamalhotra who can review it.

@paulmedynski I addressed most of the comments you added, some a bit more heavily-handed (heavy-handidly?) than others. I also merged main into it b/c I couldn't build without a fix we added a looong time ago.