I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

Rerunning as there were 5 sdk tests failures

This round seems to fail due to restore failures:

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

C:\h\w\B5080A10\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: The "ResolvePackageAssets" task failed unexpectedly.

@marcpopMSFT are you seeing in stability across the sdk repo? This pr seems to be in a whack a mole issue hunt with azure. Here is a new one:

python.exe -c "import future" || C:\Users\runner.azdo-env\Scripts\python.exe -m pip install future==0.17.1
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'future'

@jeffschwMSFT we have been having a lot of stability issues mainly related to ADO feeds throttling us and overlapping test directories. They've increased the limits for us, we've fixed the test directory at the beginning of this week, and added a retry on restore such that at least earlier this week, our reliability was looking very good. Today though is having a lot of issues so I'm not sure if there is system problem today or we just got lucky earlier in the week.

@dsplaisted is on build duty. Given the high number of failures in this run (we usually see a smaller number), might be a system problem or an issue in the payload.

@dsplaisted will you get a chance to take a look? The current set of failures look different than the last set, which is true of the first set.

@jeffschwMSFT This looks like it might be a break in the runtime that is being inserted. I'm seeing a failure down in the System.IO code:

C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: The "ResolvePackageAssets" task failed unexpectedly. [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at System.IO.Strategies.SyncWindowsFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at System.IO.Strategies.BufferedFileStreamStrategy.Flush(Boolean flushToDisk) [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing) [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at System.IO.Stream.Close() [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at System.IO.BinaryWriter.Dispose(Boolean disposing) [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at System.IO.BinaryWriter.Dispose() [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheWriter.WriteToCacheFile() in //src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 704 [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheReader.CreateReaderFromDisk(ResolvePackageAssets task, Byte[] settingsHash) in //src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 520 [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheReader..ctor(ResolvePackageAssets task) in //src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 468 [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ReadItemGroups() in //src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 300 [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ExecuteCore() in //src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 292 [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.NET.Build.Tasks.TaskBase.Execute() in //src/Tasks/Common/TaskBase.cs:line 38 [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]
C:\h\w\A6D10962\p\d\sdk\6.0.100-ci\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(241,5): error MSB4018: at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [C:\h\w\A6D10962\t\dotnetSdkTests\figp5j5f.tn0\PackMultiTargetedLibrary\PackMultiTargetedLibrary\PackMultiTargetedLibrary.csproj]

Is this the failure you were seeing?

Only on this most recent retry. I don't have the previous logs, but I do not think it was due to this. Adding@jeffhandley .

@jeffhandley can you take a look at these IO failures?
https://helix.dot.net/api/2019-06-17/jobs/58c00372-d80f-4849-8d3d-af0a41789bf1/workitems/Microsoft.NET.Build.Tests.dll.5/console
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-16810-merge-597bfe36397d4bb388/Microsoft.NET.Build.Tests.dll.4/console.92c43082.log?sv=2019-07-07&se=2021-05-02T15%3A12%3A31Z&sr=c&sp=rl&sig=EYSlR%2FEfp9ClPkVhihg6Tx77vIQx6zi8t0Ah8UlFqoE%3D
https://helix.dot.net/api/2019-06-17/jobs/597bfe36-397d-4bb3-88e1-5cd89b2526a3/workitems/Microsoft.NET.Build.Tests.dll.3/console

cc@danmoseley

@adamsitnik@carlossanlop@jozkee -- These errors look to be caused by the recent changes toFileStream. Please take a look right away to determine what we need to do here.

Here's the trimmed up call stack:

    Microsoft.NET.Build.Tests.GivenThatWeWantToBuildAnAppWithoutTransitiveProjectRefs.It_builds_the_project_successfully_when_RAR_finds_all_references [FAIL]      The "ResolvePackageAssets" task failed unexpectedly. [C:\h\w\A224091D\t\dotnetSdkTests\btnehx1n.s1g\It_builds_the---6B3FC006\3\3.csproj]      System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.         at System.IO.Strategies.SyncWindowsFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count)         at System.IO.Strategies.BufferedFileStreamStrategy.Flush(Boolean flushToDisk)         at System.IO.Strategies.BufferedFileStreamStrategy.Dispose(Boolean disposing)         at System.IO.Strategies.BufferedFileStreamStrategy.DisposeInternal(Boolean disposing)         at System.IO.FileStream.Dispose(Boolean disposing)         at System.IO.Stream.Close()         at System.IO.BinaryWriter.Dispose(Boolean disposing)         at System.IO.BinaryWriter.Dispose()         at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheWriter.WriteToCacheFile() in /_/src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 704         at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheReader.CreateReaderFromDisk(ResolvePackageAssets task, Byte[] settingsHash) in /_/src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 520         at Microsoft.NET.Build.Tasks.ResolvePackageAssets.CacheReader..ctor(ResolvePackageAssets task) in /_/src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 468         at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ReadItemGroups() in /_/src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 300         at Microsoft.NET.Build.Tasks.ResolvePackageAssets.ExecuteCore() in /_/src/Tasks/Microsoft.NET.Build.Tasks/ResolvePackageAssets.cs:line 292         at Microsoft.NET.Build.Tasks.TaskBase.Execute() in /_/src/Tasks/Common/TaskBase.cs:line 38         at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()         at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask)

@carlossanlop@jozkee repro:

git clone https://github.com/dotnet/sdk reprocd reprogit checkout darc-main-0fd96bf7-148d-4cf6-97b0-d3934c67609e.\build.cmd -c Debug.\.dotnet\dotnet.exe test -c Debug src\Tests\Microsoft.NET.Build.Tests\Microsoft.NET.Build.Tests.csproj --filter Microsoft.NET.Build.Tests.GivenThatWeWantToBuildASelfContainedApp.It_succeeds_when_RuntimeIdentifier_and_PlatformTarget_mismatch_but_PT_is_AnyCPU

@jeffschwMSFT@marcpopMSFT we have aruntime PR out to revert the compat flag that enabled the new FileStream rewrite by default. This will unblock dotnet/sdk and will allow us to investigate the root cause more carefully.

Thanks@carlossanlop. Please prioritize getting that in and flowing as this PR is needed to unblockeddotnet/installer#10185 and I need a new build of installer with runtime changes to insert into VS previews as well.

CC@joeloff

Update: We havea PR out to fix the actual issue. If the CI of that PR finishes successfully, we'll get that merged. Otherwise, we'll mergemy PR reverting the compat flag change.

The PR with the fix was merged. Is dotnet-maestro going to update this PR automatically with the commit that contains the fix, or do we need to do something manually?

Hopefully it is updated by maestro once a new runtime build is ready

Thanks@carlossanlop. The current failures are due to a missing browswer wasm package.

/private/tmp/helix/working/ABAC0950/w/B36A0A16/e/testExecutionDirectory/BuildMinimal_---F77D8391/blazorwasm-minimal.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Runtime.browser-wasm with version (= 6.0.0-preview.4.21213.1)

@marek-safar@steveisok can y'all take a look at what is missing?

@steveisok that's the renaming issue, right?

Yes, we are working on fixing

@steveisok the run just finished and we are now seeing the following: Unable to find package Microsoft.NETCore.App.Runtime.browser-wasm with version (= 6.0.0-preview.4.21213.3)

@jeffschwMSFTdotnet/installer#10206 is part of the fix.

Installer change is merged now.

@marcpopMSFT would unblocking this PR need a new stage0 SDK?

@pranavkm The changes made to OverrideAndCreateBundledNETCoreAppPackageVersion.cs allow them to avoid needing a stage 0 update. It basically adds the changes that would be from the new installer build to avoid the sdk vs. installer deadlock.

@dotnet/dnceng The Ubuntu build has been queued for an hour now without starting. Is there an issue in the lab with resources or too many builds at once?

@marcpopMSFT yes, the queue is its max capacity limit. I just scaled the queue over this limit to speed things up.

FYI, the capacity limit will be increased tomorrow to prevent this.

What's the best way for folks like@marcpopMSFT to look at the queue depth themselves?

There is a Helix API call that you can do to get the queueDepth

You can callhttps://helix.dot.net/api/info/queues/buildpool.ubuntu.1604.amd64.open?includeQueueDepth=true&api-version=2019-06-17 but it looks like the information isn't correct.@MattGal@alexperovich do you know if we stopped supporting this?

There is a Helix API call that you can do to get the queueDepth

You can callhttps://helix.dot.net/api/info/queues/buildpool.ubuntu.1604.amd64.open?includeQueueDepth=true&api-version=2019-06-17 but it looks like the information isn't correct.@MattGal@alexperovich do you know if we stopped supporting this?

new to me, checking it out.

Um. I think it's an artifact from when@alexperovich rewrote the API? The old one has it thenewer one doesn't really

            int queueDepth = 0;            if (includeQueueDepth)            {                // TODO: get this            }

It's intentional on both API versions:

#pragma warning disable 1998        private async Task<long> GetQueueDepth(string queueId)#pragma warning restore 1998        {            // we are unable to get this with the current sdks            // replace this when we switch to Helix.Utility.Azure stuff            Trace.WriteLine(_serviceBusClientFactory.GetType().Name);

... looks like updating SDKs took away functionality, sorry users!

Remind me again what we use for our grafana alerting?

Our monitoring system definitely has the queue depth information@marcpopMSFT was looking for. I'll share it with him but it isn't recommended approach for all users

Our monitoring system definitely has the queue depth information@marcpopMSFT was looking for. I'll share it with him but it isn't recommended approach for all users

it probably just uses the old SDK where that API was still available.

it probably just uses the old SDK where that API was still available.

That makes sense. Marc could just use that too yes?

it probably just uses the old SDK where that API was still available.

That makes sense. Marc could just use that too yes?

Correct, we just don't usually point users at Grafana as@ulisesh said.

@markwilkie PR improvement suggestion to show the queue in the new PR experience. I can certainly use the dashboard or api but I'll likely forget before the next time it gets backed up on a high visibility change.

@garath /@missymessa - could one of you make sure we're tracking the suggestion to add queue depth to the PR experience please?

@marcpopMSFT - it'll be good to track this, but given that the build won't even have started, it'll be tough to show results of something that hasn't started yet. :) Sounds like you could make an API call as well....

@garath /@ilyas1974 - what's the latest thinking around opening up grafana for more folks to see. My thinking is to allow folks to self-help queue depth. thoughts?

No issues on my side opening up Grafana to more of our customers to view queue status.

Is the scenario you're thinking about something like - my build didn't start properly (i.e. couldn't get a machine before timeout) and in the failure we also show what the current queue depth is?

woot! Thanks everyone

My scenario was: "All the builds are started except for Ubuntu. Why is it still queued and how long should I expect to wait?" That's why I pinged dnceng in the first place as this PR was blocking an installer PR which was blocking a VS PR....lots waiting for it to get fixed.

Is opening up grafana something you can follow up on by any chance@ilyas1974 ? Probably check w/@Chrisboh etc to see what (if anything) we need to think about. If you're full up, I can do it as well, just let me know.

Happy to help to drive this.

@marcpopMSFT, for your specific scenario, hooking into Grafana may not provide the answers you are looking for.

• With Grafana, we only have visibility into the non-hosted pools.
• While Grafana has the potential to answer the second part of your question - how long should I expect to wait - it will not address the first part - Why is it still queued.
  • What it sounds like you are asking for is a centralized location that lists on-going issues.

could one of you make sure we're tracking the suggestion to add queue depth to the PR experience please?

I've opened dotnet/core-eng#12899 in the epic to track.