- Notifications
You must be signed in to change notification settings - Fork5.2k
[cdac] cdac-build-tool#100650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
[cdac] cdac-build-tool#100650
Uh oh!
There was an error while loading.Please reload this page.
Conversation
We don't parse them yet, however
Co-authored-by: Elinor Fung <elfung@microsoft.com>
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
src/coreclr/tools/cdac-build-tool/JsonConverter/GlobalModelJsonConverter.cs OutdatedShow resolvedHide resolved
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
…nConverter.csCo-authored-by: Elinor Fung <elfung@microsoft.com>
lambdageek commentedApr 18, 2024
All the windows build lanes are failing with something like: which looks like we're running the in-tree |
elinor-fung commentedApr 19, 2024
I think Lines 15 to 18 in3cea1ee
Maybe we should set |
shushanhf commentedApr 19, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
After this PR, there is an error during coreclr camke-config step by command ####linux-debian-AMD64 and loongarch64 are both failed !!!./build-runtime.sh -debug -loongarch64 -nopgooptimize -skipmanaged-- Lookingfor process_vm_readv-- Lookingfor process_vm_readv - foundCMake Error at debug/runtimeinfo/CMakeLists.txt:57 (message): No cdac-build-toolset or does not exist-- Configuring incomplete, errors occurred!See also"/home/qiao/work_qiao/runtime/artifacts/obj/coreclr/linux.loongarch64.Debug/CMakeFiles/CMakeOutput.log".See also"/home/qiao/work_qiao/runtime/artifacts/obj/coreclr/linux.loongarch64.Debug/CMakeFiles/CMakeError.log". |
| set(GENERATED_CDAC_DESCRIPTOR_DIR"${CMAKE_CURRENT_BINARY_DIR}/cdac") | ||
| set(CONTRACT_DESCRIPTOR_OUTPUT"${GENERATED_CDAC_DESCRIPTOR_DIR}/contract-descriptor.c") | ||
| if("${CDAC_BUILD_TOOL_BINARY_PATH}"STREQUAL""ORNOTEXISTS"${CDAC_BUILD_TOOL_BINARY_PATH}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
nit:
- "${CDAC_BUILD_TOOL_BINARY_PATH}" STREQUAL ""+ NOT CDAC_BUILD_TOOL_BINARY_PATH
it checks both "not defined" and "evaluates to empty string".
| add_custom_command( | ||
| OUTPUT"${CONTRACT_DESCRIPTOR_OUTPUT}" | ||
| VERBATIM | ||
| COMMAND dotnet${CDAC_BUILD_TOOL_BINARY_PATH} compose -o"${CONTRACT_DESCRIPTOR_OUTPUT}" -c"${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
src/coreclr/build-runtime.shdirect invocation skips requiring dotnet/msbuild, which is necessary for new platform port work. This PR assumes dotnet/msbuild are always available. We should handle that case.Secondly, can use published cdac-build-tool instead of assembly path so instead of requiring
dotnetin PATH, it uses executable (AOT'd, singilefilehost, apphost -based doesn't matter)? If that's not possible (e.g. crossbuild case), then please passDOTNET_HOST_PATH(set by SDK to the executing dotnet(1)) via cmakeargs as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Can we just make embedding the contract descriptor optional if someone is doing a build without msbuild?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
If making it optional doesn't affect debugging, that's good. Second part is:
- <_CoreClrBuildArg Include="-cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH=$(RuntimeBinDir)cdac-build-tool\cdac-build-tool.dll"" />+ <_CoreClrBuildArg Include="-cmakeargs "-DCDAC_BUILD_TOOL_BINARY_PATH='$(RuntimeBinDir)cdac-build-tool/cdac-build-tool.dll'" -cmakeargs "-DCLR_DOTNET_HOST_PATH='$(DOTNET_HOST_PATH)'"" />
- COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>+ COMMAND "${CLR_DOTNET_HOST_PATH}" "${CDAC_BUILD_TOOL_BINARY_PATH}" compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>
which will resolve topath/to/runtime/.dotnet/dotnet.
lambdageekApr 19, 2024 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
If making it optional doesn't affect debugging, that's good
in the future, it will make using something like dotnet-sos impossible
I don't understand what scenarios don't have a .NET sdk available. any new platform bringup will start with cross compiling from an existing platform with a .NET sdk and a cross compiler toolchain, right? that case should be handled by the cmake+msbuild infrastructure
I added#101297
- COMMAND dotnet ${CDAC_BUILD_TOOL_BINARY_PATH} compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>+ COMMAND "${CLR_DOTNET_HOST_PATH}" "${CDAC_BUILD_TOOL_BINARY_PATH}" compose -o "${CONTRACT_DESCRIPTOR_OUTPUT}" -c "${CONTRACT_FILE}" $<TARGET_OBJECTS:cdac_data_descriptor>
I had something like this before and it broke source build because it was pointing to the repo root.dotnet/dotnet, not toartifacts/sb/.dotnet/dotnet. AFAICT it's fine to just call "dotnet" from the path. eng/commons/build.sh always adds the right SDK dir to the path - it's done by InitializeDotNetCli, and it is source-build aware
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Update I'm educating myself over indotnet/installer#19534 (comment)
I think previously I used something other thanDOTNET_HOST_PATH
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Updated#101297 to use DOTNET_HOST_PATH if it's set
# cDAC Build Tool## SummaryThe purpose of `cdac-build-tool` is to generate a `.c` file that contains a JSON cDAC contract descriptor.It works by processing one or more object files containing data descriptors and zero or more textfiles that specify contracts.## Running```console% cdac-build-tool compose [-v] -o contractdescriptor.c -c contracts.txt datadescriptor.o```## .NET runtime build integration`cdac-build-tool` is meant to run as a CMake custom command.It consumes a target platform object file and emits a C sourcefile that contains a JSON contract descriptor. The C sourceis the included in the normal build and link steps to create the runtime.The contract descriptor source file depends on `contract-aux-data.c` which is a source file that containsthe definitions of the "indirect pointer data" that is referenced by the data descriptor. This is typically the addresses of important global variables in the runtime.Constants and build flags are embedded directly in the JSON payload.Multiple data descriptor source files may be specified (for example if they are produced by different components of the runtime, or by different source languages). The final JSON payload will be a composition of all the data descriptors.Multiple contracts text files may be specified. This may be useful if some contracts are conditionally included (for example if they are platform-specific). The final JSON payload will be a composition of all the contracts files.```mermaidflowchart TB headers("runtime headers") data_header("datadescriptor.h") data_src("datadescriptor.c") compile_data["clang"] data_obj("datadescriptor.o") contracts("contracts.txt") globals("contractpointerdata.c") build[["cdac-build-tool"]] descriptor_src("contractdescriptor.c") vm("runtime sources") compile_runtime["clang"] runtime_lib(["libcoreclr.so"]) headers -.-> data_src headers ~~~ data_header data_header -.-> data_src headers -.-> globals headers -.-> vm data_src --> compile_data --> data_obj --> build contracts ---> build build --> descriptor_src descriptor_src --> compile_runtime data_header -.-> globals ----> compile_runtime vm ----> compile_runtime --> runtime_lib```--- * add implementation note notes* add an emitter* read in the directory header* contract parsing* indirect pointer value support* move sample to tool dir* Take baselines from the docs/design/datacontracts/data dir We don't parse them yet, however* Add README* fix BE Store the magic as a uint64_t so that it will follow the platform endianness. Store endmagic as bytes so that it directly follows the name pool - and fix the endmagic check not to look at the endianness* hook up cdac-build-tool to the coreclr build; export DotNetRuntimeContractDescriptor* cleanup; add contracts.txt* add diagram to README* move implementation notes* better verbose output from ObjectFileScraper* turn off whole program optimizations for data-descriptor.obj On windows /GL creates object files that cdac-build-tool cannot read It's ok to do this because we don't ship data-descriptor.obj as part of the product - it's only used to generate the cDAC descriptor* C++-ify and add real Thread offsets* no C99 designated initializers in C++ until C++20* build data descriptor after core runtime* fix gcc build* simplify ObjectFileScraper just read the whole file into memory* invoke 'dotnet cmake-build-tool.dll' instead of 'dotnet run --project'* clean up macro boilerplate* platform flags* turn off verbose output* can't use constexpr function in coreclr because debugreturn.h defines a `return` macro that expands to something that is not c++11 constexpr* Rename "aux data" to "pointer data"* rename "data-descriptor" to "datadescriptor"* simplify linking* cdac-build-tool don't build dotnet tool; turn on analyzers* rationalize naming; update docs; add some inline comments* renamce cdac.h to cdacoffsets.h* improve output: hex offsets; improved formatting* don't throw in ParseContracts; add line numbers to errors* change input format for contracts to jsonc* add custom JsonConverter instances for the compact json representation* simplify; bug fix - PointerDataCount include placeholder* one more set of feedback changes: simpler json converters* set _RequiresLiveILLink=false for cdac-build-tool.csproj fixes windows builds: error MSB3026: (NETCORE_ENGINEERING_TELEMETRY=Build) Could not copy "D:\a\_work\1\s\artifacts\obj\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll" to "D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll". Beginning retry 1 in 1000ms. The process cannot access the file 'D:\a\_work\1\s\artifacts\bin\ILLink.Tasks\Debug\net9.0\ILLink.Tasks.dll' because it is being used by another process. ---------Co-authored-by: Elinor Fung <elfung@microsoft.com>Co-authored-by: Aaron Robinson <arobins@microsoft.com>
Uh oh!
There was an error while loading.Please reload this page.
Contributes to#99298
cDAC Build Tool
Summary
The purpose of
cdac-build-toolis to generate a.cfile that contains a JSON cDAC contract descriptor.It works by processing one or more object files containing data descriptors and zero or more text
files that specify contracts.
Running
%cdac-build-tool compose [-v] -o contractdescriptor.c -c contracts.txt datadescriptor.o.NET runtime build integration
cdac-build-toolis meant to run as a CMake custom command.It consumes a target platform object file and emits a C source
file that contains a JSON contract descriptor. The C source
is the included in the normal build and link steps to create the runtime.
The contract descriptor source file depends on
contract-aux-data.cwhich is a source file that containsthe definitions of the "indirect pointer data" that is referenced by the data descriptor. This is typically the addresses of important global variables in the runtime.
Constants and build flags are embedded directly in the JSON payload.
Multiple data descriptor source files may be specified (for example if they are produced by different components of the runtime, or by different source languages). The final JSON payload will be a composition of all the data descriptors.
Multiple contracts text files may be specified. This may be useful if some contracts are conditionally included (for example if they are platform-specific). The final JSON payload will be a composition of all the contracts files.
flowchart TB headers("runtime headers") data_header("datadescriptor.h") data_src("datadescriptor.c") compile_data["clang"] data_obj("datadescriptor.o") contracts("contracts.txt") globals("contractpointerdata.c") build[["cdac-build-tool"]] descriptor_src("contractdescriptor.c") vm("runtime sources") compile_runtime["clang"] runtime_lib(["libcoreclr.so"]) headers -.-> data_src headers ~~~ data_header data_header -.-> data_src headers -.-> globals headers -.-> vm data_src --> compile_data --> data_obj --> build contracts ---> build build --> descriptor_src descriptor_src --> compile_runtime data_header -.-> globals ----> compile_runtime vm ----> compile_runtime --> runtime_lib