Walkthrough

Adds four new composite GitHub Actions to fetch CPython and RustPython, integrate selected CPython files into the RustPython tree, and run per-file smoke tests with timeouts; plus a new workflow that orchestrates these actions across an OS/Python matrix. Actions export branch/sha/paths/files for downstream steps.

Changes

Sequence Diagram(s)

sequenceDiagram  autonumber  actor Runner as GitHub Runner (matrix job)  participant Workflow as .github/workflows/Check_Tests.yml  participant FetchRP as CI-5974-Fetch-RustPython  participant FetchCP as CI-5974-Fetch-CPython  participant Integrate as CI-5974-Integrate-CPython  participant Smoke as CI-5974-Test-RustPython-Integration  participant Cargo as cargo  Runner->>Workflow: start job  Workflow->>FetchRP: checkout RustPython (outputs: branch, sha, rustpython-lib-path)  FetchRP-->>Workflow: outputs  Workflow->>FetchCP: checkout CPython (outputs: branch, sha, files)  FetchCP-->>Workflow: outputs  Workflow->>Integrate: copy selected CPython files into RustPython workspace  Integrate-->>Workflow: integration done  Workflow->>Smoke: run per-file smoke tests (files, timeouts, env)  Smoke->>Cargo: invoke per-file runs (cargo run / -m unittest)  Cargo-->>Smoke: return (ok / timeout / fail)  Smoke-->>Workflow: TEST_STEP_SUMMARY, per-file outcomes  Workflow-->>Runner: job completes (post step)

sequenceDiagram  autonumber  participant Smoke as Smoke Test Action  participant Timer as run_with_timeout  participant Cargo as cargo run  participant Py as python -m unittest  Smoke->>Smoke: iterate INPUT_FILES  alt file contains unittest.main()    Smoke->>Timer: run_with_timeout(cargo run -- <file>)    Timer->>Cargo: execute    Cargo-->>Timer: exit code  else file is unittest module    Smoke->>Timer: run_with_timeout(cargo run -- -m unittest <test_name>)    Timer->>Py: execute    Py-->>Timer: exit code  end  Timer-->>Smoke: outcome (success / timeout / failure)  Smoke->>Smoke: record outcome, append to TEST_STEP_SUMMARY

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested reviewers

• youknowone

Poem

I hop through branches, files in tow,
I copy tests where carrots grow.
Timers tick and cargo sings,
Fixes, failures, tiny things.
CI lights blink — a rabbit's glow. 🥕🐇

Pre-merge checks and finishing touches

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment@coderabbitai help to get the list of available commands and usage tips.}

Demo of Outputs:

Most results are not copy/drop-in compatible (yet?)

Log output has a summary for each test: pass/failed/timed-out

Some files do just work and could be directly used without issue (all are trivial though)

@reactive-firewall tysm for the PR!
Do you mind uploading here a JSON / yaml of the results? something machine readable that I can parse. I want to go over the results.

What is in your opinion the next step? Having an automated PR system that updates those files automatically?

@ShaharNaveh Could you please be a primary reviewer of this patch?

@reactive-firewall tysm for the PR! Do you mind uploading here a JSON / yaml of the results? something machine readable that I can parse. I want to go over the results.

@ShaharNaveh currently I don't have it generating a machine readable result in json or yaml, it only puts those values in the "${GITHUB_ENV}" file at the moment. The initial idea here was to provide a human readable result to what the enormous logs report (similar to whats_left).

This PR is more about the automation tooling in CI/CD than the actual test results (ATM); visibility.

However, as I agree that we totally need to have the machine readable content too, I'll have to hash out another helper tool to generate next.

What is in your opinion the next step? Having an automated PR system that updates those files automatically?

I actually have been conflicted on this very point for a while now, and opened this PR to get feedback regarding how the project best benefits. My interest was the plumbing of the CI/CD initially.

My two concerns about next steps are:

• how to keep the integration seamless (project convention compliance included; e.g., nomenclature is hard, and new things are always suspect, etc.) considering this ismy first contribution to the main branch.
  AND
• staying focused on the initial topic of GHI[RFC] The way to update Python standard library #5974 (e.g. automation around updating the python standard library) and its foundational requirements. (e.g., not pulling in the requirements and constraints from similar components like whats_next.py unnecessarily)

That said, IMHO, the next steps are indeed to start work on automations based on the periodic results, including things like opening PRs to update trivial changes upstream in the reference python implementation (e.g. c-python), and github issues/file diagnostic comments (an idea previously mentioned by@arihant2math a few times in previous issues as a wishlist item); of-course, that is after the short-term of adding/integrating a helper tool to provideaccessible machine-readable results as you mentioned.

Regarding the automation of PRs, the tinkering of how to separate out what to include is still a TODO (especially considering it should record providence details from the upstream commits like sha and commit-message across the reference git-tree) and will require some care around providing write access permissions for PRs/content (for secure implementation)

Regarding the immediate next step: I'll get to work on the machine-readable part sometime this week, and update this PR with those results when I have something worth looking at on that front.

🙇 thank you for taking the time to read this, and I hope this helps.

@coderabbitai,

Context for AI: we have been discussing adding a helper tool to help generate machine-readable JSON or YAML outputs from the new github actions.

Goal for AI: please suggest a plan and implementation for me to use as a starting point.

#!/bin/bash# Read the test integration action to understand current output formatcat .github/actions/CI-5974-Test-RustPython-Integration/action.yaml

@reactive-firewall,

I'll suggest a comprehensive plan and implementation for generating machine-readable outputs from your GitHub Actions workflow.

Plan Overview

1.Structured Output Schema

Create a JSON/YAML schema that captures:

• Test run metadata (timestamp, Python version, OS, commit SHA)
• Per-file test results (outcomes, timing, test method used)
• Summary statistics (pass/fail/timeout counts)
• Provenance information (CPython SHA, RustPython SHA)

2.Helper Script Implementation

A Python script (scripts/generate_test_results.py) that:

• Parses test outcomes from GITHUB_ENV or command-line arguments
• Aggregates results into structured format
• Outputs both JSON and YAML formats
• Can be called at the end of the test step

3.Integration Points

• Add a new step in.github/actions/CI-5974-Test-RustPython-Integration/action.yaml after smoke testing
• Upload results as workflow artifacts
• Optionally post summary as PR comment

Implementation

Schema Definition

# scripts/generate_test_results.py#!/usr/bin/env python3"""Generate machine-readable test results from GitHub Actions workflow.This script parses test outcomes and generates structured JSON/YAML outputfor CPython integration test results."""importargparseimportjsonimportosimportsysfromdataclassesimportdataclass,asdictfromdatetimeimportdatetime,timezonefrompathlibimportPathfromtypingimportDict,List,Optionalimportyaml@dataclassclassTestResult:"""Represents the outcome of a single test file."""file_path:strraw_copy_outcome:str# compatible, incompatible, failingfix_copy_outcome:str# fixed, unfixed, reverted, skippedtest_method:str# direct, unittest, skippedduration_seconds:Optional[float]=None@dataclassclassTestRunMetadata:"""Metadata about the test run environment."""timestamp:strpython_version:stros:strrustpython_sha:Optional[str]=Nonecpython_sha:Optional[str]=Nonecpython_branch:Optional[str]=Nonemax_test_time:int=30cargo_args:Optional[str]=None@dataclassclassTestRunSummary:"""Summary statistics for the test run."""total_tests:intcompatible_count:intincompatible_count:intfixed_count:intunfixed_count:intreverted_count:intskipped_count:intinconclusive_count:int@dataclassclassTestRunResults:"""Complete test run results with metadata and individual results."""metadata:TestRunMetadatasummary:TestRunSummaryresults:List[TestResult]classTestResultsGenerator:"""Generates machine-readable test results from various sources."""def__init__(self):self.results:List[TestResult]= []self.metadata:Optional[TestRunMetadata]=Nonedefparse_github_env(self,env_file_path:str)->None:"""Parse test outcomes from GITHUB_ENV file."""env_vars= {}ifnotPath(env_file_path).exists():print(f"Warning: GITHUB_ENV file not found at{env_file_path}",file=sys.stderr)returnwithopen(env_file_path,'r')asf:forlineinf:line=line.strip()if'='inlineandnotline.startswith('#'):key,value=line.split('=',1)env_vars[key]=valueself._parse_env_vars(env_vars)defparse_environment(self)->None:"""Parse test outcomes from current environment variables."""self._parse_env_vars(dict(os.environ))def_parse_env_vars(self,env_vars:Dict[str,str])->None:"""Parse environment variables to extract test results."""# Parse metadataself.metadata=TestRunMetadata(timestamp=datetime.now(timezone.utc).isoformat(),python_version=env_vars.get('PYTHON_VERSION','3.13'),os=env_vars.get('OS',env_vars.get('RUNNER_OS','unknown')),rustpython_sha=env_vars.get('RUSTPYTHON_SHA'),cpython_sha=env_vars.get('CPYTHON_SHA'),cpython_branch=env_vars.get('CPYTHON_BRANCH'),max_test_time=int(env_vars.get('SUBSHELL_TIMEOUT','30')),cargo_args=env_vars.get('CARGO_ARGS')        )# Parse test resultstest_files= {}forkey,valueinenv_vars.items():ifkey.startswith('RAW_COPY_')andkey.endswith('_OUTCOME'):# Extract filename from key: RAW_COPY_<filename>_OUTCOMEfilename=key[9:-8]# Remove prefix and suffixiffilenamenotintest_files:test_files[filename]= {}test_files[filename]['raw_copy']=valuetest_files[filename]['file_path']=self._reconstruct_file_path(filename)elifkey.startswith('FIX_COPY_')andkey.endswith('_OUTCOME'):filename=key[9:-8]iffilenamenotintest_files:test_files[filename]= {}test_files[filename]['fix_copy']=valuetest_files[filename]['file_path']=self._reconstruct_file_path(filename)# Convert to TestResult objectsforfilename,dataintest_files.items():result=TestResult(file_path=data.get('file_path',filename),raw_copy_outcome=data.get('raw_copy','unknown'),fix_copy_outcome=data.get('fix_copy','unknown'),test_method=self._infer_test_method(data)            )self.results.append(result)def_reconstruct_file_path(self,filename:str)->str:"""Reconstruct the original file path from the env var key."""# Handle cases like "test_foo.py" or "test_foo"ifnotfilename.endswith('.py'):filename=filename+'.py'returnf"Lib/test/{filename}"def_infer_test_method(self,data:Dict[str,str])->str:"""Infer the test method used based on outcomes."""raw_outcome=data.get('raw_copy','')ifraw_outcome=='inconclusive':return'skipped'# Could be enhanced by parsing additional metadatareturn'direct'# Default assumptiondefcalculate_summary(self)->TestRunSummary:"""Calculate summary statistics from test results."""summary=TestRunSummary(total_tests=len(self.results),compatible_count=sum(1forrinself.resultsifr.raw_copy_outcome=='compatible'),incompatible_count=sum(1forrinself.resultsifr.raw_copy_outcome=='incompatible'),fixed_count=sum(1forrinself.resultsifr.fix_copy_outcome=='fixed'),unfixed_count=sum(1forrinself.resultsifr.fix_copy_outcome=='unfixed'),reverted_count=sum(1forrinself.resultsifr.fix_copy_outcome=='reverted'),skipped_count=sum(1forrinself.resultsifr.fix_copy_outcome=='skipped'),inconclusive_count=sum(1forrinself.resultsifr.raw_copy_outcome=='inconclusive')        )returnsummarydefgenerate_output(self,output_format:str='json')->str:"""Generate formatted output in JSON or YAML."""ifnotself.metadata:raiseValueError("Metadata not initialized. Call parse_github_env() or parse_environment() first.")summary=self.calculate_summary()test_run=TestRunResults(metadata=self.metadata,summary=summary,results=self.results        )# Convert dataclasses to dictionariesoutput_dict=asdict(test_run)ifoutput_format.lower()=='json':returnjson.dumps(output_dict,indent=2)elifoutput_format.lower()=='yaml':returnyaml.dump(output_dict,default_flow_style=False,sort_keys=False)else:raiseValueError(f"Unsupported output format:{output_format}")defwrite_output(self,output_path:str,output_format:str='json')->None:"""Write formatted output to a file."""content=self.generate_output(output_format)withopen(output_path,'w')asf:f.write(content)print(f"Results written to{output_path}")defmain():"""Main entry point for the script."""parser=argparse.ArgumentParser(description='Generate machine-readable test results from GitHub Actions workflow'    )parser.add_argument('--env-file',help='Path to GITHUB_ENV file to parse (default: read from environment)',default=None    )parser.add_argument('--output','-o',help='Output file path (default: stdout)',default=None    )parser.add_argument('--format','-f',choices=['json','yaml'],default='json',help='Output format (default: json)'    )parser.add_argument('--summary-only',action='store_true',help='Output only summary statistics'    )args=parser.parse_args()generator=TestResultsGenerator()# Parse input sourceifargs.env_file:generator.parse_github_env(args.env_file)else:generator.parse_environment()# Generate outputtry:ifargs.summary_only:summary=generator.calculate_summary()output_dict=asdict(summary)ifargs.format=='json':content=json.dumps(output_dict,indent=2)else:content=yaml.dump(output_dict,default_flow_style=False)else:content=generator.generate_output(args.format)# Write to file or stdoutifargs.output:withopen(args.output,'w')asf:f.write(content)print(f"Results written to{args.output}",file=sys.stderr)else:print(content)exceptExceptionase:print(f"Error generating output:{e}",file=sys.stderr)sys.exit(1)if__name__=='__main__':main()

Integration into GitHub Actions

Add this step to.github/actions/CI-5974-Test-RustPython-Integration/action.yaml after the "Try Smoke Testing" step:

    -name:"Generate Machine-Readable Results"id:generate_resultsif:${{ !cancelled() }}shell:bashrun:|        # Install PyYAML if not present        python3 -m pip install --quiet PyYAML 2>/dev/null ||true        # Generate JSON output        python3 ./scripts/generate_test_results.py \          --format json \          --output "test-results-${{ runner.os }}-${PYTHON_VERSION}.json"        # Generate YAML output        python3 ./scripts/generate_test_results.py \          --format yaml \          --output "test-results-${{ runner.os }}-${PYTHON_VERSION}.yaml"        # Display summary to console        echo "::group::Test Results Summary"        python3 ./scripts/generate_test_results.py --format json --summary-only        echo "::endgroup::"

Add Artifact Upload Step to Workflow

In.github/workflows/Check_Tests.yml, add after the smoke test step:

    -name:"Upload Test Results"if:${{ !cancelled() }}uses:actions/upload-artifact@v4with:name:test-results-${{ matrix.os }}-${{ matrix.python-version }}path:|          rustpython/test-results-*.json          rustpython/test-results-*.yaml          rustpython/CPython-Summary-Artifact-*.mdretention-days:30

Example Output

JSON format:

{"metadata": {"timestamp":"2025-01-15T10:30:45.123456+00:00","python_version":"3.13","os":"ubuntu-latest","rustpython_sha":"abc123...","cpython_sha":"def456...","cpython_branch":"v3.13.1","max_test_time":30,"cargo_args":"--release"  },"summary": {"total_tests":50,"compatible_count":5,"incompatible_count":30,"fixed_count":10,"unfixed_count":15,"reverted_count":5,"skipped_count":15,"inconclusive_count":0  },"results": [    {"file_path":"Lib/test/test_grammar.py","raw_copy_outcome":"compatible","fix_copy_outcome":"skipped","test_method":"direct","duration_seconds":null    }  ]}

Benefits

1. Machine-readable: Easy to parse with standard JSON/YAML libraries
2. Comprehensive: Captures all relevant metadata and results
3. Extensible: Schema can be extended with additional fields as needed
4. Minimal changes: Integrates with existing workflow without disrupting current functionality
5. Dual format: Supports both JSON (for processing) and YAML (for readability)
6. Artifact storage: Results preserved for later analysis

Next Steps

1. Create thescripts/generate_test_results.py file with the above implementation
2. Add PyYAML to any requirements files if not already present
3. Integrate the generation step into the workflow
4. Test with a few sample runs to validate output format
5. Consider adding a follow-up step to post results summary as a PR comment

Would you like me to help with any specific aspect of the implementation or suggest additional features for the output format?

@reactive-firewall tysm for the PR! Do you mind uploading here a JSON / yaml of the results? something machine readable that I can parse. I want to go over the results.

@ShaharNaveh currently I don't have it generating a machine readable result in json or yaml, it only puts those values in the "${GITHUB_ENV}" file at the moment. The initial idea here was to provide a human readable result to what the enormous logs report (similar to whats_left).

Generally, you'd find it easier to first normalize data into a machine readable format, and generate human readable output from that. I think that the main issue is that you wrote all the "business logic" in bash.

I'm fine with small bash scripts here and there but this is not the case. it be easier to maintain (and review) if it was written in Rust/Python.

This PR is more about the automation tooling in CI/CD than the actual test results (ATM); visibility.

However, as I agree that we totally need to have the machine readable content too, I'll have to hash out another helper tool to generate next.

What is in your opinion the next step? Having an automated PR system that updates those files automatically?

I actually have been conflicted on this very point for a while now, and opened this PR to get feedback regarding how the project best benefits. My interest was the plumbing of the CI/CD initially.

My two concerns about next steps are:

• how to keep the integration seamless (project convention compliance included; e.g., nomenclature is hard, and new things are always suspect, etc.) considering this ismy first contribution to the main branch.

This is not easy. And ty for working on this! (especially for a first PR)

AND

• staying focused on the initial topic of GHI[RFC] The way to update Python standard library #5974 (e.g. automation around updating the python standard library) and its foundational requirements. (e.g., not pulling in the requirements and constraints from similar components like whats_next.py unnecessarily)

That said, IMHO, the next steps are indeed to start work on automations based on the periodic results, including things like opening PRs to update trivial changes upstream in the reference python implementation (e.g. c-python), and github issues/file diagnostic comments (an idea previously mentioned by@arihant2math a few times in previous issues as a wishlist item); of-course, that is after the short-term of adding/integrating a helper tool to provideaccessible machine-readable results as you mentioned.

Regarding the automation of PRs, the tinkering of how to separate out what to include is still a TODO (especially considering it should record providence details from the upstream commits like sha and commit-message across the reference git-tree) and will require some care around providing write access permissions for PRs/content (for secure implementation)

Regarding the immediate next step: I'll get to work on the machine-readable part sometime this week, and update this PR with those results when I have something worth looking at on that front.

🙇 thank you for taking the time to read this, and I hope this helps.

I'll explain why I think a machine readable format is necessary as the next step; I want to see if there are "low hanging fruits" that we can update immediately, and maybe start to narrow down the scope of the PR to handle only the most simple cases and expand it later.

@reactive-firewall I suggest you take a look athttps://github.com/termux/termux-packages
they have a very robust auto updating mechanism for updating packages from many different upstreams, maybe we can get inspiration from some of the things they are doing.

What I have in mind (at least for now) is a process where this runs once a week and open PRs for the low hanging fruits, and opens issue(s) for failing updates, similar to whatbevy does. (Don't implement any of this ATM)

imo the next steps are:

• rewriting the logic in Python/Rust
• Outputting a machine readable format + human readable (upload the machine readable format withhttps://github.com/actions/upload-artifact)