- Notifications
You must be signed in to change notification settings - Fork714
fix django-parser's defect that it cannot handle multiline test cases.#314
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
john-b-yang commentedFeb 28, 2025
@BoxiYu Thanks so much for the fix. For changes like these, I'm going to run on all Django instances to see if the gold result comes back correctly. |
BoxiYu commentedMay 24, 2025
Hi John@john-b-yang , we did research on the parser and the test suite in SWE-Bench. I am happy to share that the work is accepted by ACL 2025 main. We found that (1) there are annotation errors given by the parsers' defects on the existing bench, also affecting other repositories in addition to Django, (2) many of the instances in SWE-Bench Verified still have an insufficient test suite that may. [UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench (https://github.com/CUHK-Shenzhen-SE/UTBoost)]. Maybe the bench needs an update? If there is anything I could do, I am glad to help with. |
Reference Issues/PRs
#275
What does this implement/fix? Explain your changes.
It adopts a more robust log parsing algorithm for the Django log.
Any other comments?
We should build more robust parsers, I think this is important to the robsut evaluation of swebench
🧡 Thanks for contributing!