Hmm I understand the problem, thanks for noticing it. But I probably wouldn't prefer this be the default behavior. Perhaps we can condition this statement on whether Modal eval is being used.

It's not a big difference I guess, but the reason this statement exists is to make sure text not related to testing output is ignored.

@john-b-yang that makes sense, I modified the code to skip the content slicing on Modal only, could you help take another look?

Hi@john-b-yang, just a friendly ping on this PR - please let me know if there's anything I can clarify or improve. Thanks!

Hi@john-b-yang and/or@klieret, would it be possible to get this merged? We're currently having to work off a branch of swe-bench and it's causing us logistics issues at the moment. Thanks a lot for the consideration!