- Notifications
You must be signed in to change notification settings - Fork746
Factor some evaluation functionality into helpers#5163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Uh oh!
There was an error while loading.Please reload this page.
Conversation
…erval for efficiency
…sorzero into alan/evals-batch-variants
amishler commentedDec 15, 2025
@codex review |
Codex Review: Didn't find any major issues. Another round soon, please! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Uh oh!
There was an error while loading.Please reload this page.
af0d0e4Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
process_batch()that accepts batches of datapoints and batches of variants, with accompanying structs and enums. (Previously only batching over datapoints was available.)process_batch()is called by the existingrun_evaluation_core_streaming()with a single variant, and it will also be called by the top-k variant selection module, where we need to run evals on batches of variants. So the purpose is to avoid having to duplicate functionality intopk.rsin order to handle batches of variants.Important
Refactor evaluation logic by introducing
process_batch()to handle batch processing of datapoints and variants, improving modularity and reducing code duplication.process_batch()inlib.rsto handle batches of datapoints and variants.process_batch()is used inrun_evaluation_core_streaming()for single variant processing and will be used for top-k variant selection.ProcessBatchParams,DatapointVariantResult,DatapointVariantError, andBatchItemResultstructs to support batch processing.collect_batch_result()to convertJoinSetresults intoBatchItemResult.run_evaluation_core_streaming()and replaces it withprocess_batch()call.This description was created by
forde4a614. You cancustomize this summary. It will automatically update as commits are pushed.