Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Factor some evaluation functionality into helpers#5163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
amishler merged 58 commits intomainfromalan/evals-batch-variants
Dec 16, 2025

Conversation

@amishler
Copy link
Member

@amishleramishler commentedDec 12, 2025
edited by ellipsis-devbot
Loading

  • Moves core evaluation functionality into a helper functionprocess_batch() that accepts batches of datapoints and batches of variants, with accompanying structs and enums. (Previously only batching over datapoints was available.)
  • process_batch() is called by the existingrun_evaluation_core_streaming() with a single variant, and it will also be called by the top-k variant selection module, where we need to run evals on batches of variants. So the purpose is to avoid having to duplicate functionality intopk.rs in order to handle batches of variants.

Important

Refactor evaluation logic by introducingprocess_batch() to handle batch processing of datapoints and variants, improving modularity and reducing code duplication.

  • Behavior:
    • Introducesprocess_batch() inlib.rs to handle batches of datapoints and variants.
    • process_batch() is used inrun_evaluation_core_streaming() for single variant processing and will be used for top-k variant selection.
  • Infrastructure:
    • AddsProcessBatchParams,DatapointVariantResult,DatapointVariantError, andBatchItemResult structs to support batch processing.
    • Implementscollect_batch_result() to convertJoinSet results intoBatchItemResult.
  • Misc:
    • Removes redundant task spawning logic fromrun_evaluation_core_streaming() and replaces it withprocess_batch() call.

This description was created byEllipsis forde4a614. You cancustomize this summary. It will automatically update as commits are pushed.

chatgpt-codex-connector[bot] reacted with thumbs up emojiellipsis-dev[bot] reacted with rocket emoji
@amishleramishler marked this pull request as draftDecember 12, 2025 22:42
@virajmehtavirajmehta removed their assignmentDec 15, 2025
@amishleramishler changed the titleEnable batch processing of variants in evalsFactor some evaluation functionality into helperDec 15, 2025
@amishleramishler changed the titleFactor some evaluation functionality into helperFactor some evaluation functionality into helpersDec 15, 2025
@amishler
Copy link
MemberAuthor

@codex review

@chatgpt-codex-connector

Codex Review: Didn't find any major issues. Another round soon, please!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Aaron1011
Aaron1011 previously approved these changesDec 16, 2025
@amishleramishler added this pull request to themerge queueDec 16, 2025
Merged via the queue intomain with commitaf0d0e4Dec 16, 2025
53 checks passed
@amishleramishler deleted the alan/evals-batch-variants branchDecember 16, 2025 17:57
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@chatgpt-codex-connectorchatgpt-codex-connector[bot]chatgpt-codex-connector[bot] left review comments

@Aaron1011Aaron1011Aaron1011 approved these changes

@GabrielBianconiGabrielBianconiAwaiting requested review from GabrielBianconi

Assignees

@Aaron1011Aaron1011

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@amishler@Aaron1011@virajmehta

[8]ページ先頭

©2009-2025 Movatter.jp