Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[wip] Kylel/mmlu subset results#79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
kyleclo wants to merge3 commits intomain
base:main
Choose a base branch
Loading
fromkylel/mmlu-subset-results
Open
Changes from1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
NextNext commit
logic that gets task subcategories
  • Loading branch information
@kyleclo
kyleclo committedMay 20, 2025
commit1756a756dd4988f12923d26fecd42f00111e43bb
25 changes: 21 additions & 4 deletionssrc/cookbook/cli/eval.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -451,11 +451,28 @@ def get_results(
skip_on_fail=skip_on_fail,
)

# if a task starts with *, it means it is a named group and we need to expand it
tasks = [e for t in tasks for e in (ALL_NAMED_GROUPS.get(t.lstrip("*"), [t]) if t.startswith("*") else [t])]
# First expand any named groups
expanded_tasks = []
for task in tasks:
if task.startswith("*"):
# Handle named groups
expanded_tasks.extend(ALL_NAMED_GROUPS.get(task.lstrip("*"), [task]))
else:
# Handle task patterns
expanded_tasks.append(task)

# Then create patterns for matching
task_patterns = []
for task in expanded_tasks:
# If the task contains a colon, it's a pattern that should match all subcategories
if ":" in task:
# Convert the pattern into a regex that matches all subcategories
pattern = task.replace(":", ".*:")
task_patterns.append(re.compile(pattern))
else:
# For regular tasks, use the display tasks mapping or the task itself
task_patterns.extend([re.compile(t_) for t_ in ALL_DISPLAY_TASKS.get(task, [task])])

# after that, we check for task patterns
task_patterns = [re.compile(t_) for task in tasks for t_ in ALL_DISPLAY_TASKS.get(task, [task])]
results = (all_averages + all_metrics).keep_cols(*task_patterns)

if len(models) > 0:
Expand Down

[8]ページ先頭

©2009-2025 Movatter.jp