Summary

Add auto-evaluation capability to the playground, allowing users to run evaluations on inference outputs directly from the playground interface.This feature is behind a feature flag and disabled by default.

Feature Flag

To enable playground evaluations, set the environment variable:

VITE_TENSORZERO_FF_PLAYGROUND_EVALS=1

Backend Changes

• Update tensorzero-node to supportdatapointIds for evaluating specific datapoints
• AddinternalDynamicVariantConfig parameter for evaluating edited (non-builtin) variants
• AddrunNativeEvaluationStreaming wrapper in native_client.server.ts

API

• Add/api/playground/evaluate streaming endpoint using NDJSON format
• Configure cache modes: inference useson (read from cache), evaluation also uses cache for determinism

UI Components

• AddEvaluationCombobox component for selecting evaluations with evaluator names annotation
• AddMetricBadge component for displaying evaluation results with threshold-based red coloring
• AddAnimatedEllipsis component for loading states
• RefactorCombobox to supportgetItemAnnotation prop for showing annotations in both dropdown and input

Playground Integration

• AddusePlaygroundEvaluation hook encapsulating evaluation streaming logic
• Integrate evaluation selector into playground UI (behind feature flag)
• Display evaluation metrics below each inference output
• Support both builtin and edited variants
• Auto-run evaluation when inference completes and evaluation is selected
• Reset evaluation when function changes

MetricBadge Threshold Coloring

• Red background/text when value fails threshold (optimize="max" and value < cutoff, or optimize="min" and value > cutoff)
• For booleans: optimize="max" wants true, optimize="min" wants false
• ReusesisCutoffFailed from MetricValue for consistency

Styling

• Updatemuted-foreground color to be more neutral (remove blue tint)

Test plan

• SetVITE_TENSORZERO_FF_PLAYGROUND_EVALS=1 and verify evaluation selector appears
• Select an evaluation in playground and verify metrics appear below inference
• Test with edited variants (should still work)
• Test clear button (X) on evaluation selector
• Verify evaluation re-runs when variant config changes
• Verify evaluation resets when function changes
• Test threshold coloring: values below cutoff (for max) or above cutoff (for min) show red