Summary

We want to support the wide range of metrics implemented inscikit-learn inevaluate. While this expands the capabilities ofevaluate it also gives users from thescikit-learn ecosystem access to useful tools inevaluate such as pushing results to the hub or end-to-end evaluate models with theevaluator classes. As a bonus, all metrics will get an interactive widget that can be embedded in various places such as the docs.

Goal

The goal of this integration should be that metrics fromscikit-learn can be loaded from evaluate with the following API:

importevaluatemetric=evaluate.load("sklearn/accuracy")metric.compute(predictions=[0,1,1],references=[1,1,0])

How it can be done

For the integration we could build a script that goes through all metrics of thescikit-learn repository and automatically builds the metric repositories in theevaluate format and pushes them to the Hub. This could be a script that's executed via a GitHub action whenever a change is pushed to main similar to how it's done for the internal modules (seehere).

Besides the function, its arguments and input/output format we can also use the docs to populate thegradio widget on the hub. See theAccuracy module as an example of how the metrics could be displayed.