Repository files navigation

Code for the report"How far behind are open models?"

You can install required packages using

pip install -r requirements.txt

Results from the paper can be reproduced by running the following notebooks:

• benchmark_analysis.ipynb for the analysis of benchmark performance
• compute_analysis.ipynb for the analysis of training compute trends
• data_exploration.ipynb for the analysis of how many models with open weights and open training code have occurred

Each notebook specifies aresults_dir where results will be saved.Thebenchmark_analysis.ipynb andcompute_analysis.ipynb notebooks each have a set of parameters near the top.These parameters can be modified for sensitivity analysis.See the description of each parameter in the notebook for details.

The raw data used for analysis is in thedata/ folder.All ML Systems - full view.csv is a snapshot of the Epoch AI database of models.benchmarks_with_model_accessibility.csv contains the benchmark data that we use, except for GSM1k, which is ingsm1k_with_model_accessibility.csv.

Note that the plots produced by this code do not match the design of plots in the report, but the underlying data is the same.