Repository files navigation

The SWE-bench website for leaderboards and project information.

• SWE-bench Website
  • Table of Contents
  • Overview
  • Getting Started
    • Prerequisites
    • Installation
    • Development
  • Directory Structure
  • How the Website Works
    • Build Process
    • Template System
    • Leaderboard Data Flow
    • CSS Organization
    • JavaScript Usage
  • Customizing the Website
    • Adding a New Page
    • Updating the Leaderboard
    • Changing the Theme
    • Adding New Features
  • Deployment
  • Troubleshooting

This is the codebase for theSWE-bench website, which showcases leaderboards for the SWE-bench benchmark. SWE-bench tests systems' ability to solve GitHub issues automatically.

The site is built using:

• Jinja2 for HTML templating
• Pure CSS (organized in modular files)
• Vanilla JavaScript for interactivity
• Python for the build process

The site is statically generated and can be hosted on GitHub Pages or any other static hosting service.

• Python 3.8 or higher
• uv for Python package management

1. Clone this repository:

   git clone https://github.com/swe-bench/swe-bench.github.io.gitcd swe-bench.github.io

2. Install dependencies (uv will automatically create and manage the virtual environment):

   make install

1. Build the site:

   make build

2. Start a local development server:

   make serve

3. Open your browser and navigate tohttp://localhost:8000

.├── build.py             # Build script that generates the static site├── CNAME                # Domain configuration for GitHub Pages├── css/                 # CSS files organized by functionality│   ├── components.css   # Styles for UI components│   ├── core.css         # Core styling variables and base styles│   ├── layout.css       # Layout-related styles│   ├── main.css         # CSS entry point that imports all stylesheets│   ├── pages.css        # Page-specific styles│   └── utilities.css    # Utility classes├── data/                # Data files used in the site generation│   └── leaderboards.json  # Leaderboard data├── dist/                # Generated static site (created by build.py)├── favicon.ico          # Site favicon├── img/                 # Image assets├── js/                  # JavaScript files│   ├── citation.js      # Citation functionality│   ├── citationFormat.js # Citation format handlers│   ├── mainResults.js   # Main leaderboard functionality│   ├── tableByRepo.js   # Repository filter functionality│   ├── tableByYear.js   # Year filter functionality│   └── viewer.js        # Results viewer functionality├── Makefile             # Automation for common tasks├── pyproject.toml      # Python project configuration and dependencies└── templates/           # Jinja2 templates    ├── base.html        # Base template with common structure    ├── _leaderboard_table.html  # Reusable leaderboard table component    ├── _sidebar.html    # Sidebar component    └── pages/           # Page-specific templates        ├── citations.html        ├── contact.html        ├── index.html        ├── lite.html        ├── multimodal.html        ├── submit.html        └── viewer.html

The website is built using a static site generator implemented inbuild.py. This script:

1. Loads templates from thetemplates directory
2. Loads data fromdata/leaderboards.json
3. Renders each template intemplates/pages/ to a corresponding HTML file indist/
4. Copies static assets (CSS, JS, images, favicon, etc.) to thedist/ directory

The website uses Jinja2 for templating:

• templates/base.html: The main template that defines the site structure
• templates/_sidebar.html: The sidebar component included in the base template
• templates/_leaderboard_table.html: The reusable leaderboard table component
• templates/pages/*.html: Individual page templates that extend the base template

Templates use Jinja2 syntax for:

• Template inheritance ({% extends 'base.html' %})
• Including components ({% include "_sidebar.html" %})
• Block definitions and overriding ({% block content %}{% endblock %})
• Variable rendering ({{ variable }})
• Control structures ({% if condition %}{% endif %})

The leaderboard data follows a specific flow from JSON to rendered HTML:

1. Data Source: All leaderboard data is stored indata/leaderboards.json. This JSON file contains an array of leaderboards under the key"leaderboards", with each leaderboard having a"name" (e.g., "Test", "Lite", "Verified", "Multimodal") and a list of"results" entries.

2. Data Loading: During the build process inbuild.py, the JSON file is loaded and passed to the Jinja2 templates as theleaderboards variable:

   # From build.pywithopen(ROOT/"data/leaderboards.json","r")asf:leaderboards=json.load(f)# Passed to templates during renderinghtml=tpl.render(title="SWE-bench",leaderboards=leaderboards["leaderboards"])

3. Reusable Table Component: The_leaderboard_table.html template is a reusable component that loops through the leaderboards array and renders a table for each one:

   {% for leaderboard in leaderboards %}<divclass="tabcontent"id="leaderboard-{{leaderboard.name}}"><tableclass="table scrollable data-table"><!-- Table headers --><tbody>            {% for item in leaderboard.results if not item.warning %}<tr><!-- Row data from each result item --></tr>            {% endfor %}</tbody></table></div>{% endfor %}

4. Page-Specific Rendering: In page templates likelite.html, the leaderboard data can be rendered in a more focused way by filtering for a specific leaderboard:

   {% for leaderboard in leaderboards %}    {% if leaderboard.name == "Lite" %}<tableclass="table scrollable data-table"><!-- Only renders the "Lite" leaderboard --></table>    {% endif %}{% endfor %}

5. Dynamic Badges and Formatting: The templates add special badges and formatting to entries:

   • Medal emoji (🥇, 🥈, 🥉) for the top 3 entries
   • "New" badge (🆕) for recent submissions
   • "Open source" badge (🤠) whenitem.oss is true
   • "Verified" checkmark (✅) whenitem.verified is true
   • Percentage formatting with 2 decimal places:{{ "%.2f"|format(item.resolved|float) }}

6. JavaScript Enhancements: After the HTML is rendered, JavaScript files likemainResults.js,tableByRepo.js, andtableByYear.js enhance the tables with sorting, filtering, and tab switching functionality.

This modular approach allows for easy updates to leaderboard data - simply modify the JSON file, and the changes will propagate throughout the site during the next build.

CSS is organized into modular files and imported throughmain.css:

• core.css: Base styles, variables, and resets
• layout.css: Grid and layout components
• components.css: UI component styles
• pages.css: Page-specific styles
• utilities.css: Utility classes for common styling needs

This organization makes it easy to find and update specific styles.

JavaScript is used for interactive features:

• mainResults.js: Main leaderboard functionality including filtering and sorting
• tableByRepo.js &tableByYear.js: Additional table filtering
• citation.js &citationFormat.js: Citation formatting and copying
• viewer.js: Results viewer page functionality

To add a new page to the website:

1. Create a new HTML file intemplates/pages/, e.g.,templates/pages/new-page.html
2. Start with the basic template structure:

   {% extends 'base.html' %}{% block title %}New Page Title{% endblock %}{% block content %}<sectionclass="container"><divclass="content-section"><h2>Your New Page</h2><p>Content goes here...</p></div></section>{% endblock %}

3. Add any page-specific CSS tocss/pages.css
4. Add any page-specific JavaScript to thejs/ directory and include it in your template:

   {% block js_files %}<scriptsrc="js/your-script.js"></script>{% endblock %}

5. Update the sidebar navigation intemplates/_sidebar.html to include your new page
6. Rebuild the site withmake build

To update the leaderboard data:

1. Editdata/leaderboards.json with the new entries
2. The JSON structure follows this format:

   {"leaderboards": [    {"name":"Test","results": [        {"name":"Model Name","folder":"folder_name","resolved":33.83,"date":"2025-02-27","logs":true,"trajs":true,"site":"https://example.com","verified":true,"oss":true,"org_logo":"https://example.com/logo.png","warning":null        },// More entries...      ]    },// More leaderboards (Verified, Lite, Multimodal)...  ]}

3. Each leaderboard has aname and a list ofresults entries
4. Each result has various fields describing a model's performance
5. After updating the JSON, rebuild the site withmake build

To customize the visual appearance:

1. Main color variables are defined incss/core.css:

   • Edit color variables to change the overall color scheme
   • Update typography variables to change fonts

2. Layout structures are incss/layout.css:

   • Modify container sizes, grid layouts, etc.

3. Component styling is incss/components.css:

   • Update button styles, card styles, table styles, etc.

4. To add dark mode styles, look for.dark-mode selectors throughout the CSS files

When adding new features:

1. Avoid directly modifying existing code; extend it instead
2. Add new CSS in an appropriate file based on its purpose
3. Add new JavaScript files for new functionality
4. Update templates to include new components
5. Maintain the existing structure and coding style

The website is designed to be deployed to GitHub Pages:

1. Build the site withmake build
2. Commit and push changes to the GitHub repository under themain ormaster branch
3. Configure the domain to serve from thegh-pages branch (root directory) in your repository settings
4. The domain is configured via theCNAME file and the GitHub repository settings

Seedeploy.yml for the GitHub Actions workflow that handles automatic deployment to GitHub Pages.

Common issues:

• Build fails: Make sure you have all dependencies installed withmake install
• CSS changes not appearing: Check if you're editing the correct CSS file and if it's imported inmain.css. Force refresh the page (Cmd+Shift+R) if changes aren't showing.
• JavaScript not working: Check browser console for errors and ensure your script is included in the template
• Template changes not appearing: Make sure you're building the site after making changes withmake build.make serve also builds the site automatically.