Repository files navigation

OverFast API provides comprehensive data on Overwatch 2 heroes, game modes, maps, and player statistics by scraping Blizzard pages. Developed with the efficiency ofFastAPI andSelectolax, it leveragesnginx (OpenResty) as a reverse proxy andRedis for caching. Its tailored caching mechanism significantly reduces calls to Blizzard pages, ensuring swift and precise data delivery to users.

• ✨ Live instance
• 🐋 Run for production
• 💽 Run as developer
• 👨‍💻 Technical details
• 🐍 Architecture
• 🤝 Contributing
• 🚀 Community projects
• 🙏 Credits
• 📝 License

The live instance operates with a rate limit applied per second, shared across all endpoints. You can view the current rate limit on the home page, and this limit may be adjusted as needed. For higher request throughput, consider hosting your own instance on a dedicated server 👍

• Live instance (Redoc documentation) :https://overfast-api.tekrop.fr/
• Swagger UI :https://overfast-api.tekrop.fr/docs
• Status page :https://uptime-overfast-api.tekrop.fr/

Running the project is straightforward. Ensure you havedocker anddocker compose installed. Next, generate a.env file using the provided.env.dist template. Finally, execute the following command:

make up

Same as earlier, ensure you havedocker anddocker compose installed, and generate a.env file using the provided.env.dist template. You can customize the.env file according to your requirements to configure the volumes used by the OverFast API.

Then, execute the following commands to launch the dev server :

make build# Build the images, needed for all further commandsmake start# Launch OverFast API (dev mode with autoreload)make start TESTING_MODE=true# Launch OverFast API (testing mode, with reverse proxy)

The dev server will be running on the port8000. Reverse proxy will be running on the port8080 in testing mode. You can use themake down command to stop and remove the containers. Feel free to typemake ormake help to access a comprehensive list of all available commands for your reference.

Should you wish to customize according to your specific requirements, here is a detailed list of available settings:

• APP_VOLUME_PATH: Folder for shared app data like logs, Redis save file and dotenv file (app settings)
• APP_PORT: Port for the app container (default is80).
• APP_BASE_URL : Base URL for exposed links in endpoints like player search and maps listing.

You likely won't need to modify other generic settings, but if you're curious about their functionality, consult the docstrings within theapp/config.py file for further details.

The code quality is checked usingruff. I'm also usingruff format for imports ordering and code formatting, enforcing PEP-8 convention on my code. To check the quality of the code, you just have to run the following command :

make lint# Run ruff lintermake format# Run ruff formatter

The code has been tested using unit testing, except some rare parts which are not relevant to test. There are tests on the parsers classes, the common classes, but also on the commands (run in CLI) and the API views (using FastAPI TestClient class).

Running tests with coverage (default)

maketest

Running tests with given args (without coverage)

maketest PYTEST_ARGS="tests/common"

The project is usingpre-commit framework to ensure code quality before making any commit on the repository. After installing the project dependencies, you can install the pre-commit by using thepre-commit install command.

The configuration can be found in the.pre-commit-config.yaml file. It consists in launching 2 processes on modified files before making any commit :

• ruff for linting and code formatting (withruff format)
• sourcery for more code quality checks and a lot of simplifications

In player career statistics, various conversions are applied for ease of use:

• Duration values are converted toseconds (integer)
• Percent values are represented asintegers, omitting the percent symbol
• Integer and float string representations are converted to their respective types

OverFast API integrates aRedis-based cache system, divided into three main components:

• API Cache: This high-level cache associates URIs (cache keys) with raw JSON data. Upon the initial request, if a cache entry exists, thenginx server returns the JSON data directly. Cached values are stored with varying TTL (Time-To-Live) parameters depending on the requested route.
• Player Cache: Specifically designed for the API's players data endpoints, this cache stores both HTML Blizzard pages (profile) and search results (summary) for a given player. Its purpose is to minimize calls to Blizzard servers whenever an associated API Cache is expired, and player's career hasn't changed since last call, by usinglastUpdated value fromsummary. This cache will only expire if not accessed for a given TTL (default is 3 days).
• Search Data Cache: Cache the player search endpoint to store mappings betweenavatar,namecard, andtitle URLs and their corresponding IDs. On profile pages, only the ID values are accessible, so we initialize this "Search Data" cache when the app launches.

Below is the current list of TTL values configured for the API cache. The latest values are available on the API homepage.

• Heroes list : 1 day
• Hero specific data : 1 day
• Roles list : 1 day
• Gamemodes list : 1 day
• Maps list : 1 day
• Players career : 1 hour
• Players search : 10 min

The default case is pretty straightforward. When aUser makes an API request,Nginx first checksRedis for cached data :

• If available,Redis returns the data directly toNginx, which forwards it to theUser (cache hit).
• If the cache is empty (cache miss),Nginx sends the request to theApp server, which retrieves and parses data fromBlizzard.

TheApp then stores this data inRedis and returns it toNginx, which sends the response to theUser. This approach minimizes external requests and speeds up response times by prioritizing cached data.

sequenceDiagram    autonumber    actor User    participant Nginx    participant Redis    participant App    participant Blizzard    User->>+Nginx: Make an API request    Nginx->>+Redis: Make an API Cache request    alt API Cache is available        Redis-->>Nginx: Return API Cache data        Nginx-->>User: Return API Cache data    else        Redis-->>-Nginx: Return no result        Nginx->>+App: Transmit the request to App server        App->>+Blizzard: Retrieve data        Blizzard-->>-App: Return data        App->>App: Parse HTML page        App->>Redis: Store data into API Cache        App-->>-Nginx: Return API data        Nginx-->>-User: Return API data    end

The player profile request flow is similar to the previous setup, but with an extra layer of caching for player-specific data, including HTML data (profile page) and player search data (JSON data).

When aUser makes an API request,Nginx checksRedis for cached API data. If found, it’s returned directly. If not,Nginx forwards the request to theApp server.

It then retrieves Search data fromBlizzard and checks a Player Cache inRedis :

• If the player data is cached and up-to-date (lastUpdated from Search data has not changed),App parses it
• If not,App retrieves and parses the data fromBlizzard, then stores it in both the Player Cache and API Cache.

This additional Player Cache layer reduces external calls for player-specific data, especially when player career hasn't changed, improving performance and response times.

sequenceDiagram    autonumber    actor User    participant Nginx    participant Redis    participant App    participant Blizzard    User->>+Nginx: Make an API request    Nginx->>+Redis: Make an API Cache request    alt API Cache is available        Redis-->>Nginx: Return API Cache data        Nginx-->>User: Return API Cache data    else        Redis-->>-Nginx: Return no result        Nginx->>+App: Transmit the request to App server        App->>+Blizzard: Make a Player Search request        Blizzard-->>-App: Return Player Search data        App->>+Redis: Make Player Cache request        alt Player Cache is available and up-to-date            Redis-->>App: Return Player Cache            App->>App: Parse HTML page        else            Redis-->>-App: Return no result            App->>+Blizzard: Retrieve Player data (HTML)            Blizzard-->>-App: Return Player data            App->>App: Parse HTML page            App->>Redis: Store data into Player Cache        end        App->>Redis: Store data into API Cache        App-->>-Nginx: Return API data        Nginx-->>-User: Return API data    end

Contributions, issues and feature requests are welcome ! Do you want to update the heroes data (health, armor, shields, etc.) or the maps list ? Don't hesitate to consult the dedicatedCONTRIBUTING file.

Projects using OverFast API as a data source are listed below. Using it in your project? Reach out via email with your project link, and I'll add it!

• Counterwatch, enhances your Overwatch 2 experience (https://www.counterwatch.app)
• Datastrike, analysis and results tracking tool (https://datastrike.cloud)
• Discord Bot OW2 for stats (https://github.com/polsojac/ow2discordbot)
• OverBot, the best Overwatch bot for Discord (https://github.com/davidetacchini/overbot)
• Overfast API client (https://github.com/Sipixer/overfast-api-client)
• Overwatch Career Profile (https://github.com/EliaRenov/ow-career-profile)
• OverwatchPy, a Python wrapper for the API (https://github.com/alexraskin/overwatchpy)
• OWCOUNTER, a tool to help players learn and improve their hero selection and team strategy (https://owcounter.com/)
• Watch Over, mobile app by @Backxtar (https://play.google.com/store/apps/details?id=de.backxtar.watchoveroverwatch)

All maps screenshots hosted by the API are owned by Blizzard. Sources :

• Blizzard Press Center (https://blizzard.gamespress.com)
• Overwatch Wiki (https://overwatch.fandom.com/wiki/)

Copyright © 2021-2024Valentin PORCHET.

This project isMIT licensed.