- Notifications
You must be signed in to change notification settings - Fork1
Quickly and easily scrape Neal Agarwal's InfiniteCraft at 20-25 crafts/sec. Now part of FiniteCraft
quantumbagel/InfiniteScrape
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Here, have my proxy code@Pixelz22
This code is meant to be a smart auto-rescheduling multithreaded scraper for Neal Agarwal's Infinite Craft(https://neal.fun/infinite-craft).I have been able to get it to reach upwards of 25 crafts/sec with just 5 workers running.With actually good proxies, this could conceivably scale to hundreds, if not thousands, of crafts per second.
Currently, this project only has two folders:backend andlegacy.
legacy is me screwing around with proxies and web scraping, and shouldn't really be used.
backend is an incredibly good backend with high scalability.
Here's some features of my backend:
- Threadsafe scheduling
- MongoDB support
- No queue.Queue() lol
- Automatic proxy ranking and scheduling
- Plain awesome
@Pixelz22 decided it was a good idea to compete with me.
Here's his repository:https://github.com/Pixelz22/InfiniteCraft
- MongoDB -> neal.fun encoder
- actual algorithm
- Dockerfile
- Tyler