Repository files navigation

ScrapeCraft is a web-based scraping editor similar to Cursor but specialized for web scraping. It uses AI assistance to help users build scraping pipelines with the ScrapeGraphAI API.

• 🤖 AI-powered assistant using OpenRouter (Kimi-k2 model)
• 🔗 Multi-URL bulk scraping support
• 📋 Dynamic schema definition with Pydantic
• 💻 Python code generation with async support
• 🚀 Real-time WebSocket streaming
• 📊 Results visualization (table & JSON views)
• 🔄 Auto-updating deployment with Watchtower

• Backend: FastAPI, LangGraph, ScrapeGraphAI
• Frontend: React, TypeScript, Tailwind CSS
• Database: PostgreSQL
• Cache: Redis
• Deployment: Docker, Docker Compose, Watchtower

• Docker and Docker Compose
• OpenRouter API key (Get it fromOpenRouter)
• ScrapeGraphAI API key (Get it fromScrapeGraphAI)

1. Clone the repository

   git clone https://github.com/ScrapeGraphAI/scrapecraft.gitcd scrapecraft

2. Set up environment variables

   cp .env.example .env

   Edit the.env file and add your API keys:

   • OPENROUTER_API_KEY: Get fromOpenRouter
   • SCRAPEGRAPH_API_KEY: Get fromScrapeGraphAI

3. Start the application with Docker

   docker compose up -d

4. Access the application

   • Frontend:http://localhost:3000
   • API:http://localhost:8000
   • API Docs:http://localhost:8000/docs

5. Stop the application

   docker compose down

If you want to run the application in development mode without Docker:

cd backendpip install -r requirements.txtuvicorn app.main:app --reload --host 0.0.0.0 --port 8000

cd frontendnpm installnpm start

1. Create a Pipeline: Click "New Pipeline" to start
2. Add URLs: Use the URL Manager to add websites to scrape
3. Define Schema: Create fields for data extraction
4. Generate Code: Ask the AI to generate scraping code
5. Execute: Run the pipeline to scrape data
6. Export Results: Download as JSON or CSV

The application includes Watchtower for automatic updates:

1. Push new Docker images to your registry
2. Watchtower will automatically detect and update containers
3. No manual intervention required

• POST /api/chat/message - Send message to AI assistant
• GET /api/pipelines - List all pipelines
• POST /api/pipelines - Create new pipeline
• PUT /api/pipelines/{id} - Update pipeline
• POST /api/pipelines/{id}/run - Execute pipeline
• WS /ws/{pipeline_id} - WebSocket connection

MIT