Repository files navigation

Note: For the gateway component of this project,check this directory.

Usage demo 1:

Usage demo 2:

CodeQuery API is a lightweight Python/Flask tool designed to enable AI assistants—such as custom GPTs—to navigate and interact with local code effectively. With this API,LLM agents can query project structures and retrieve file contents, helping developers explore and manage large codebases. By adhering to customizable ignore patterns, the API ensures that only relevant files are accessed, making it an invaluable tool for AI-driven code analysis and development.

🤖Curious Fact: During its development, theCodeQuery API was an integral part of its own creation process, being used to analyze, write, and debug its own files while the project evolved. This unique feedback loop made it a participant in its own development stages! For more details on how the CodeQuery API has been applied, see theCases section.

• Designed for AI Assistants: This API was specifically designed to integrate with AI assistants such as custom GPTs, providing them with efficient access to project file structures and contents.
• Retrieve Project Structure: Get a detailed view of the project's directories and files.
• Retrieve File Contents: Access the contents of specific files in the project, with error handling for non-existent paths.
• Custom Ignore Patterns: Utilize.agentignore and/or.gitignore for specifying which files or directories to exclude from the structure retrieval.

The CodeQuery API consists of two components:

• ACore component that runs locally and accesses your files
• AGateway service that manages authentication and routes requests

• Docker: Required for running the Core component in a container
• Make: Used for simplified command execution
• curl: Needed for API key generation and testing
• ngrok account: Required for secure tunneling (free tier is sufficient)
  • Sign up athttps://dashboard.ngrok.com/signup
  • Get your authtoken athttps://dashboard.ngrok.com/get-started/your-authtoken

1. Get Required Credentials:

   # Generate a new API key from the Gateway servicecurl -X POST https://codequery.dev/api-keys/generate# Get your ngrok authtoken from https://dashboard.ngrok.com/get-started/your-authtoken# You'll need this in step 2c

2. Set Up the Core Component:

   a. Clone the repository:

   git clone https://github.com/danfmaia/CodeQuery-API.gitcd CodeQuery-API

   b. Initialize the environment:

   make init# Creates a fresh .env file from template# Use 'make clean' if you need to start fresh later

   c. Configure your.env file:

   • Set your project path
   • Add your API key from step 1
   • Add your ngrok authtoken from step 1

3. Start the Core:

   make build# Build the Docker imagemake run# Run the Core container

   The Core will automatically:

   • Start a secure ngrok tunnel
   • Register its URL with the Gateway
   • Begin accepting requests through the Gateway

4. Use the API:

   First, source the environment variables:

   source .env

   Then use the API:

   # Get the project structurecurl -H"X-API-KEY:$API_KEY" https://codequery.dev/files/structure# Get specific file contentscurl -X POST \  -H"Content-Type: application/json" \  -H"X-API-KEY:$API_KEY" \  -d'{"file_paths": ["README.md", "Makefile"]}' \  https://codequery.dev/files/content

5. Other Useful Commands:

   makehelp# Show all available commandsmake stop# Stop the containermaketest# Run testsmake logs# View container logs

The Gateway service provides functionality to manage your API keys. Here's how to perform common operations:

# Generate a key with default settings (30-day expiration, 60 requests/minute)curl -X POST https://codequery.dev/api-keys/generate# Generate a key with custom settingscurl -X POST -H"Content-Type: application/json" -d'{  "expiration_days": 90,  "requests_per_minute": 100}' https://codequery.dev/api-keys/generate

When you no longer need an API key, you can purge it from the server. This will:

• Delete the API key from the system
• Remove all associated data
• Invalidate any active ngrok tunnels for this key

To purge your API key:

source .env&& curl -X DELETE -H"X-API-KEY:$API_KEY" https://codequery.dev/api-keys/your-api-key

Note:

• You can only purge your own API key
• Once purged, a key cannot be recovered
• Make sure to generate a new key before purging if you plan to continue using the service

The.env file created bymake init contains all necessary configuration. The main variables you'll need to set are:

# Required settingsPROJECT_PATH=      # Path to the project you want to analyzeAPI_KEY=          # Your API key from step 1NGROK_AUTHTOKEN=  # Your ngrok authtoken

Other settings like ports and timeouts have sensible defaults and usually don't need modification.

For more detailed information about the API endpoints and advanced usage, see theDocumentation.

• Description: If you have a paid ngrok plan, set up a permanent public URL by running the Core component locally and using the ngrok URL for external access.

• Command:

  python run_local.py

• Use Case: Suitable for users who require a consistent external URL and prefer a simple setup.

• Description: For users with a static IP or home server, you can host the Core directly using your ISP's services, avoiding ngrok or Gateway usage.

• Command:

  python run_local.py

• Steps:

  1. Check Static IP Availability: Ensure your ISP offers a static IP.
  2. Port Forwarding: Configure your router to forward traffic on port 5001 to the local machine.
  3. Domain Setup: Consider using a custom domain for access.
  4. SSL/TLS Configuration: Use services like Let's Encrypt to secure the server.

• Endpoint:/files/structure

• Method:GET

• Description: Retrieves the directory structure of the project, respecting the ignore patterns in.agentignore. This is useful for tools that need to understand the file organization, such as code editors or static analysis tools.

• Response Example:

  {".": {"directories": ["backend","frontend","config"],"files": [".env","README.md"]  },"backend": {"directories": ["controllers","models","services"],"files": ["app.py","database.py"]  },"frontend": {"directories": ["components","pages"],"files": ["index.html","app.js"]  },"config": {"directories": [],"files": ["settings.yaml","logging.conf"]  }}

• Error Scenarios:

  • 500 Internal Server Error: If there's a failure in reading the directory structure, such as permission issues or corrupted files, an internal error response will be returned.

  Example Error Response:

  {"error":"Failed to retrieve directory structure: [Detailed error message]"}

• Endpoint:/files/content

• Method:POST

• Description: Retrieves the content of specified files. Useful for directly accessing specific source files or configuration files.

• Request Body:

  {"file_paths": ["backend/app.py","frontend/app.js"]}

• Response Example:

  {"backend/app.py": {"content":"# Main application file\nfrom flask import Flask\napp = Flask(__name__)\n\n@app.route('/')\ndef index():\n    return 'Hello, World!'"  },"frontend/app.js": {"content":"// Frontend application logic\nimport React from 'react';\nfunction App() {\n    return (<div>Hello, World!</div>);\n}"  }}

• Error Scenarios:

  • 400 Bad Request: If no file paths are provided in the request body.

  Example Error Response:

  {"error":"No file paths provided in the request."}

  • 404 Not Found: If all the requested file paths do not exist or are missing.

  Example Error Response:

  {"error":"All requested files are missing"}

  • 422 Unprocessable Entity: If a directory is specified instead of a file.

  Example Error Response:

  {"frontend/components": {"error":"Cannot read directory: frontend/components"  }}

  • 500 Internal Server Error: If there's a failure in reading a file due to permissions, encoding issues, or other OS-level errors.

  Example Error Response:

  {"backend/app.py": {"error":"Error reading file: [Detailed error message]"  }}

The.agentignore file works similarly to.gitignore, allowing you to specify files and directories that should be excluded from file structure queries (/files/structure endpoint).

Example:

# General.git/.gitignore.vscode/assets/public/# Pythonsrc/__pycache__/tests/__pycache__/__pycache__/venv/.benchmarks/.pytest_cache/.envrequirements.txt# JavaScriptnode_modules/package-lock.jsonpackage.json

This API was designed to be used by custom AI assistants. If you are a ChatGPT Premium user, you can create a custom GPT using theChatGPT Builder.

1. Go to theGPT Builder in your ChatGPT Premium account.
2. Access theCreate tab.
3. Send the following prompt to the GPT Builder to create your custom GPT:

Name: CodeQueryGPTDescription: Helps developers analyze code, debug issues, and develop features, by leveraging an API to retrieve project structure and files.Instructions:"""You are CodeQueryGPT, an AI specialized in assisting with software development tasks by actively querying project files, analyzing code structure, and providing coding support. You use an external API to fetch file structures and retrieve file contents as needed.Your goal is to assist with code analysis, feature development, debugging, and understanding code dependencies, contributing directly to the development process. You should determine when to query the project structure or relevant files, integrating this step into your workflow naturally.Use a Chain of Thought (CoT) approach to break down complex tasks into clear steps. When debugging or implementing features, proactively query test results, logs, and relevant files to understand the problem or requirements. Focus on suggesting clear, actionable code changes or improvements while ensuring that you have all the necessary context to perform the task effectively."""Conversation Starters:- Analyse the code following a thorough CoT process. Use both endpoints.- Help me investigate and debug an issue in the code.- I need assistance in developing a new feature.- Query the main files. Then pick some method(s) refactor them for better performance.

4. Once the GPT is created, go to theConfigure tab.
5. [Optional] Customize the GPT initialization settings as needed.
6. Enable the"Code Interpreter & Data Analysis" option.
7. Create a newAction by providing the followingOpenAPI schema:

{"openapi":"3.1.0","info": {"title":"CodeQuery API","description":"A Flask API to retrieve the file structure and contents of a project directory.","version":"1.0.0"  },"servers": [    {"url":"https://codequery.dev"    }  ],"paths": {"/files/structure": {"get": {"summary":"Retrieve the file structure","description":"Returns the file structure of the project directory in a nested format, showing directories and files.","operationId":"getFileStructure","responses": {"200": {"description":"Successful response with the file structure","content": {"application/json": {"schema": {"type":"object","properties": {"directories": {"type":"array","items": {"type":"string"                      },"description":"List of directory names"                    },"files": {"type":"array","items": {"type":"string"                      },"description":"List of file names"                    }                  }                }              }            }          }        }      }    },"/files/content": {"post": {"summary":"Retrieve file contents","description":"Accepts a list of file paths and returns their contents or an error message if the file does not exist.","operationId":"retrieveFiles","requestBody": {"content": {"application/json": {"schema": {"type":"object","properties": {"file_paths": {"type":"array","items": {"type":"string"                    },"description":"A list of file paths to retrieve"                  }                },"required": ["file_paths"]              }            }          }        },"responses": {"200": {"description":"Successful response with file contents","content": {"application/json": {"schema": {"type":"object","properties": {"file_path": {"type":"object","additionalProperties": {"type":"object","properties": {"content": {"type":"string","description":"The content of the file"                          },"error": {"type":"string","description":"Error message in case of failure"                          }                        }                      }                    }                  }                }              }            }          },"400": {"description":"Error when no file paths are provided","content": {"application/json": {"schema": {"type":"object","properties": {"error": {"type":"string","description":"Error message"                    }                  }                }              }            }          },"404": {"description":"Error when all requested files are missing","content": {"application/json": {"schema": {"type":"object","properties": {"error": {"type":"string","description":"Error message"                    }                  }                }              }            }          }        }      }    }  }}

TheCoreQuery API itself is the first use case of the CodeQuery API, and it's the project you're currently exploring. It serves as a powerful development tool, integrating with AI assistants (such as theCodeQueryGPT) to support developers by providing a structured way to query project files, understand code dependencies, and interact with large codebases. This project was developed using aTest-Driven Development (TDD) approach to ensure the correctness of the AI-generated code.

SkillChrono is a Python-based tool designed to help developers organize and visualize their technical skills across various projects. It processes structured data, aggregates experience per technology, and generates markdown reports sorted both alphabetically and by experience duration. SkillChrono was also built using aTDD approach, and theCodeQuery API was integral to its development, supporting everything from feature implementation to documentation generation.

For more details, see theSkillChrono repository.

For information on how data is handled by this API, please refer to thePrivacy Policy. The policy explains what data is processed, how it's used, and the lack of data retention within the API.

This project is licensed under the Apache License, Version 2.0.
You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0