Repository files navigation

This repository includes a simple Python FastAPI app that streams responses from Azure OpenAI GPT models.

The repository is designed for use withDocker containers, both for local development and deployment, and includes infrastructure files for deployment toAzure Container Apps. 🐳

We recommend first going through thedeployment steps before running this app locally,since the local app needs credentials for Azure OpenAI to work properly.

This project hasDev Container support, so it will be be setup automatically if you open it in Github Codespaces or in local VS Code with theDev Containers extension.

If you're not using one of those options for opening the project, then you'll need to:

1. Create aPython virtual environment and activate it.

2. Install the requirements:

   python3 -m pip install -r requirements-dev.txt

3. Install the pre-commit hooks:

   pre-commit install

This repo is set up for deployment on Azure Container Apps using the configuration files in theinfra folder.

1. Sign up for afree Azure account and create an Azure Subscription.
2. Request access to Azure OpenAI Service by completing the form athttps://aka.ms/oai/access and awaiting approval.
3. Install theAzure Developer CLI. (If you open this repository in Codespaces or with the VS Code Dev Containers extension, that part will be done for you.)

1. Login to Azure:

   azd auth login

2. Provision and deploy all the resources:

   azd up

   It will prompt you to provide anazd environment name (like "chat-app"), select a subscription from your Azure account, and select alocation where OpenAI is available (like "francecentral"). Then it will provision the resources in your account and deploy the latest code. If you get an error or timeout with deployment, changing the location can help, as there may be availability constraints for the OpenAI resource.

3. Whenazd has finished deploying, you'll see an endpoint URI in the command output. Visit that URI, and you should see the chat app! 🎉

4. When you've made any changes to the app code, you can just run:

   azd deploy

You can pair this backend with a frontend of your choice.The frontend needs to be able to read NDJSON from a ReadableStream,and send JSON to the backend with an HTTP POST request.The JSON schema should conform to theChat App Protocol.

Here are frontends that are known to work with this backend:

• OpenAI Chat App Frontend - VanillaJS

To pair a frontend with this backend, you'll need to:

1. Deploy the backend using the steps above. Make sure to note the endpoint URI.
2. Open the frontend project.
3. Deploy the frontend using the steps in the frontend repo, following their instructions for setting the backend endpoint URI.
4. Open this project again.
5. Runazd env set ALLOWED_ORIGINS "https://<your-frontend-url>". That URL (or list of URLs) will specified in the CORS policy for the backend to allow requests from your frontend.
6. Runazd up to deploy the backend with the new CORS policy.

Pricing varies per region and usage, so it isn't possible to predict exact costs for your usage.The majority of the Azure resources used in this infrastructure are on usage-based pricing tiers.However, Azure Container Registry has a fixed cost per registry per day.

You can try theAzure pricing calculator for the resources:

• Azure OpenAI Service: S0 tier, ChatGPT model. Pricing is based on token count.Pricing
• Azure Container App: Consumption tier with 0.5 CPU, 1GiB memory/storage. Pricing is based on resource allocation, and each month allows for a certain amount of free usage.Pricing
• Azure Container Registry: Basic tier.Pricing
• Log analytics: Pay-as-you-go tier. Costs based on data ingested.Pricing

⚠️ To avoid unnecessary costs, remember to take down your app if it's no longer in use,either by deleting the resource group in the Portal or runningazd down.

Assuming you've run the steps inOpening the project and have runazd up, you can now run the FastAPI app locally using theuvicorn server:

python -m uvicorn"src.api:create_app" --reload --factory

You may want to save costs by developing against a local LLM server, such asllamafile. Note that a local LLMwill generally be slower and not as sophisticated.

Once you've got your local LLM running and serving an OpenAI-compatible endpoint, defineLOCAL_OPENAI_ENDPOINT in your.env file.

For example, to point at a local llamafile server running on its default port:

LOCAL_OPENAI_ENDPOINT="http://localhost:8080/v1"

If you're running inside a dev container, use this local URL instead:

LOCAL_OPENAI_ENDPOINT="http://host.docker.internal:8080/v1"

In addition to theDockerfile that's used in production, this repo includes adocker-compose.yaml forlocal development which creates a volume for the app code. That allows you to make changes to the codeand see them instantly.

1. InstallDocker Desktop. If you opened this inside Github Codespaces or a Dev Container in VS Code, installation is not needed.⚠️ If you're on an Apple M1/M2, you won't be able to rundocker commands inside a Dev Container; either use Codespaces or do not open the Dev Container.

2. Make sure that the.env file exists. Theazd up deployment step should have created it.

3. Add your Azure OpenAI API key to the.env file if not already there. You can find your key in the Azure Portal for the OpenAI resource, underKeys and Endpoint tab.

   AZURE_OPENAI_KEY="<your-key-here>"

4. Start the services with this command:

   docker-compose up --build

5. Click 'http://0.0.0.0:3100' in the terminal, which should open a new tab in the browser. You may need to navigate to 'http://localhost:3100' if that URL doesn't work.