About HTTPS¶

It is easy to assume that HTTPS is something that is just "enabled" or not.

But it is way more complex than that.

Tolearn the basics of HTTPS, from a consumer perspective, checkhttps://howhttps.works/.

Now, from adeveloper's perspective, here are several things to keep in mind while thinking about HTTPS:

• For HTTPS,the server needs tohave "certificates" generated by athird party.
  • Those certificates are actuallyacquired from the third party, not "generated".
• Certificates have alifetime.
  • Theyexpire.
  • And then they need to berenewed,acquired again from the third party.
• The encryption of the connection happens at theTCP level.
  • That's one layerbelow HTTP.
  • So, thecertificate and encryption handling is donebefore HTTP.
• TCP doesn't know about "domains". Only about IP addresses.
  • The information about thespecific domain requested goes in theHTTP data.
• TheHTTPS certificates "certify" acertain domain, but the protocol and encryption happen at the TCP level,before knowing which domain is being dealt with.
• By default, that would mean that you can only haveone HTTPS certificate per IP address.
  • No matter how big your server is or how small each application you have on it might be.
  • There is asolution to this, however.
• There's anextension to theTLS protocol (the one handling the encryption at the TCP level, before HTTP) calledSNI.
  • This SNI extension allows one single server (with asingle IP address) to haveseveral HTTPS certificates and servemultiple HTTPS domains/applications.
  • For this to work, asingle component (program) running on the server, listening on thepublic IP address, must haveall the HTTPS certificates in the server.
• After obtaining a secure connection, the communication protocol isstill HTTP.
  • The contents areencrypted, even though they are being sent with theHTTP protocol.

It is a common practice to haveone program/HTTP server running on the server (the machine, host, etc.) andmanaging all the HTTPS parts: receiving theencrypted HTTPS requests, sending thedecrypted HTTP requests to the actual HTTP application running in the same server (theFastAPI application, in this case), take theHTTP response from the application,encrypt it using the appropriateHTTPS certificate and sending it back to the client usingHTTPS. This server is often called aTLS Termination Proxy.

Some of the options you could use as a TLS Termination Proxy are:

• Traefik (that can also handle certificate renewals)
• Caddy (that can also handle certificate renewals)
• Nginx
• HAProxy

Let's Encrypt¶

Before Let's Encrypt, theseHTTPS certificates were sold by trusted third parties.

The process to acquire one of these certificates used to be cumbersome, require quite some paperwork and the certificates were quite expensive.

But thenLet's Encrypt was created.

It is a project from the Linux Foundation. It providesHTTPS certificates for free, in an automated way. These certificates use all the standard cryptographic security, and are short-lived (about 3 months), so thesecurity is actually better because of their reduced lifespan.

The domains are securely verified and the certificates are generated automatically. This also allows automating the renewal of these certificates.

The idea is to automate the acquisition and renewal of these certificates so that you can havesecure HTTPS, for free, forever.

HTTPS for Developers¶

Here's an example of how an HTTPS API could look like, step by step, paying attention mainly to the ideas important for developers.

Domain Name¶

It would probably all start by youacquiring somedomain name. Then, you would configure it in a DNS server (possibly your same cloud provider).

You would probably get a cloud server (a virtual machine) or something similar, and it would have afixedpublic IP address.

In the DNS server(s) you would configure a record (an "A record") to pointyour domain to the publicIP address of your server.

You would probably do this just once, the first time, when setting everything up.

DNS¶

Now let's focus on all the actual HTTPS parts.

First, the browser would check with theDNS servers what is theIP for the domain, in this case,someapp.example.com.

The DNS servers would tell the browser to use some specificIP address. That would be the public IP address used by your server, that you configured in the DNS servers.

TLS Handshake Start¶

The browser would then communicate with that IP address onport 443 (the HTTPS port).

The first part of the communication is just to establish the connection between the client and the server and to decide the cryptographic keys they will use, etc.

This interaction between the client and the server to establish the TLS connection is called theTLS handshake.

TLS with SNI Extension¶

Only one process in the server can be listening on a specificport in a specificIP address. There could be other processes listening on other ports in the same IP address, but only one for each combination of IP address and port.

TLS (HTTPS) uses the specific port443 by default. So that's the port we would need.

As only one process can be listening on this port, the process that would do it would be theTLS Termination Proxy.

The TLS Termination Proxy would have access to one or moreTLS certificates (HTTPS certificates).

Using theSNI extension discussed above, the TLS Termination Proxy would check which of the TLS (HTTPS) certificates available it should use for this connection, using the one that matches the domain expected by the client.

In this case, it would use the certificate forsomeapp.example.com.

The client alreadytrusts the entity that generated that TLS certificate (in this case Let's Encrypt, but we'll see about that later), so it canverify that the certificate is valid.

Then, using the certificate, the client and the TLS Termination Proxydecide how to encrypt the rest of theTCP communication. This completes theTLS Handshake part.

After this, the client and the server have anencrypted TCP connection, this is what TLS provides. And then they can use that connection to start the actualHTTP communication.

And that's whatHTTPS is, it's just plainHTTP inside asecure TLS connection instead of a pure (unencrypted) TCP connection.

HTTPS Request¶

Now that the client and server (specifically the browser and the TLS Termination Proxy) have anencrypted TCP connection, they can start theHTTP communication.

So, the client sends anHTTPS request. This is just an HTTP request through an encrypted TLS connection.

Decrypt the Request¶

The TLS Termination Proxy would use the encryption agreed todecrypt the request, and would transmit theplain (decrypted) HTTP request to the process running the application (for example a process with Uvicorn running the FastAPI application).

HTTP Response¶

The application would process the request and send aplain (unencrypted) HTTP response to the TLS Termination Proxy.

HTTPS Response¶

The TLS Termination Proxy would thenencrypt the response using the cryptography agreed before (that started with the certificate forsomeapp.example.com), and send it back to the browser.

Next, the browser would verify that the response is valid and encrypted with the right cryptographic key, etc. It would thendecrypt the response and process it.

The client (browser) will know that the response comes from the correct server because it is using the cryptography they agreed using theHTTPS certificate before.

Multiple Applications¶

In the same server (or servers), there could bemultiple applications, for example, other API programs or a database.

Only one process can be handling the specific IP and port (the TLS Termination Proxy in our example) but the other applications/processes can be running on the server(s) too, as long as they don't try to use the samecombination of public IP and port.

That way, the TLS Termination Proxy could handle HTTPS and certificates formultiple domains, for multiple applications, and then transmit the requests to the right application in each case.

Certificate Renewal¶

At some point in the future, each certificate wouldexpire (about 3 months after acquiring it).

And then, there would be another program (in some cases it's another program, in some cases it could be the same TLS Termination Proxy) that would talk to Let's Encrypt, and renew the certificate(s).

TheTLS certificates areassociated with a domain name, not with an IP address.

So, to renew the certificates, the renewal program needs toprove to the authority (Let's Encrypt) that it indeed"owns" and controls that domain.

To do that, and to accommodate different application needs, there are several ways it can do it. Some popular ways are:

• Modify some DNS records.
  • For this, the renewal program needs to support the APIs of the DNS provider, so, depending on the DNS provider you are using, this might or might not be an option.
• Run as a server (at least during the certificate acquisition process) on the public IP address associated with the domain.
  • As we said above, only one process can be listening on a specific IP and port.
  • This is one of the reasons why it's very useful when the same TLS Termination Proxy also takes care of the certificate renewal process.
  • Otherwise, you might have to stop the TLS Termination Proxy momentarily, start the renewal program to acquire the certificates, then configure them with the TLS Termination Proxy, and then restart the TLS Termination Proxy. This is not ideal, as your app(s) will not be available during the time that the TLS Termination Proxy is off.

All this renewal process, while still serving the app, is one of the main reasons why you would want to have aseparate system to handle HTTPS with a TLS Termination Proxy instead of just using the TLS certificates with the application server directly (e.g. Uvicorn).

Proxy Forwarded Headers¶

When using a proxy to handle HTTPS, yourapplication server (for example Uvicorn via FastAPI CLI) doesn't known anything about the HTTPS process, it communicates with plain HTTP with theTLS Termination Proxy.

Thisproxy would normally set some HTTP headers on the fly before transmitting the request to theapplication server, to let the application server know that the request is beingforwarded by the proxy.

Nevertheless, as theapplication server doesn't know it is behind a trustedproxy, by default, it wouldn't trust those headers.

But you can configure theapplication server to trust theforwarded headers sent by theproxy. If you are using FastAPI CLI, you can use theCLI Option--forwarded-allow-ips to tell it from which IPs it should trust thoseforwarded headers.

For example, if theapplication server is only receiving communication from the trustedproxy, you can set it to--forwarded-allow-ips="*" to make it trust all incoming IPs, as it will only receive requests from whatever is the IP used by theproxy.

This way the application would be able to know what is its own public URL, if it is using HTTPS, the domain, etc.

This would be useful for example to properly handle redirects.

Recap¶

HavingHTTPS is very important, and quitecritical in most cases. Most of the effort you as a developer have to put around HTTPS is just aboutunderstanding these concepts and how they work.

But once you know the basic information ofHTTPS for developers you can easily combine and configure different tools to help you manage everything in a simple way.

In some of the next chapters, I'll show you several concrete examples of how to set upHTTPS forFastAPI applications. 🔒