Jul 23, 2021

How to send emails from Dream

I recentlylearned theOCaml ecosystem has most of the libraries needed to build emailfeatures, but I wasn't able to find an example that brings all ofthose pieces together. I wanted a simple example I could easily adaptto build common account verification and password recovery workflowsin the Dream web framework. This post is my attempt to solve thatproblem and explain what I learned in the process. The accompanyingcode can be foundhere.

Defining the problem

As an application developer, my main concerns whensending mail are:

1. Trying to ensure the address I'm sending to is valid.
2. Handling the email synthesis process asynchronously, so it doesn't block the web server from handling requests quickly.
3. Transmitting the email content (SMTP or a REST API call)
4. Monitoring that the applications messages aren't getting blocked/dropped for some reason.

To test how easily I could satisfy these requirements, I built a webapp that lets the user send an arbitary email.

The app has two parts:

• A server process responsible for showing the form above and validating user inputs.
• A "worker" process responsible for sending mail. The worker gets its tasks from a queue populated by the server.

Validating Email Addresses

I decided to utilize the libraryemile for addressvalidation. An email address might be invalid for reasons outside ofthe application's control (like the domain going way), butemile atleast provides a way to check a given string conforms to 5(!) RFCsrelevant to parsing email addresses. Check out theteststo see the impressive diversity of valid email addresses on the web.

The library provides several different predicates for checkingaddresses. I utilizedEmile.of_string, which generates a simpleresult type, like this:

letform_postrequest=match%lwtDream.formrequestwith|`Ok["address",address;"message",message;"subject",subject]->letparse=Emile.of_stringaddressinletalert=matchparsewith|Ok_->Printf.sprintf"Sent an email to %s."address|Error_->Printf.sprintf"%s is not a valid email address."addressinletqueue_req=matchparsewith|Ok_->queue_emailaddresssubjectmessage|Error_->Lwt.return_unitinqueue_req>>=(fun_->Dream.html(Template.show_form~alertrequest))|_->Dream.empty`Bad_Request

This way, if a user provides an invalid address, the form displays a simplewarning message and won't queue any email tasks.

Queuing Tasks with RabbitMQ

The second issue I needed to address is handling the email sendprocess with a background worker (queue_email in the code excerptabove). Sending an email can be a slow process, especially if you needto make additional database queries or render images to produce acustomized email. That shouldn't block the web server from quicklyresponding to the user's requests. Instead, the server should recordthe email task in aqueue, to be handled by a background worker thatisn't as sensitive to latency.

There are several different technologies that can be used to solvethis problem, for exampleRabbitMQ,RedisandSQS. I decided to use RabbitMQbecause I didn't want to tie my implementation to one cloud providerand RabbitMQ is supported byamqp-client onOPAM. This isn't the only way to go: for Redis and SQS, you might havesuccess with the packagesredis oraws-sqs, respectively.

The worker and server process share a common function for establishinga connection to RabbitMQ:

letrabbit_connection()=let%lwtconnection=Connection.connect~id:"dream""localhost"inlet%lwtchannel=Connection.open_channel~id:"email"Channel.no_confirmconnectioninlet%lwtqueue=Queue.declarechannel~arguments:[Rpc.Server.queue_argument]"email"inLwt.return(channel,queue)

I was initially little confused about why I needed a "channel" when Ialready had a connection to the broker. Both the worker and the serverconnect to the RabbitMQ Broker via TCP. It's common for clients toneed several connections to the broker, but it's costly to have manyTCP connections. RabbitMQ lets us avoid this cost with channels, whichare like "lightweight connections that share a single TCPconnection", according to thesedocs.

I created this function to submit messages to the queue:

typeemail={address:string;subject:string;text:string;}[@@derivingyojson]letqueue_emailaddresssubjecttext=letemail={address;subject;text}inlettext=email|>yojson_of_email|>Yojson.Safe.to_stringinlet%lwtchannel,queue=rabbit_connection()inlet%lwt_=Queue.publishchannelqueue(Message.maketext)inLwt.return_unit

Here the server writes the address, subject, and text into a recordtype that can be serialized to JSON before being pushed into the queue.

The worker process,queue_worker, is fairly simple to define:

lethandle_message(m:Message.t)=lettext=sndm.messageinletemail=trySome(text|>Yojson.Safe.from_string|>email_of_yojson)with_->Printf.printf"Received invalid email record from RabbitMQ: %s"text;Noneinmatchemailwith|Somee->send_emaile|None->Lwt.return_unitletqueue_worker()=Printf.printf"Starting Queue Worker\n";let%lwtchannel,queue=rabbit_connection()inletrechandle()=Queue.get~no_ack:truechannelqueue>>=funmessage->lettask=matchmessagewith|Somem->flushstdout;handle_messagem|_->Printf.printf"No new tasks, waiting.\n";flushstdout;Lwt_unix.sleep5.intask>>=handleinhandle()

The worker consists of an infinite loop. When the worker wakes andfinds tasks in the queue, it usesyojson to deserialize themessages and passes them tosend_email. If no work is available,the worker just sleeps for 5 seconds.

Sending Email withcohttp and Mailgun

In order to send mail, I set up a trialMailgunaccount. I selected this provider for several reasons:

• Mailgun has a generous free tier (5000 emails).
• A new account also comes with a "sandbox" domain, so I didn't have to deal with registering a domain.
• Mailgun supports both a REST API and SMTP, allowing me to try different integrations.
• Mailgun provides dashboards for tracking messages sent over time as well as emails that were rejected (bounced, marked as spam, etc.).

Mailgun's API allowed me to send an email by sending a POST with an API key and formdata, like this:

letsend_email(e:email)=letopenCohttpinletopenCohttp_lwt_unixinletapi_key=Sys.getenv"MAILGUN_API_KEY"inletsender=Sys.getenv"MAILGUN_SEND_ADDRESS"inletapi_base=Sys.getenv"MAILGUN_API_BASE"inletparams=[("from",[sender]);("to",[e.address]);("subject",[e.subject]);("text",[e.text])]inletcred=`Basic("api",api_key)inleturi=api_base^"/messages"|>Uri.of_stringinletheaders=Header.add_authorization(Header.init())credinletstart=Printf.printf"Initiating email send to %s.\n"e.address;Unix.gettimeofday()inlet%lwtresp,rbody=Client.post_formuri~params~headersinlet%lwtfinish=Unix.gettimeofday()|>Lwt.returninlet%lwtrbody_str=(Cohttp_lwt.Body.to_stringrbody)inPrintf.printf"API Response Status: %d.\n"(Code.code_of_statusresp.status);Printf.printf"API Response body %s.\n"rbody_str;Printf.printf"Time to send an email: %.2fs\n"(finish-.start);Lwt.return()

The endpoint replies with either 200 and a JSON body like this:

{"id":"<some mailgun email address here>""message":"Queued. Thank you."}

or else a 400 and an explanation of why the request was rejected. Inmy tests, a successful API request typically took between 0.5 and 1seconds. For comparison, the webserver can serve the email form inless than 200 microseconds, so the performance benefits to shiftingemail tasks to a background queue are nontrivial.

Sending email with SMTP

Mailgun supports SMTP but they recommend using their API, especiallyfor sending large amounts of mail at once. I usedletters to test their SMTPsupport with this script:

letbody=Letters.Plain{|Thisisatestemailbody.|}letsend_email()=letconfig=(Letters.Config.make~username:Sys.getenv"MAILGUN_SMTP_USER"~password:Sys.getenv"MAILGUN_SMTP_PASSWORD"~hostname:"smtp.mailgun.org"~with_starttls:true)inletsender=Sys.getenv"MAILGUN_SMTP_SENDER"inletrecipients=[Letters.To(Sys.getenv"EMAIL_ADDRESS")]inletsubject="Test email from mailgun."inletmail=Letters.build_email~from:sender~recipients~subject~bodyinPrintexc.record_backtracetrue;matchmailwith|Okmessage->(try%lwtLetters.send~config~sender~recipients~messagewithe->Printf.printf"Error. %s\n"(Printexc.to_stringe);Printexc.print_backtracestdout;Lwt.return_unit)|Error_->Printf.printf"Message synthesis failed.";flushstdout;Lwt.return_unitlet()=Lwt_main.run@@send_email()

I ran into somedifficulties becauseMailgun does not consistently follow RFC 4954 (thanks to CalascibettaRomain for figuring out the problem and creating apatch). Once I appliedthe patch, I was able to send mail successfully.

I haven't fully explored the tradeoffs of using SMTP versus anAPI. The main goal of my experiements was to confirm that if I neededto integrate with a specific SMTP server, I knew how to do so.

Discussion

This example focused on email, but task queues and background workerscan be used to implement other important pieces of functionality forweb applications. For example, I've utilizedceleryin the past to generate weekly reports and schedule recurring ETLjobs. This exercise was a nice opportunity to see how one wouldconstruct a system similar tocelery in OCaml. It also provides asimple illustration of how we can utilize queues to facilitatehorizontal scaling. If the volume of email requests grows too big forone machine, we can put the web server and the worker on separateinstances or have multiple worker instances consuming from the emailqueue.

I didn't cover error handling in this example, but that's clearly animportant piece of functionality to include when building a productionemail system. At a minimum, it would be useful to add a retrymechanism in the event that Mailgun is unavailable.

References

I found the resources below helpful for working on this project:

• Theamqp-clientdocumentation, especially theexamples were helpful for setting up with RabbitMQ.
• Theclient tutorial was helpful for getting started withcohttp. Later, I utilizedthis section of the docs to understand that I needed to usepost_form instead ofpost when interacting with Mailgun's API.

Feedback

If you ended up looking through the source code for this example, letme know how what you thought! I'm interested in adding more tutorialresources to the OCaml ecosystem, so feel free to post a PR or issuetodream-email-exampleif you have ideas about how to make these resources better.