Concurrent Data Processing in Elixir

Fast, Resilient Applications with OTP, GenStage, Flow, and Broadway

by Svilen Gospodinov

Learn different ways of writing concurrent code in Elixir and increaseyour application’s performance, without sacrificing scalability orfault-tolerance. Most projects benefit from running background tasks andprocessing data concurrently, but the world of OTP and various librariescan be challenging. Which Supervisor and what strategy to use? Whatabout GenServer? Maybe you need back-pressure, but is GenStage, Flow, orBroadway a better choice? You will learn everything you need to know toanswer these questions, start building highly concurrent applications inno time, and write code that’s not only fast, but also resilient toerrors and easy to scale.

Whether you are building a high-frequency stock trading application or aconsumer web app, you need to know how to leverage concurrency to buildapplications that are fast and efficient. Elixir and the OTP offer arange of powerful tools, and this guide will show you how to choose thebest tool for each job, and use it effectively to quickly start buildinghighly concurrent applications.

Learn about Tasks, supervision trees, and the different types ofSupervisors available to you. Understand why processes and processlinking are the building blocks of concurrency in Elixir. Getcomfortable with the OTP and use the GenServer behavior to maintainprocess state for long-running jobs. Easily scale the number of runningprocesses using the Registry. Handle large volumes of data and trafficspikes with GenStage, using back-pressure to your advantage. Create yourfirst multi-stage data processing pipeline using producer, consumer, andproducer-consumer stages. Process large collections with Flow, usingMapReduce and more in parallel. Thanks to Broadway, you will see howeasy it is to integrate with popular message broker systems, or evenexisting GenStage producers.

Start building the high-performance and fault-tolerant applicationsElixir is famous for today.

What You Need

Resources

• Errata, typos, suggestions

• Source Code (zip file)

Releases:

Contents & Extracts

• Introduction
  • Who Should Read This Book?
  • About This Book
  • About the Code
  • Online Resources
• Easy Concurrency with the Task Module
  • Introducing the Task Module
  • Creating Our Playgroundexcerpt
  • Starting Tasks and Retrieving Results
  • Managing Series of Tasks
  • Linking Processes
  • Meeting the Supervisor
  • Understanding Let It Crash
  • Wrapping Up
• Long-Running Processes Using GenServerexcerpt
  • Starting with a Basic GenServer
  • GenServer Callbacks In Depth
  • Building a Job-Processing System
  • Introducing DynamicSupervisor
  • Implementing a Supervisor
  • Naming Processes Using the Registry
  • Inspecting Supervisors at Runtime
  • Wrapping Up
• Data-Processing Pipelines with GenStageexcerpt
  • Understanding Back-Pressure
  • Introducing GenStage
  • Building Your Data-Processing Pipeline
  • Adding Concurrency with ConsumerSupervisor
  • Creating Multi-Stage Data Pipelines
  • Choosing the Right Dispatcher
  • Wrapping Up
• Processing Collections with Flow
  • Analyzing Airports Data
  • Creating Flows and Reading Files
  • Performing Reduce Concurrently with Partitions
  • Using Windows and Triggers
  • Adding Flow to a GenStage Pipeline
  • Wrapping Up
• Data-Ingestion Pipelines with Broadway
  • Processing Ticketing Events
  • Broadway Callbacks In Depth
  • Batching Messages
  • Using GenStage Producers
  • Wrapping Up

Author

Related Titles:

• Designing Elixir Systems with OTP

• Functional Web Development with Elixir, OTP, and Phoenix