Building Spokane Tech: Part 7
Building Spokane Tech: Part 7
Welcome to part 7 of the "Building Spokane Tech" series! In this article, we'll explore integrating Celery for scheduling tasks, executing work asynchronously, and monitoring task statuses.
See the live site at:https://www.spokanetech.org
See the latest code on:github
Django Tasks and Celery
In a Django project, tasks.py is typically used to define background tasks that are executed asynchronously. This is common in applications where long-running or scheduled operations (such as sending emails, processing files, or performing API calls) should not block the main request-response cycle.
Celery is a distributed task queue that integrates well with Django to handle background jobs asynchronously. It allows you to schedule and execute tasks efficiently.
Creating Tasks
In our service we'll use tasks defined in the tasks.py file to perform data ingestion from eventbrite and meetup. We have a few different tasks defined, namely a 'parent' launcher task, and a separate 'child' task for performing the bulk of the work.
Parent Tasks
The parent task approach allows us to enable some basic control over what work gets queued for execution. For example, to run event ingestion for eventbrite groups we can first get a list of groups from our database, then launch individual jobs for each. Each job can succeed or fail independently of other jobs, and we'll have separate outputs and results.
Here's an example of a parent task:
@shared_task(time_limit=900,max_retries=0,name="web.launch_eventbrite_event_ingestion")deflaunch_eventbrite_event_ingestion():"""parent task for ingesting future events for tech groups on Eventbrite"""tech_group_list=TechGroup.objects.filter(enabled=True,platform__name="Eventbrite")forgroupintech_group_list:job=ingest_future_eventbrite_events.s(group.pk)job.apply_async()In the code above, the
is a queryset of groups that are enabled and on the eventbrite platform. The ```ingest_future_eventbrite_events``` is the task to be executed, and ```job.apply_async()``` is the code to put the job on the queue for a background worker to consume.## **Scheduling Tasks*****Django Celery Beat***Django Celery Beat is an extension for Celery that enables scheduling periodic tasks in a cron-like manner using the Django Admin panel. It allows you to manage scheduled tasks dynamically without modifying code.In our service we use this to automatically run the parent tasks at regularly scheduled intervals, for example, every eight hours.## **Monitoring Tasks*****Celery Flower***Celery Flower is a real-time web-based monitoring tool for Celery. It provides insights into task execution, workers, queues, and performance. Key features include:Dashboard Overview- Displays task statistics (Succeeded, Failed, Pending, etc.)- Shows real-time execution of Celery tasksWorkers Monitoring- Lists all active workers- Shows worker status, uptime, and queue loadTask Monitoring- Provides details of each task (Arguments, Status, Runtime, etc.)- Allows you to retry failed tasksQueues & Broker Inspection- Displays Celery queues and their task load- Shows messages in Redis or RabbitMQ queuesTop comments(0)
For further actions, you may consider blocking this person and/orreporting abuse
