Posted onJan 2, 2023 • Originally published atprabinpoudel.com.np

Setup Active Job with Sidekiq in Rails

#ruby #rails

Active Job is a framework for declaring jobs and making them run on a variety of queuing backends. These jobs can be everything from regularly scheduled clean-ups, to billing charges, to mailings. Anything that can be chopped up into small units of work and run in parallel.

Active Job comes pre installed in Rails.

Now on to Sidekiq; it is one of the most widely used background job frameworks that you can implement in a Rails application. It is multi-threaded and utilizes Redis for its queuing storage.

Today we will be working on setting up Active Job with Sidekiq while also following the official Rails documentation.

Skills required to follow the tutorial

Intermediate:

Rails
Linux skills to work with commands in server where your app has been deployed

You should have

Existing Rails app
Linux server already setup to run Rails app

Step 1: Install Capistrano Gems

Add the following to your Gemfile under development group:

gem'sidekiq'

Run the following in command line:

$bundleinstall

That should install the latest version of Sidekiq gem, now let's lock the version by looking at the "Gemfile.lock", this version was what I had in my lock file, yours could be different.

gem'sidekiq','~> 7.0.0'

Step 2: Enable Sidekiq

Add the following to "config/application.rb"

# Use sidekiq for active jobsconfig.active_job.queue_adapter=:sidekiq

Step 3: Install Redis

Redis is an open source, in-memory data store used by millions of developers as a database, cache, streaming engine, and message broker. It is an in-memory key-value store known for its flexibility and performance.

Sidekiq is backed by Redis as a job management store to process thousands of jobs per second.

Official Redis Documentation has got you covered for the installation of Redis in any OS.

On MacOS via Homebrew

If you are on MacOS, you can follow instructions atInstall Redis on MacOS (Official Documentation).

Linux

You can either choose to install fromsource or fromAPT repository.

I had installed Redis from the source and these were the instructions that I needed to perform from the command line:

$wget http://download.redis.io/redis-stable.tar.gz$tarxvzf redis-stable.tar.gz$cdredis-stable$make

Copy the executables to you local usr bin folder so you can run the redis-server and cli commands from your home/user directory

sudo cpsrc/redis-server /usr/local/bin/sudo cpsrc/redis-cli /usr/local/bin/

Automatically start on machine restart

NOTE: You can also find these instructions in the official Redis documentation atGetting Started > Installing Redis more properly

Create a directory in which to store your Redis config files and your data:
```
sudo mkdir /etc/redissudo mkdir /var/redis
```
Copy the init script that you'll find in the Redis distribution under the utils directory into /etc/init.d. We suggest calling it with the name of the port where you are running this instance of Redis. For example:
```
sudo cpredis-stable/utils/redis_init_script /etc/init.d/redis_6379
```
Edit the init script
```
sudonano /etc/init.d/redis_6379
```
Make sure to modify REDISPORT according to the port you are using. Both the pid file path and the configuration file name depend on the port number. I had used the port "6379"
Copy the template configuration file you'll find in the root directory of the Redis distribution into /etc/redis/ using the port number as name, for instance:
```
sudo cpredis-stable/redis.conf /etc/redis/6379.conf
```
Create a directory inside /var/redis that will work as data and working directory for this Redis instance:
```
sudo mkdir /var/redis/6379
```
Edit the configuration file withsudo nano /etc/redis/6379.conf, making sure to perform the following changes:
- Change the port accordingly. In our example it is not needed as the default port is already 6379. It is under the section ## NETWORK ##
- Set daemonize to yes (by default it is set to no). It is under the section ## GENERAL ##.
- Set the pidfile to /var/run/redis_6379.pid (modify the port if needed). It is under the section ## GENERAL ##
- Set your preferred loglevel (notice by default). It is under the section ## GENERAL ##
- Set the logfile to /var/log/redis_6379.log ("" by default). It is under the section ## GENERAL ##
- Set the dir to /var/redis/6379 (very important step!). It is under the section ## SNAPSHOTTING ##
Finally add the new Redis init script to all the default runlevels using the following command
```
sudoupdate-rc.d redis_6379 defaults
```
You are done! Now you can try running your instance with:
```
sudo /etc/init.d/redis_6379 start
```
Test if it working correctly
- Try pinging your instance with redis-cli using the commandredis-cli ping, you should see PONG
- Do a test save withredis-cli save and check that the dump file is correctly stored into /var/redis/6379/ (you should find a file called dump.rdb).
- Check that your Redis instance is correctly logging in the log file withtail -f /var/log/redis_6379.log, you can exit withCTRL + c
- If you are on a new machine/server where you can try things without taking anything else down in the machine/server make sure that after a reboot everything is still working (redis should start automatically on machine restart)

Windows

If you are on Windows, you can follow instructions atInstall Redis on Windows (Official Documentation).

Step 4: Enable Web UI to Monitor Jobs

Add the following to your "config/routes.rb":

# config/routes.rbrequire'sidekiq/web'Myapp::Application.routes.drawdo# mount Sidekiq::Web in your Rails appmountSidekiq::Web=>"/sidekiq"end

Step 6: Accessing the Sidekiq Web UI

Run rails server:rails s
Go to "http://localhost:3000/sidekiq"

You should now see the UI similar to this:

Step 6: Add Basic Authentication to Web UI for Preventing Unauthorized Access

By default there is no authentication and anyone that knows your API URL will be able to access the sidekiq UI.

You can allow any authenticatedUser to access the Sidekiq UI by adding following configurations to your "config/routes.rb":

# config/routes.rbauthenticate:userdomountSidekiq::Web=>'/sidekiq'end

NOTE: This configuration is for theDevise gem, if you are using any other authentication gem then this configuration might be different.

You can view more options for authenticating users to restrict Sidekiq UI atAuthentication (Sidekiq Gem Github).

Example Job

Now let's look at how a job can look like in the Rails App after setting up Sidekiq.

Application Job

Your "app/jobs/application.job" should look similar to the following:

classApplicationJob<ActiveJob::Basequeue_as:default# Automatically retry jobs that encountered a deadlock# retry_on ActiveRecord::Deadlocked# Most jobs are safe to ignore if the underlying records are no longer available# discard_on ActiveJob::DeserializationErrorend

Only change from the default ApplicationJob that Rails generates by default is that we have addedqueue_as :default so we don't have to add it in all other jobs we add in our app.

If you want your job to be queued with the highest priority then in the new job you generate, you should updatequeue_as to high e.g.queue_as :high to override thequeue_as :default inside the Application Job.

Expire User Note Job

Let's say you have a job that automatically removes the status of the user after certain time e.g. in the communication app calledSlack we can add status and set expiration date and time. Your job could like the following for such feature:

classExpireUserStatusJob<ApplicationJobdefperform(user_status_id)status=UserStatus.find(user_status_id)returnifstatus.blank?||!status.expired?status.update!(description:nil,expiry_date:nil,expiry_time:nil)endend

And you could queue the job from UserStatus model by triggering the method "schedule_expiration" from the controller whenever the user status is updated:

classUserStatus<ApplicationRecordvalidates:expiry_date,presence:true,if:->{expiry_time}validates:expiry_time,presence:true,if:->{expiry_date}validates:description,presence:true,if:->{expiry_date||expiry_time}defschedule_expirationreturnifexpiry_date.blank?||expired?expiry_date_time=combine_date_time(expiry_date,expiry_time)ExpireUserStatusJob.set(wait_until:expiry_date_time).perform_later(id)endprivatedefexpired?returnifexpiry_date.blank?||expiry_date<Time.zone.todayexpiry_time.blank?||expiry_time.to_s(:time)<=Time.zone.now.to_s(:time)enddefcombine_date_time(date,time)DateTime.new(date.year,date.month,date.day,time.hour,time.min,time.sec)endend

Bonus: Setup Sidekiq in Ubuntu Server

Setup

Create a file called sidekiq.servicesudo nano /lib/systemd/system/sidekiq.service
Copy content fromExample Sidekiq systemd configurations to this file.
Update the file:
- Change the WorkingDirectory to the folder where your app is deployed e.g.WorkingDirectory=/home/deploy/my-app/current
- Update the ExecStart path. This specifies the path and command to start Sidekiq. E.g. my current ExecStart path with rbenv is:ExecStart=/home/deploy/.rbenv/shims/bundle exec sidekiq -e production.NOTE: If you are using different Ruby version manager then "/home/deploy/.rbenv/shims/bundle" will be different.
- AddExecReload=/usr/bin/kill -TSTP $MAINPID belowExecStart=....
- Remove comment from# User=deploy and update deploy to be the user you are using in the server, mine was deploy so I changed the line toUser=deploy
- Uncomment# Group=sudo to make itGroup=sudo
- Uncomment# UMask=0002 to make itUMask=0002
- UpdateWantedBy=multi-user.target toWantedBy=default.target
- Save the file

Run Sidekiq

You can enable the Sidekiq service withsudo systemctl daemon-reload

You can start the Sidekiq service withsudo systemctl start sidekiq

Here are some useful commands for changing status of Sidekiq:

"sudo systemctl {start,stop,status,restart} sidekiq" e.g.sudo systemctl status sidekiq

You can also use "sudo service sidekiq {start,stop,status,restart}" to perform commands e.g.sudo service sidekiq status

Sidekiq with Capistrano

If you are using capistrano for deployment, you can refer to the section "Bonus 2: Sidekiq for background jobs" to configure Sidekiq with Capistrano atDeploy API only Rails App with Capistrano.