Last Tuesday, I gave a talk at Austin on Rails about background processing options available in Ruby on Rails. The talk was divided into 3 sections.
The first, was ad hoc background processing solutions based on built in Rails tools or simple gems. These strategies tends to be good for background processing that need to run on a fixed schedule. These strategies not persistent and are memory intensive.
The second category I talked about was simple queues. There are several gems and plugins that fit this category. The jobs are primarily persisted in the database. These solutions are relatively simple to set up, typically persist to a database table or an external queue. These are great for user generated jobs that need to run asynchronously from the typical request/response cycle. I covered some of the more popular options: BackgrounDRb, Background Job, Delayed Job, Working/Starling. We have had success with Background Job and Delayed Job where I work, but the other offer unique advantages and are worth investigation. Workling not a complete background solution by itself, but an interface to background job creation and management. In addition to Workling, you also need a runner to handle the actual running/queuing of the jobs. Spawn and Starling are popular options here. Workling allows you to switch out the underlying system that manages the queue as your needs change.
The final section of the talk focused on more advanced queues. These are all external queues completely separate from your Rails application. I mentioned Amazon SQS, RabbitMQ, Kestrel as good options to explore if you are looking for high traffic background processing.
Very useful presentation, thank!. A video would have been even better to hear you talk about when to use what (ala last slide). Thanks again.
Thanks for the presentation! Nice to get some kind of overview. I second Kuna1 on more information on “When to use what”.
Thanks for the comments. I’m glad the presentation was useful. A new background processing option worth evaluating is Resque. It is a Redis based queuing system with a Sintra UI for managing jobs. Resque is built by the guys at Github. Source here: http://github.com/defunkt/resque and a good blog post to start with here http://github.com/blog/542-introducing-resque. It seems like an excellent choice if your application does a lot of background processing.