Michael de Silva's Blog

Software Engineer. Rubyist and Roboticist.

Michael de Silva's Blog

Software Engineer. Rubyist and Roboticist.

HOWTO Hunt (Resque) Daemons with Bluepill

I've blogged on using God for process monitoring in the past and I really wish I'd gone with Bluepill instead.

It's DSL is extremely compact and throws much of the 'boilerplate' that God requires; the CLI tool also provides far better feedback and bluepill log makes life so much easier rather than having to tail -f some log file.

RAILS_ROOT  = ENV['RAILS_ROOT'] || "/home/appuser/sites/foo"
RAILS_ENV = 'production'

Bluepill.application("app_name", :log_file => "/home/appuser/sites/foo/log/resque_workers.log") do |app|
  app.uid = app.gid = "appuser"  
  5.times do |i|
    app.process("resque_worker_#{i}") do |process|
      process.working_dir = RAILS_ROOT
      process.group = "resque"
      process.pid_file = "/home/appuser/sites/foo/tmp/resque_worker_#{i}.pid"
      process.start_command = "/usr/bin/env VERBOSE=true RAILS_ENV=#{RAILS_ENV} QUEUE=* rake resque:work --trace"
      process.stop_command = "kill -QUIT {{PID}}"
      process.monitor_children do |c|
        c.stop_command = "kill -USR1 {{PID}}"
        c.checks :mem_usage, :every => 30.seconds, :below => 100.megabytes, :fires => :stop
      end
      process.daemonize = true
      process.start_grace_time = 3.seconds
      process.stop_grace_time = 5.seconds
      process.restart_grace_time = 8.seconds

      process.checks :mem_usage, :below => 350.megabytes, :every => 1.minute, :times => 3
    end
  end
end

Don't forget to throw this in as an initialiser as well:

# add as an initializer
Resque.after_fork do |job|
  ActiveRecord::Base.establish_connection
end

It may not hurt to add the resque-ensure-connected gem as well to your Gemfile. Just go with bluepill load /home/appuser/sites/foo/config/resque.pill to whip those daemons into shape!

You may also want to monitor Redis and simply create an upstart script to launch bluepill.

comments powered by Disqus