Mad Marmot A blog about programming, ruby, rails.

Running ar_sendmail with monit

Sending email from a web application, especially blast emails to a lot of people, can take a lot of time. Generally you don't want the user to wait until all the emails have been handed off to the smtp server. You also probably don't want to tie up an entire mongrel with sending mail. The ar_mailer gem solves this problem in excellent fashion, by saving pending emails to the database and having a separate ruby daemon process periodically check the database and send emails. I recently set up one of our rails apps at work to use ar_mailer. Configuring it to use ar_mailer was incredibly easy, but it was tricky to get the ar_sendmail ruby daemon process to run under monit. On our production servers which we have hosted at Engine Yard, we want every process that our application depends on to be monitored by monit. 

The primary feature that ar_sendmail lacks to play nice with monit is the ability to leave a pid file after it starts up and to remove it when the process exits. This has already been pointed out on rubyforge as a feature request. Here is what I did to get ar_sendmail working under monit: (ar_mailer 1.3.1)

0) Find the file ar_sendmail.rb on your system and open it for editing.

For me this was at /usr/local/lib/ruby/gems/1.8/gems/ar_mailer-1.3.1/lib/action_mailer/.  Another common location is /usr/lib/ruby/gems/1.8/gems/ar_mailer-1.3.1/lib/action_mailer/

1) Create a class variable to store the pid file path in and a method to remove the pid file.

I put this just below "attr_accessor :failed_auth_count"

  @@pid_file = nil

  def self.remove_pid_file
    FileUtils.rm(@@pid_file) if @@pid_file
  end   

2) Modify the self.run method in ar_sendmail.rb to create a pid file, and to not start up if an ar_sendmail process is already running

    if options[:Daemon] then
      require 'webrick/server'
      @@pid_file = "#{options[:Chdir]}/log/ar_mailer.pid"
      if File.exists? @@pid_file
        # check to see if process is actually running
        pid = ''
        File.open(@@pid_file, 'r') {|f| pid = f.read.chomp }
        if system("ps -p #{pid} | grep #{pid}") # returns true if process is running, o.w. false
          $stderr.puts "Warning: The pid file #{@@pid_file} exists and ar_sendmail is running. Shutting down."
          exit
        else
          # not running, so remove existing pid file and continue
          self.remove_pid_file
          log "ar_sendmail is not running. Removing existing pid file and starting up..."
        end
      end
      WEBrick::Daemon.start
      File.open(@@pid_file, 'w') {|f| f.write("#{Process.pid}\n")}
    end

If the pid file already exists, this code will check to see if the process is actually running.  If it is then it will exit, otherwise it will remove the file and continue. This is useful for when the process dies ungracefully somehow (server crashes, killed with -9, .etc), in which case it will leave a pid file behind. Note that this differs from the code suggested on the rubyforge feature request, not only in that it checks the existence of the pid file, but it references options[:Chdir] instead of Dir.pwd in order to be compatible with the -c and --chdir ar_sendmail option. 

3) Modify the do_exit method in ar_sendmail.rb to remove the pid file

  def do_exit
    log "caught signal, shutting down and removing pid file"
    self.remove_pid_file
    exit
  end

4) Create the monit file /etc/monit.d/ar_sendmail.teamsport.monitrc

check process ar_sendmail_teamsport
  with pidfile /data/teamsport/current/log/ar_mailer.pid
  start program = "/usr/bin/ar_sendmail -d -e production -c /data/teamsport/current/" as uid teamsport and gid teamsport
  stop program = "/usr/local/bin/stop_ar_sendmail" as uid teamsport and gid teamsport
  if totalmem is greater than 65.0 MB for 2 cycles then restart      # eating up memory?
  if loadavg(5min) greater than 10 for 8 cycles then restart          # bad, bad, bad
  if 20 restarts within 20 cycles then timeout                        # something is wrong, call the sys-admin
  group ar_sendmail

Note that I am using a simple shell script script called stop_ar_sendmail to stop the ar_sendmail process.  ar_sendmail has signal handlers for SIGINT and SIGTERM so we should use these signals to kill it, which will cause the do_exit method to be triggered. The stop_ar_sendmail script looks like this:

#!/bin/bash
kill -2 `cat /data/teamsport/current/log/ar_mailer.pid`

Originally I tried putting the contents of this shell script in the .montrc file like this:

stop program = "/bin/kill -2 `cat /data/teamsport/current/log/ar_mailer.pid`" as uid teamsport and gid teamsport 

This however does not work, since apparently monit doesn't know what to do with the backticks. Alternatively you could use some sort of grep and kill script, such as pkill, to stop the ar_sendmail process. Ideally in the future, ar_sendmail will support some sort of stop command in the same manner as mongrel so that you could run "ar_sendmail stop" to stop it.

Once you have this all setup you can control ar_sendmail via monit. When creating the ar_sendmail.teamsport.monitrc file, make sure you change "teamsport" to the user that you want to run the process as.  Then do a "sudo monit reload" and monit should see that ar_sendmail is not running and will start it for you. To make sure everything is working correctly try "sudo monit stop ar_sendmail_teamsport" and "sudo monit start ar_sendmail_teamsport" (replacing "teamsport" with the appropriate user name).

Other than not working well with monit out of the box, the only other issue I have with ar_sendmail is the memory footprint, which is dependent to some degree on your rails app. The ar_sendmail process for my app runs at around 50 MB, and just to send mail! I assume this is due to the fact that ar_sendmail loads the rails app's environment.rb file. The environment.rb file runs the boot.rb file, which bootstraps and initializes the entire rails app. Additionally, our environment.rb has several other plugins required inside of it. I think the environment.rb file is loaded primarily just to get at the ActiveMailer smtp_settings, which is a slick way to allow for easy integration of ar_mailer with minimal changes to your existing rails app. Many people wouldn't think twice about 50 MB, but rails hosts charge quite a bit for RAM. I can definitely envision a slimmed down ar_sendmail that doesn't load the rails app's environment.rb file, but it seems almost impossible to do this without making integration with existing rails apps more difficult.