The shell script that the cron runs nightly is simple:

# remove_cache_dirs.sh
rm -rf /data/tst/caches_to_remove
mkdir /data/tst/caches_to_remove

We have a capistrano task to empty the cache files, which simply moves the cache directory into the caches_to_remove directory.

set :remove_files_dir, '/data/tst/caches_to_remove/'
namespace :cache do
  desc "Delete all cache files on disk."
  task :empty, :roles => :web do
    sudo "mv /data/tst/cache #{remove_files_dir} && mkdir /data/tst/cache"
  end 
end

We also want a deploy to do a mv instead of a rm -rf on the release to be "removed". Our application has quite a few files and some 33,000 lines of code not counting all the plugins, gems and Rails itself which are vendored, so doing a rm -rf at the end of deploy considerably slows our app down for several minutes. So we overrode the built-in capistrano deploy:cleanup task to accomplish this.

namespace(:deploy) do
  # overriding cleanup task, changing it from a rm -rf to a mv
  desc <<-DESC
    Clean up old releases. By default, the last 5 releases are kept on each \
    server (though you can change this with the keep_releases variable). All \
    other deployed revisions are removed from the servers. By default, this \
    will use sudo to clean up the old releases, but if sudo is not available \
    for your environment, set the :use_sudo variable to false instead.
  DESC
  task :cleanup, :except => { :no_release => true } do
    count = fetch(:keep_releases, 5).to_i
    if count >= releases.length
      logger.important "no old releases to clean up"
    else
      logger.info "keeping #{count} of #{releases.length} deployed releases"

      # COMMENT OUT THIS CODE
      # directories = (releases - releases.last(count)).map { |release|
      #   File.join(releases_path, release) }.join(" ")
      # invoke_command "rm -rf #{directories}", :via => run_method
     
      # ADD THIS CODE
      directories = (releases - releases.last(count)).each do |release|
        directory = File.join(releases_path, release)
        invoke_command "mv #{directory} #{remove_files_dir}", :via => run_method
      end
    end
  end
end

And the world is a happier place!

To accomplish this rolling restart with our setup we have to stop nginx on the slice that is down. This prevents the load balancer from sending requests to the slice that is down. If we leave nginx up and only stop the mongrels then requests will still be routed to this slice and will hang in a similar manner as if we had restarted all the mongrels at once. We put together this capistrano task:
     
namespace :mongrel do
  desc <<-DESC
  Rolling restart, 1 server at a time.
  DESC
  task :rolling_restart do
    find_servers(:roles => :app).each do |server|
      ENV['HOSTS'] = "#{server.host}:#{server.port}"
      nginx.stop
      puts "Sleeping 10 seconds to wait for mongrels to finish."
      sleep 10
      mongrel.restart
      puts "Sleeping 30 seconds to wait for mongrels to start up."
      sleep 30
      nginx.start
    end
  end
end

This task iterates over each server/slice and stops nginx, waits for 10 seconds to let the mongrels finish what they are doing, restarts the mongrels, waits 30 seconds for the mongrels to boot up, and then starts nginx up again. This capistrano task assumes the existence of nginx.stop, nginx.start, and mongrel.restart tasks.

With this mongrel:rolling_restart task in place, we then defined a deploy:rolling task like this:

desc <<-DESC
  A deploy without migrations where the mongrels restarted in a rolling manner.
DESC
task :rolling do
  update
  mongrel.rolling_restart
end

When using this deploy:rolling task our site remains up and responsive during the entire deploy. This approach is useful for small bug fix roll outs where there are no migrations that need to be ran. There is a short window of time in which some of your servers will be out-of-date. For example you may see issues if your bug fix includes changes to a view file and a controller, and say a user hits a mongrel and is served the new view and then makes a post to an out-of-date mongrel with the new controller. However this is usually preferred to forcing all of your users to wait 30 seconds while all the mongrels restart. I would rather impact a very small percentage of our users than 100% of our users.