Capistrano Tip to Avoid Disk Intensive Removal of Files
At TST Media we host our Rails app at Engine Yard on four slices which all utilize the same shared disk via gfs. Anytime there is any hard core disk activity our sites slow down to a crawl. This makes removing files a bit tricky, such as when we want to empty our cache files or when Capistrano removes a release at the end of a deploy. The strategy we have come up with to handle this is to move the files to a specific directory we call the "caches_to_remove" directory, and use a cron task to empty this directory at night when most of our users are asleep. Moving files is extremely fast and not disk intensive, as long as the source and destination is on the same disk of course.
The shell script that the cron runs nightly is simple:
# remove_cache_dirs.sh
rm -rf /data/tst/caches_to_remove
mkdir /data/tst/caches_to_remove
We have a capistrano task to empty the cache files, which simply moves the cache directory into the caches_to_remove directory.
set :remove_files_dir, '/data/tst/caches_to_remove/'
namespace :cache do
desc "Delete all cache files on disk."
task :empty, :roles => :web do
sudo "mv /data/tst/cache #{remove_files_dir} && mkdir /data/tst/cache"
end
end
We also want a deploy to do a mv instead of a rm -rf on the release to be "removed". Our application has quite a few files and some 33,000 lines of code not counting all the plugins, gems and Rails itself which are vendored, so doing a rm -rf at the end of deploy considerably slows our app down for several minutes. So we overrode the built-in capistrano deploy:cleanup task to accomplish this.
namespace(:deploy) do
# overriding cleanup task, changing it from a rm -rf to a mv
desc <<-DESC
Clean up old releases. By default, the last 5 releases are kept on each \
server (though you can change this with the keep_releases variable). All \
other deployed revisions are removed from the servers. By default, this \
will use sudo to clean up the old releases, but if sudo is not available \
for your environment, set the :use_sudo variable to false instead.
DESC
task :cleanup, :except => { :no_release => true } do
count = fetch(:keep_releases, 5).to_i
if count >= releases.length
logger.important "no old releases to clean up"
else
logger.info "keeping #{count} of #{releases.length} deployed releases"
# COMMENT OUT THIS CODE
# directories = (releases - releases.last(count)).map { |release|
# File.join(releases_path, release) }.join(" ")
# invoke_command "rm -rf #{directories}", :via => run_method
# ADD THIS CODE
directories = (releases - releases.last(count)).each do |release|
directory = File.join(releases_path, release)
invoke_command "mv #{directory} #{remove_files_dir}", :via => run_method
end
end
end
end
And the world is a happier place!
