Deploying rails application with passenger

For the last few months, we have been having tons of problems with our rails deployment on lighttpd/fastcgi. One ruby process would ended up consuming all of the machine's memory and would then crash the machine. The only way to fix the issue was to use the excellent watchdog program monit to monitor memory usage and kill lighttpd when memory is too high.

This works ok if you only have 1 application running on the web server. However, if you have more than 1 app hosted on the same server, all applications will restart when lighttpd restarts. In a sense, 1 misbehaving app would cause the rest of the apps to suffer.

So I looked at mongrel, and the situation isn't much better. In fact, it is recommended that mongrel be used with either monit or god (ruby version of the watchdog program) to monitor individual mongrel instances and kill when memory usage is too high. The problem with this approach is that the watchdog program now has to monitor multiple mongrel instances and killing of an individual mongrel may leave orphan processes behind.

So my only other choice was to try out mod_rails. I was reluctant to do this since the mod_rails software has only been around for a few months. But seeing that there are really no better alternatives, I figure it's worth a shot.

After testing in a controlled environment for a couple of weeks, I deployed it in production. Within a couple of hours during times of heavy traffic, apache crashed with an out of memory error. It turns out that mod_rails were spawning way too many rails instances without recycling them properly - and it all has to do with the "PassengerPoolIdleTime" setting.

You see, mod_rails is suppose to recycle individual ruby instances when it idles for a specific period of time, the default is 5 minutes. On a site with constant traffic, it means that the processes will never time out. So these ruby instances just sits there and grow to no end, and consumes all of the system's memory at the end.

Now comes the fun part - just what is the ideal setting for PassengerPoolIdleTime? In order to conserve memory and avoid crashing the server, I have set this to 30 seconds. I realized that I am incurring a penalty for frequent re-spawning of rails instances, but as it turns out, mod_rails caches the entire rails app in memory so spawning is relatively fast.

The next version of mod_rails come with a new memory limit setting, which will further increase the stability of rails deployment. Can't wait for it to be released.