litany against fear
Rock Solid Ruby Deployments Notes
Part of my series of notes from the Professional Ruby Conference. See them all here.
This talk is from Philippe Hanrigou (http://ph7spot.com)
Usually deployment is a bit issue/doubt with management and many others when choosing to go with Ruby. As Rubyists we may not be 100% familiar with all of the other tools that help web developers out, like lsof, strace, and gdb.
The secret to troubleshooting is knowing your stack! You just need to know the basic, fundamental principles that are operating in your stack. You also should know the tools to help out with poking around the stack. There’s some ways to look at the current Ruby interpreter thread: Kernel#caller, Kernel#raise. Thread#raise too, but not so much.
The next layer: Mongrel. When mongrel starts, there’s one thread (or cog, visually). The first thing Mongrel does when a request comes in is handle the request and spin off a new thread. There’s a Rails handler to marshal the request and attempt to get a hold of the Rails lock. From there, Rails takes over and talks to the database, etc…but that might take a long time. Other requests are going to spin new threads in Mongrel and sleep until they can obtain the lock on Rails. Phillippe created caller_for_all_threads, which gets a hash of all traces from all threads in Mongrel. You can send a signal to Mongrel and get a dump of where all of the Mongrels are. This will be in the next version of Ruby Enterprise Edition too.
DTrace! A dynamic tracing tool that can trace the entire fucking stack. All of the other tools like GDB and strace live in the user space, not the kernel space. DTrace runs within the kernel and it can see everything: application, interpreter, and system. Using one tool, you can get information from your DB, memcache, Rails, Apache. It’s not a static view tool: it can view multiple parts of your system through probes that can be tracked. Since probes are off by default, adding in probes is very cheap.
The main problem is that dtrace isn’t ready yet for Linux. Paul Fox (http://www.crisp.demon.co.uk) is working on it, but it’s not production-ready.
DTrace is really good at inspecting garbage collection. Can see when it starts, stops, when exceptions are thrown, and even more. The tool lets you drill down and find the problem easier. It helps you pinpoint the actual performance bottleneck in your infrastructure. You can even see which ruby methods are slowing a particular group of Rails actions. Obviously, database methods will be slow. Can also figure out your memcache hit ratio, slow db queries, network i/o, and more. There are plenty of ways to hook into DTrace from Ruby: tracer, ruby-dtrace. Can allow you to hook into Rails very easily. Once again, know your stack!