litany against fear

coding with spice ¤ by nick quaranto

Riding Rails at the New York Times Notes

published 18 Nov 2008

Part of my series of notes from the Professional Ruby Conference. See them all here.

This talk is from Ben Koski from the NYT’s Interactive News Technology team. The team is a recent addition to the company…6 backend, 3 frontend people. They’re a part of the newsroom…directly working with reporters! Journalistic instead of business.

Big difference between journalism on the web vs. of the web. On: copy paste from the newspaper. Of: connected multimedia, assets, and more.

Plenty of Rails apps living at the Times. Big ones like the election tracker to smaller ones like a Guantanamo detainee library. Each application should tell a story…not just a data dump on the web. Showing off the other various apps that they’ve done. Dynamic backends, internal research apps, and even more static include generators.

Going over the election results from the AP feed and the architecture they used to tackle Election night this year. Feed had live results, county-by-county. They had to figure out how to split up 153 different files every 3-5 minutes that were comma-delineated. They used this for the primaries…but it wasn’t going to cut it for the general election.

AP schema feed just wasn’t rails friendly. (Duh.) FTP is slow, long updates, long rollbacks because of massive amounts of data. Interesting way how they handled updating the production tables from the AP feed by adding in staging tables to ‘buffer’ changes. Some awesome stats about how their architecture handled the lode.

Ben is now going over some what they used to speed up the app. Second highest day of traffic ever in Times history. How did they scale Rails? They didn’t! Well, wait a minute. They used Rails to generate the HTML and uploaded that up to /www. Works well for high traffic since there’s no need for dynamic content/interaction on the site.

More scaling static content techniques: Used memcache and load-balancer caching. Pulled AJAX requests from a CDN, reduced how big the page was, and grouped flash content into one file.

Strengths:

Rapid start, raw data to app in hours
Tweak-friendly. Can easily change stuff because of Capistrano.
Ability to drop down to SQL in AR is essential.
Web code works well on command line. (In other words, PHP can’t do that.)

Weaknesses:

NYTimes is mostly a Sun shop. They use EC2 for most stuff.
Deployment. What? I missed this.
Admin interface isn’t there out of the box like Django.