Tuesday, November 5, 2013

murder - twitter's bt deploy system

Twitter – Murder Bittorrent Deploy System from Larry Gadea on Vimeo.

the murder story

Slow deploys

By sitting beside a particularly vocal Release Engineer, I received first-hand experience of the frustration caused by slow deploys. We needed a way to dramatically speed things up. I thought of some quick hacks to get this fixed: maybe replicate the git repo or maybe shard it so everyone isn’t hitting the same thing at once. Most of these quasi-centralized solutions will still require re-replicating or re-sharding again in the near future though (especially at our growth).

It was time for something completely different, something decentralized, something more like…BitTorrent…running inside of our datacenter to quickly copy files around. Using the file-sharing protocol, we launched a side-project called Murder and after a few days (and especially nights) of nervous full-site tinkering, it turned a 40 minute deploy process into one that lasted just 12 seconds!

To the rescue

Murder (which by the way is the name for a flock of crows) is a combination of scripts written in Python and Ruby to easily deploy large binaries throughout your company’s datacenter(s). It takes advantage of the fact that the environment in a datacenter is somewhat different from regular internet connections: low-latency access to servers, high bandwidth, no NAT/Firewall issues, no ISP traffic shaping, only trusted peers, etc. This let us come up with a list of optimizations on top of BitTornado to make BitTorrent not only reasonable, but also effective on our internal network.

Since at the time we used Capistrano for signaling our servers to perform tasks, Murder also includes a Capistrano deploy strategy to make it easy for existing users of Capistrano to convert their file distribution to be decentralized. The final component is the work Matt Freels (@mf) did in bundling everything into an easy to install ruby gem. This further helped get Murder to be usable for more deploy tasks at Twitter.

the murder repo

from Lizard’s Ghost http://ghost-lizard.rhcloud.com/murder-twitters-bt-deploy-system/


murder - twitter's bt deploy system

No comments:

Post a Comment