What's That Noise?! [Ian Kallen's Weblog]

« Cloning VMware Machi... | Main | Business Models in... »

20090521 Thursday May 21, 2009

Configuring Web Clouds with Chef

I'm not generally passionate about network and system operations, I prefer to focus my attention and creativity on system and software architectures. However, infrastructure provisioning, application deployment, monitoring and maintenance are facts of life for online services. When those basic functions aren't functioning well, then I get passionate about them. When service continuity is impacted and operations staff are overworked, it really bothers me; it tells me that I or other developers I'm working with are doing a poor job of delivering resilient software. I've had many conversations with folks who've accepted as a given that development teams and operations teams have friction between them; some even suggest that they should. After all, so goes that line of thinking, the developers are graded on how rapidly they implement features and fix bugs whereas the operators are graded on service availability and performance. Well, you can sell that all you want but I won't buy it.

In my view, developers need to deliver software that can be operated smoothly and operators need to provide feedback on how smoothly the software is operating; dev and ops must collaborate. I accept as a given that developers

Conversely, I accept as a given that operators I don't want to oversimplify, there's more to the obligations that dev and ops have to each other in order to collaborate effectively. What I've noticed though is that a lot of operators might be skilled with configuring specific server infrastructure or performing OS analysis but configuration management and automation requires using really good tools that they lack. I've seen situations where the available tools are perceived as too complicated and so tools are developed that usually consist of a lot of specialized shell scripts (or perhaps it's just plain old NIH). Cfengine is a good start but the reports I have is that it's difficult to work with and, if you're not very careful, may automatically manage to misconfigure your systems. Puppet was developed to be a more powerful system for configuration management but the feedback I've seen on it is adding new functionality is hard; it has its own configuration language and when you want to extend it you have to deal with a lot of complicated mechanisms. Chef was developed to answer that frustration; by making the configuration language a DSL on top of an already widely used scripting language (ruby), the chef code base provides an easier way to extend it and the chef codebase itself is reportedly an order of magnitude smaller and simpler than puppet's (cavaet: I generally distrust SLoC metrics but just sayin').

So I've been giving Chef a test-drive for this infrastructure-on-EC2 management project that's been cooking. The system implemented the following use cases:

  1. Launch web app servers on EC2 with Apache, Passenger, RoR (+other gems) and overlay a set of rails apps out of git
  2. Launch a pair of reverse proxies (with ha-proxy) in front of the app servers - and reconfigure them when the set of app servers is expanded or contracted
  3. Configure the proxy for failover with heartbeat
  4. Add new rails apps to the set of app servers
  5. Updating/rolling back rails apps
The system is enabled through a combination of the EC2 API (via RightAWS) and Chef's REST API as well as using chef-deploy (think: Capistrano run by a system provisioning agent) to augment Chef's functionality. So far, it seems to be working great!

There's a lot of energy in the Chef community (check out Casserole), combined with monitoring, log management and cloud technologies, I think there's a lot of IT streamlining ahead. Perhaps the old days of labor and communication intensive operations will give way to a new era of autonomic computing. I'll post further about some of the mechanics of working with ruby, rails, chef, EC2, chef-deploy and other tools in the weeks ahead (particularly now that EC2 has native load balancing, monitoring and auto-scaling capabilities). I'll also talk a bit about this stuff at a Velocity BoF. If you're thinking about attending Velocity, O'Reilly is offering 30% off to the first 30 people to register today with the code vel09d30 today (no I'm not getting any kinduva kickback from O'Reilly). And you can catch Infrastructure in the Cloud Era with Adam Jacob (Opscode), Ezra Zygmuntowicz (EngineYard) to learn more about Chef and cloud management.


( May 21 2009, 12:30:07 PM PDT ) Permalink
Comments [2]


Great post, Ian! I smell a whole new world shaping up, with the dev/ops mingling, and automated infrastructure work being done, like with Chef. Until Puppet/Chef/Capistrano, tools of those kind have pretty much sucked for years. (I still have a warm place in my heart for SystemImager/Configurator) I expect OpsCode to own this game. :)

Posted by Allspaw on May 21, 2009 at 02:29 PM PDT #

In addition to the talk at Velocity, we'll basically be packing up all of Opscode and transferring it to a table at the conference. :) We'll be super easy to find.

Posted by Adam Jacob on May 22, 2009 at 09:45 AM PDT #

Post a Comment:

Comments are closed for this entry.