Web giants like Google and Amazon have long used software that automatically configures the vast collection of machines driving their online services. But these tools were never available to the outside world. In 2005, Luke Kanies set out to provide Google-like IT automation for the rest of us, founding an open source project he called Puppet.
“Google does [things] differently, and in many cases, they do it better. Amazon is the same way. But what’s really frustrating is that no one else can use their software,” says Kanies. “I wanted to build a tool that would help other companies solve the same problem.”
He also built a commercial company around his project — the Portland-based Puppet Labs — and two years later, he had some competition from the man once known as Amazon’s “Master of Disaster.” Jesse Robbins came by the name not because he broke things, but because he fixed them. He oversaw the operation of all Amazon websites. His new venture, Opscode, grew up around an open source project called Chef.
“The idea was that IT automation would be the basis for all really scalable startups — that it would become the secret sauce that everyone had,” Robbins says.
Jason Stowe compares “DevOps” tools like Chef and Puppet to the Apple iPod. DevOps has exactly nothing to do with digital music or consumer gadgetry of any kind. Chef and Puppet automatically set up and tweak the operating systems and programs that run in massive data centers and the new-age “cloud” services, designed to replace massive data centers. But Stowe argues that much like the iPod finally brought digital music to the average Joe, Chef and Puppet have at long last taken IT automation mainstream.
“There were music players before the iPod, but the iPod was the first one to get it right,” says Stowe, the CEO of Cycle Computing, a Greenwich, Connecticut-based startup that uses Chef to manage the software underpinning the online “supercomputing” service it offers to big businesses and academic outfits. “Before this, there were ways of configuring servers and managing them, but DevOps has gotten it right.”
Both Chef and Puppet are known for automating the configuration of “cloud” services like the one run by Cycle Computing, but they can also streamline IT tasks across a company’s internal servers. “It’s moving from Web 2.0, Facebook- and Zynga-type companies into the enterprise,” says Jay Lyman, an analyst with research outfit The 451 Group. “It’s a dramatic change, but I think many businesses see there’s a need for this change.”
This week, Opscode took another step towards Joe Business, publicly releasing a new set of Chef scripts — or “recipes” — that automate the configuration of Windows machines. Traditionally, Chef and Puppet are used to manage Linux — the OS that drives most cutting edge startups — but both now have agents that run on Windows as well. For Lyman, this is a must for the move into big business. “There’s a real need for DevOps on the Windows side,” he says. “Most enterprises — and services providers, for that matter — mix Windows machines with Linux machines.”
Dev Mates With Ops
Jason Stowe’s Cycle Computing is already using Chef to configure both Linux and Windows underneath its online service. Known as Cycle Cloud, the service is a way for big businesses and research outfits to run supercomputing applications without actually building their own supercomputers. “We call it utility supercomputing,” says Stowe. “Clients use our cloud to run massive calculations.” These clients include JP Morgan and other big financial institutions, pharmaceutical companies such as Pfizer, DNA sequencing outfits, and even video game makers.
The service runs atop Amazon’s Elastic Compute Cloud (EC2). Amazon provides the virtual machines, and Cycle runs its software atop these VMs, spinning up new server instances as needed. In the past, Stowe and company would simply apply stock software images to each machine they spun up, but with Chef, they can build and run scripts that not only customize each image as its deployed, but automatically update running servers when the time comes.
“Baking the software you’re going to use into Amazon images seems like a good idea, but it’s actually a very bad idea,” Stowe says. “Configuring machines, reacting to errors, being able to deal with the ranging stampeded of servers that we provision is very very difficult, and Chef gives us tools for dealing with all this.”
Cycle is constantly refreshing the software running across all those virtual machines, and in many cases, different customers require different software. Using Chef recipes — which are coded in a language based on Ruby — the company can makes changes without manually updating each machine.
“We can deploy additional software applications on virtual machine instances without dealing with the overhead of doing everything manually,” Stowe explains. “We can do it with code — recipes that define how various applications and libraries are deployed and configured.” According to Stowe, creating and deploying a new software image now takes minutes or hours rather than hours or weeks.
They call this technique DevOps because it applies traditional programming techniques to system administration tasks. “It’s just treating IT operations as a software development problem,” says Stowe. But Puppet Labs’ Kanies goes further. “Developers are always adding new generations of technology, always adopting the best, newest tools that they can so they can build the best products,” he says. “We’ve taken that same attitude.”
Sexy IT Admin
Puppet differs from Chef in that it doesn’t use Ruby, a kind of meta-programming language that has gained popularity among web developers in recent years. Puppet uses its own scripting language that Kanies describes as much simpler. Naturally, each company pitches its own setup as more proficient than the other, but both outfits target more than just seasoned programmers.
Like Puppet Labs, Opscode publishes recipes that can be reused by non-coders. “I can definitely see, in larger deployments, having people that are not writing recipes but just using Chef to manage machines,” says Stowe. Opscode is hoping to reach such sysadmins with its new collection of Windows recipes.
DevOps is yet another example of a technology that was spawned on the web but is now spreading elsewhere. Stowe’s Cycle Computing is a microcosm of this shift. The company uses Chef to configure its own internal servers — the systems that run its website and the rest of its business — and Stowe envisions using the platform to configure the company’s desktop machines.
Opscode founder Jessie Robbins, now the company’s community manager, says that like many other web-influenced enterprise startups, Opscode often enters businesses through the backdoor. The company offers Chef as an online service as well as a server platform you can install on your own.
“We have this open-source proliferation model — so lots of people use it — and then these people, developers and systems engineers, can sign up for our hosted service on our website,” he says. “Then, when they’re running 100 or 200 nodes, we suddenly have buying conversation with an enterprise company. And it all began with that one small user inside the organization.”
But at the same time, he acknowledges that old school sysadmins will require some convincing — that many will respond to his company’s pitch with the usual skepticism. “Sysamdins have been fed a lot of shit by a lot of different vendors for years — while simultaneously being made to feel bad about the job they do. So they’re sort of inoculated to new pitches,” he says. “They’re tired of shit that doesn’t work.”
The irony is that Opscode is an outfit founded by sysadmins, including Robbins himself. “I wanted to make IT admin sexy. Because it is,” he says. And for many, including Cycle Computing, tools like Chef are what the average sysadmin needs most. “When you’re running a service like we do, taking advantage of DevOps is a non-brainer,” says Jason Stowe. “It’s just common sense.”
How does Robbins respond to the naysayers? “I tell them they can try it on the web,” he says, “for free.”