Discover PerformanceHP Software's community for IT leaders // July 2014
How to lead a DevOps transformation
Gene Kim writes about Gary Gruver’s innovative leadership of the HP LaserJet Firmware team.
By Gene Kim
This is one of the most startling stories of DevOps-style transformation I’ve ever seen. It improved the productivity of developers by two or three times, but it wasn’t for software supporting a website—it was for the firmware software that supported the enterprise HP LaserJet family of printers.
The 2006 project was led by Gary Gruver as director of LaserJet Firmware for Hewlett-Packard. (He is now VP of Quality Engineering, Release, and IT Operations at Macy’s.) Gary wrote about this transformation in depth in his book, A Practical Approach to Large-Scale Agile Development, and he’ll be at the DevOps Enterprise conference in San Francisco, Oct. 21–23, which I’m cosponsoring.
Gary describes this transformation as fundamentally DevOps, even though firmware isn’t delivered as a service by IT operations. Gary’s story shows us that DevOps is not just for unicorns (the "exceptional" companies like Google, Amazon, Netflix, Etsy, and Twitter)—DevOps is for any value stream where development, testing, and operations must work together to achieve organizational goals.
The business problem
Gary’s group was responsible for the firmware code that ran on all enterprise LaserJet products. Like the consumer printer market, the enterprise market was incredibly competitive, with new and innovative offerings showing up nearly every month.
The firmware group was having tremendous difficulty keeping up with demand for new, innovative features, despite having somewhere between 400 and 800 developers supporting more than 10 million lines of code. They were completing two software releases per year, with the majority of their time spent porting code to new products. Only about 5 percent of their time was spent creating or supporting new features.
The net result: Gary’s group was the bottleneck for the entire business line, and marketing finally gave up asking for new things. "When that happens," Gary says, "you know you’re in a bad situation, and you certainly can’t win in a competitive marketplace like we were in."
Slow feedback loops kill
Here are some relevant statistics of Gary’s team before the transformation that may seem all too familiar to other software projects:
- 5 percent of development time spent writing new features
- 15 to 20 percent of development time spent integrating code into mainline
- When a developer checked code in, it took about six weeks to learn whether the code worked
- It took one week to determine whether it integrated successfully
- It took another day to reach the "integration build"
- Another day was spent running acceptance tests
- It took six weeks to complete full (manual) integration testing
"We were constantly being interrupted by new defects that were introduced into the code base up to six weeks before," Gary says. "How can you expect a developer to learn anything under these conditions—so that we could prevent it from happening in the future? You can’t! The only thing that happens is management yells at someone for something they did six weeks ago."
That is a profound observation: no one can learn anything from slow feedback. Learning can occur only if we can see the cause-and-effect linkage between the work and the outcomes.
Implementing continuous integration, and a dramatic architecture change
1. Architecture changes: Gary moved all developers onto a common code base. They eliminated separate branches for all products, putting all LaserJet models into trunk.
2. Automated testing: To support self-testing builds, they built a set of automated unit, acceptance, and integration tests, which would continually run against trunk. "Without automated testing," Gary says, "continuous integration is the fastest way to get a big pile of junk that never compiles or runs correctly."
Automation created fast feedback. A developer would know within hours whether the code they committed worked. The before-and-after:
- Build cycle time: 1 week -> 3 hours (10–15 builds per day)
- Commits: 1 code commit/day -> 100 commits/day
- Regression test cycle time: 6 weeks -> 24 hours
3. Stopping the line: Once they had an automated test suite, Gary’s team created a culture where all work stopped when someone broke the build or the test suite. To help developers get the deployment pipeline running again, they created a chat room to enable quick and easy communication.
Here’s where Gary’s team ended up:
- 75K–100K lines of code changes every day
- 100–150 code commits per day
- 5 percent to 40 percent of time spent writing new features
Gary told me that the key to a DevOps breakthrough is understanding that code isn’t done when it runs in a dev environment—and the Agile value of always keeping the code base stable and close to releasable.
"To me, DevOps in the enterprise is all about driving that discipline into large organizations with different groups and applications that have to work together," he says. It’s about everyone focusing on the end-user experience. "At its core, it is about defining and driving a more robust enterprise-level definition of done."
This is a shortened version of a post on my ITRevolution blog. For the full story, read Gary’s book A Practical Approach to Large-Scale Agile Development. You can find his blog at PracticalLargeScaleAgile.com. Gary Gruver will also be speaking at the DevOps Enterprise conference in San Francisco, Oct. 21–23, 2014. I hope to see you there!
Keep up with IT trends and leadership insights by subscribing to Discover Performance.
HP’s CEO and her software management team discuss how enterprise IT can stay on top of the latest technology influencers. (On demand)
Join thousands of IT execs, engineers, and solution experts to explore IT trends, strategies, and best practices. (Barcelona,
HP Software’s Paul Muller hosts a weekly video digging into the hottest IT issues. Check out the latest episodes.
Dev Center 20/20
How will we organize development centers for the apps that will power our enterprises?
Introduction to Enterprise 20/20
What will a successful enterprise look like in the future?
Challenges and opportunities for the CIO of the future.
Welcome to a new reality of split-second decisions and marketing by the numbers.
IT Operations 20/20
How can you achieve the data center of the future?
What the workforce of 2020 can expect from IT, and what IT can expect from the workforce.
Preparing today for tomorrow’s threats.
Looking toward the era when everyone — and everything — is connected.
Data Center 20/20
The innovation and revenue engine of the enterprise.