Discover Performance

HP Software's community for IT leaders // July 2014
Subscribe

How to lead a DevOps transformation

Gene Kim writes about Gary Gruver’s innovative leadership of the HP LaserJet Firmware team.

HP's DevOps resources

*registration required

By Gene Kim

This is one of the most startling stories of DevOps-style transformation I’ve ever seen. It improved the productivity of developers by two or three times, but it wasn’t for software supporting a website—it was for the firmware software that supported the enterprise HP LaserJet family of printers.  

The 2006 project was led by Gary Gruver as director of LaserJet Firmware for Hewlett-Packard. (He is now VP of Quality Engineering, Release, and IT Operations at Macy’s.) Gary wrote about this transformation in depth in his book, A Practical Approach to Large-Scale Agile Development, and he’ll be at the DevOps Enterprise conference in San Francisco, Oct. 21–23, which I’m cosponsoring.

Gary describes this transformation as fundamentally DevOps, even though firmware isn’t delivered as a service by IT operations. Gary’s story shows us that DevOps is not just for unicorns (the "exceptional" companies like Google, Amazon, Netflix, Etsy, and Twitter)—DevOps is for any value stream where development, testing, and operations must work together to achieve organizational goals.

The business problem

Gary’s group was responsible for the firmware code that ran on all enterprise LaserJet products. Like the consumer printer market, the enterprise market was incredibly competitive, with new and innovative offerings showing up nearly every month.


Gary Gruver
The firmware group was having tremendous difficulty keeping up with demand for new, innovative features, despite having somewhere between 400 and 800 developers supporting more than 10 million lines of code. They were completing two software releases per year, with the majority of their time spent porting code to new products. Only about 5 percent of their time was spent creating or supporting new features.

The net result: Gary’s group was the bottleneck for the entire business line, and marketing finally gave up asking for new things. "When that happens," Gary says, "you know you’re in a bad situation, and you certainly can’t win in a competitive marketplace like we were in."

Slow feedback loops kill

Here are some relevant statistics of Gary’s team before the transformation that may seem all too familiar to other software projects:

  • 5 percent of development time spent writing new features
  • 15 to 20 percent of development time spent integrating code into mainline
  • When a developer checked code in, it took about six weeks to learn whether the code worked
  • It took one week to determine whether it integrated successfully
  • It took another day to reach the "integration build"
  • Another day was spent running acceptance tests
  • It took six weeks to complete full (manual) integration testing

"We were constantly being interrupted by new defects that were introduced into the code base up to six weeks before," Gary says. "How can you expect a developer to learn anything under these conditions—so that we could prevent it from happening in the future? You can’t! The only thing that happens is management yells at someone for something they did six weeks ago."

That is a profound observation: no one can learn anything from slow feedback. Learning can occur only if we can see the cause-and-effect linkage between the work and the outcomes.

Implementing continuous integration, and a dramatic architecture change

1. Architecture changes: Gary moved all developers onto a common code base. They eliminated separate branches for all products, putting all LaserJet models into trunk.

2. Automated testing: To support self-testing builds, they built a set of automated unit, acceptance, and integration tests, which would continually run against trunk. "Without automated testing," Gary says, "continuous integration is the fastest way to get a big pile of junk that never compiles or runs correctly."

Automation created fast feedback. A developer would know within hours whether the code they committed worked. The before-and-after:

  • Build cycle time: 1 week -> 3 hours (10–15 builds per day)
  • Commits: 1 code commit/day -> 100 commits/day
  • Regression test cycle time: 6 weeks -> 24 hours

3. Stopping the line: Once they had an automated test suite, Gary’s team created a culture where all work stopped when someone broke the build or the test suite. To help developers get the deployment pipeline running again, they created a chat room to enable quick and easy communication.

Performance breakthroughs

Here’s where Gary’s team ended up:

  • 75K–100K lines of code changes every day
  • 100–150 code commits per day
  • 5 percent to 40 percent of time spent writing new features

Gary told me that the key to a DevOps breakthrough is understanding that code isn’t done when it runs in a dev environment—and the Agile value of always keeping the code base stable and close to releasable.

"To me, DevOps in the enterprise is all about driving that discipline into large organizations with different groups and applications that have to work together," he says. It’s about everyone focusing on the end-user experience. "At its core, it is about defining and driving a more robust enterprise-level definition of done."

This is a shortened version of a post on my ITRevolution blog. For the full story, read Gary’s book A Practical Approach to Large-Scale Agile Development. You can find his blog at PracticalLargeScaleAgile.com. Gary Gruver will also be speaking at the DevOps Enterprise conference in San Francisco, Oct. 21–23, 2014. I hope to see you there!

Keep up with IT trends and leadership insights by subscribing to Discover Performance.


x

IT leader assessment

This tool evaluates the correlation between IT attributes and business success and, based on how your answers compare with average scores, will advise you where to invest in IT.

It is based on data HP collected from 650 global companies about a range of IT characteristics (server capacities, approach to information management, security, BYOD, etc.) and how they correlate to revenue gain. This assessment will compare your answers to the average scores in that study.

There are 12 questions that will require an estimated 10 minutes of your time. You'll receive a summary of your rating upon completion.



Let's get started
x

Please select an answer.
x

Analysis:

Your answer:
Your score:
Average score:
Revenue leaders' score:


x

Please select an answer.


x

Results

Your score:
Average score:
Revenue leaders' score:


Get detailed results:

Subscribe

Popular tags

Events

Meg Whitman: Deliver amazing apps now

HP’s CEO and her software management team discuss how enterprise IT can stay on top of the latest technology influencers. (On demand)


HP Discover 2014

Join thousands of IT execs, engineers, and solution experts to explore IT trends, strategies, and best practices. (Barcelona,
Dec. 2–4)


Discover Performance Weekly

HP Software’s Paul Muller hosts a weekly video digging into the hottest IT issues. Check out the latest episodes.


Enterprise 20/20

Dev Center 20/20

How will we organize development centers for the apps that will power our enterprises?

Introduction to Enterprise 20/20

What will a successful enterprise look like in the future?

CIO 20/20

Challenges and opportunities for the CIO of the future.

Marketing 20/20

Welcome to a new reality of split-second decisions and marketing by the numbers.

IT Operations 20/20

How can you achieve the data center of the future?

Employee 20/20

What the workforce of 2020 can expect from IT, and what IT can expect from the workforce.

Security 20/20

Preparing today for tomorrow’s threats.

Mobility 20/20

Looking toward the era when everyone — and everything — is connected.

Data Center 20/20

The innovation and revenue engine of the enterprise.

Read more

HP Software related

Most read articles

Discover Performance

Archive

Tweets @ HPITperformance