Image by SerwanJN
Big Ideas

Obstacles to Innovation Within Large Organizations: The Invisible Engineering Perspective

Innovation is notoriously challenging within large organizations. We work closely with firms like IDEO and McKinsey who have spent years helping break down the innovation-preventing barriers. To attempt to summarize everything that’s been said about innovation would be like sweeping sand off the beach.

I’d like to share some observations from a perspective most people don’t see—from inside an engineering team tasked with developing innovative technology products. This perspective is so rarely seen due to the chasm that typically exists between executive and engineering teams. Who’s asked how to solve the innovation problem? Not the engineering team.

For you business folks, consider this a peek through an engineer’s glasses at a few of the challenges developing innovative technology products specific to large organizations.

Hire aligned engineers, not just engineers.

In addition to basics like personality and ability to work with others, most engineering hiring is done exclusively based on engineering knowledge—the ability to answer engineering questions at a whiteboard.  Since that isn’t what most engineers do all day, why do we put so much emphasis on it?

Yes, knowledge is table stakes, but an effective engineer is one who can make his or her knowledge valuable to your team.

Tests for effectiveness might include good interpersonal skills or the natural inclination to work collaboratively as a team. Those are pretty obvious.

Here’s an indicator of effectiveness that’s almost always overlooked: alignment with the way your team does engineering. 

In fact, beyond software engineering, the way engineering is done on most teams isn’t even a conscious decision, so it can’t be tested for. Therefore, large engineering teams typically have zero alignment on basic methods by which to do the engineering they do. Without a unified approach, these teams perform more like a group of individual consultants than a team. Each person does his or her best based on individual experience. If any specific methods by which to do engineering on a day-to-day basis are introduced to help a team improve, energy that could have gone toward forward progress is wasted overcoming resistance.

Once a team reaches a certain size, it’s virtually impossible to align an entire team to all agree to do engineering the same way—personal experience is too diverse. If compatibility with shared engineering methods aren’t tested as a team is formed, it becomes difficult to introduce new ways to help a team improve later.

Hire engineers committed to a better way.

From my experience, engineers can be classified in one of two ways: those who have an ever-shrinking solution space defined by what they believe will deterministically lead to success, and those who have an ever-expanding solution space as a result believing there is always a better, more effective way to do what they do.

Put another way, there are engineers defined by their aversion to risk, and engineers eager to try new things to improve. The former have an increasingly narrow belief in what can be effective, because anytime something doesn’t work, even when it’s a “5% case,” it’s never to be done that way again—too risky. The latter continually perform experiments, on a small or large scale, consciously or subconsciously, and rapidly synthesize lessons learned to work toward what provides improvement 95% of the time.

I have been shocked at how cleanly engineers fall into one of these two categories, and how deep-rooted this trait is within an engineer’s psyche. The larger a team is, the more risk-averse engineers there tend to be, who thwart the efforts of those committed to continual improvement. If the risk-averse engineers win, the capability of the team actually decreases over time as safe, lowest common denominator approaches settle out as the norm.

Don’t let the executive team define both the product and release date.

We used to succumb to this on a regular basis: the executive team dictates a product and ship date, then it’s up to the engineering team to pull off a miracle. Now I’m flabbergasted by this, because I can’t think of a more effective way to kill innovation.

The executive team defines what is shipping and when, because within a large organization there are many moving parts, and those parts must be coordinated as far in advance as possible. Typically the ask date is far too aggressive because the executive team is basing release timing solely on business needs, without clear knowledge of what will be required to engineer the product.

The engineering team works hard (and blindly) to meet the milestones requested of them, and when the deadlines inevitably aren’t met, they get pushed out, and the engineering team quickly burns out with push after push. Decoupling product definition and ship date results in shipping better products.

The executive team sets the development priorities of the engineering team throughout development, and the engineering team tells the executive team how long certain aspects of the product will take to develop. The entire team works closely together to find the sweet spot of product and release date.

You might wonder how the executive team can operate if they don’t know what they’re getting and when. We’ve found the best way to handle this is to find middle ground. The engineering team does their best to give a schedule they know can be met for a product that meets the nominal requirements, and the executive team understands that visibility into both product details and ship date will increase gradually as the product progresses further down the development path. By softening the need to know exactly what product is shipping and when, and continual communication between executive and engineering teams, the best product is released at the right time.

These factors are invisible to most executive teams, yet they severely impede innovation within large organizations.