Moving fast, without breaking things: Modern software delivery with feature flags

Foreword

No matter what industry you're in, the pace of change keeps ratcheting up. Organisations see increased competition from both established incumbents and nimble upstarts, while at the same time the external landscape these businesses operate in seems increasingly turbulent—regulations change, supply chains are shocked, revolutionary technology appears seemingly overnight…

To respond to all this change, enterprises need to become more nimble—they need to be able to make quick decisions, experiment, and adapt. And given the increasingly digital nature of every enterprise, this naturally leads to higher and higher expectations around the pace of software delivery. 

But while expectations around the pace of software delivery are increasing, expectations around quality, security, and compliance remain the same, particularly for larger enterprises operating in higher regulatory environments. 

On the face of it, this appears to be an unresolvable conflict. Surely accelerating the pace of change means accepting a trade-off in terms of risk—higher risk for bugs and errors slipping past QA, higher risk for deployment incidents, higher risk for security vulnerabilities. "Move fast and break things", as the saying goes.

Happily, that's not necessarily the case. Over the last couple of decades, leading software organisations have demonstrated that accelerating the rate of change can actually reduce risk. It turns out that, when you use the right practices, you can "move fast, with safety". 

“Our biggest challenge right now is modernisation—what tooling can help and where do we start?”

“How can we reduce risk in our releases without slowing down our development cycles?”

“How do we introduce feature flags across our entire engineering organisation?”

“How are other large organisations using feature flags?”

‍

These are questions we get a lot. Questions that often kick-start really interesting conversations with software architects, engineering leaders, and developers working in regulated industries. The problem? They’re all happening behind closed doors. 

This guide throws open those doors, exploring these questions and serving as a roadmap for introducing feature flags into large, regulated, or complex organisations—and then scaling them.

Why feature flags?

Feature flags aren’t a new phenomenon. Notable early converts include companies like Flickr, Etsy, and Facebook, with Martin Fowler and Pete Hodgson writing this foundational 2017 article that still influences thinking today.

As their popularity has grown, companies operating in complex environments have started to see the benefits. At a high level, they can: 

• remove development bottlenecks
• minimise risk around releases
• speed up time-to-market for new features
• and improve end-user experience (by significantly shortening feedback loops)

Building a strong foundation for feature flags

How to fight vendor lock-in: Insights from OpenFeature

Michael Beemer, OpenFeature Governance Board Member

‍

Feature flags have become essential to modern software delivery, but implementation across your enterprise presents a critical choice: how should feature flags be managed? Many options exist, ranging from homegrown solutions to third party vendors. Until now, each choice required developers to use solution-specific feature flagging patterns, locking organisations into their choice and presenting a high switching cost to explore other options.

‍

Following in the footsteps of OpenTelemetry, OpenFeature, a CNCF incubating project with broad industry support, solves this problem by providing a vendor-neutral, open standard for feature flags. Your development teams write code against one consistent API, while your organisation maintains the freedom to use any provider. Many organisations report that introducing OpenFeature’s abstraction layer is a key step in migrating from disparate internal solutions to a consolidated feature-flagging provider, or even abstracting multiple different providers behind a shared interface. 

Why vendor-neutral standards matter

Many enterprises face a common challenge that compounds as they scale: feature flags exist in multiple forms across the organisation, from dedicated vendors to multiple homegrown feature flagging and configuration management solutions. This fragmentation creates inconsistent developer experiences and complicates governance, especially in regulated industries where oversight is paramount.

People & processes

Doing a good job of introducing any new technology into a large organisation starts with people—the people behind the technology and the internal teams who will adopt it. 

Driving change in large organisations is an iterative approach—it won’t happen overnight and it doesn’t need to be overwhelming, especially if you take the time to plan out your implementation. Keep these steps in mind when approaching your planning:

1. Ensure that company leadership is on board—transformations driven by executive sponsorship tend to have a higher rate of success
2. Identify a champion or working group who can own the implementation, help with internal onboarding, and work directly with your feature flag provider to get up and running
3. Decide if a pilot programme is needed
4. Come up with a plan for a phased rollout (more on this on page x)
5. Map out how you are currently releasing new features and who is part of the process
6. Identify any existing feature flagging solutions currently in use—they might be disguised as configuration management

Tailoring feature flags to your organisation’s needs

As you begin to determine how to introduce feature flags into your release processes, ensure your team is familiar with the different kinds of flags as well as best practices to keep your flags healthy and reduce tech debt.

Create governance for how you will use feature flags 

Flags can become an anti-pattern if implemented incorrectly, so keep in mind their different types and lifecycles as you begin to determine how your organisation will use them. We like to think of feature flag governance as a series of layers—every type of flag will look different depending on your infrastructure and development practices and the level of security you need. 

Set roles & permissions 

Data security is top of mind for every company, but organisations operating in regulated industries are held to a higher standard.

Determining feature management access levels will be part of the conversation as you begin to introduce feature flags at scale. It’s important to find the right balance between rigorous security standards and productivity. You want to avoid creating unnecessary bottlenecks. 

One way to manage this is by creating roles and permissions for users and taking advantage of features your provider has, like change requests and feature approval workflows.

5 feature flag best practices

1. Clean up your feature flags! 

The best way to get the most out of feature flags (and enjoy using them while you’re at it) is to clean them up after they’ve served their purpose. Create a built-in process for flag archiving, making this a regular part of your engineers’ workflows, just as important as reviewing pull requests or clearing issues in your issue tracker. This will help manage tech debt and save your team time and messiness down the road. Part and parcel of this is understanding the difference between long-lived and short-lived flags and when to use them (more on flag lifecycles below).

2. Design features around flags

Feature flag deployment should be planned early on in the software development process. If flags are simply layered in as an afterthought, they won’t be as effective. Whenever you start work on a new feature, think about how you can put it behind a feature flag. The possibilities of feature flags are endless, which makes it even more important to define a flag’s scope.

Using feature flags for safer releases & faster development

It may seem counterintuitive to introduce greater flexibility and speed into your release practices when stability and predictability are paramount. Yet, predictability has a downside: if your company’s deployment practices involve long development cycles that are predictably slow and painful, predictability isn’t doing you any favours, it’s likely blocking innovation and increasing the risk of something breaking along the way.

Our co-founder Kyle has this to say on the matter, “You would think that releasing more rapidly would decrease stability but usually, the inverse is true. If releases are less of a big deal, you make fewer mistakes because it’s more manageable to release code.”

Feature flags can help you reduce risk, improve development velocity, and shorten feedback loops, strengthening your product offering.

Easy reference guide for feature flag types & uses

Short-lived feature flags enable you to integrate new code into your main codebase while keeping it inactive until release. For example, you can use a feature toggle if you want to hide a new transaction feature during testing. When you’re done testing, switch it on and release it to all users before removing from your code, or setting a reminder to.

Long-lived feature flags remain in the codebase for extended periods—or permanently. They encompass kill switches, and support for multitenant architectures. You need to be careful and ensure they aren’t removed since they’re critical to the stability or usability of your application.

-------------------------------------------------

User-based feature flags control feature visibility based on user attributes or segments. Let’s say you’re releasing a new tax filing feature in a government portal. You can release it to only small business users or sole proprietors before releasing it to your entire user base.

System-based feature flags control functionality based on technical or environmental factors—not user attributes. They work well in cases where you use different deployment environments or system conditions. For example, a finance app could use these flags to deploy region-specific compliance features based on the server’s geographic location.

eBay case study: Migrating to OpenFeature at scale: 0 to trillions of calls

Where it started

eBay was navigating challenges familiar to many large organisations—long development cycles, riskier-than-necessary deployments, and a siloed problem-solving culture.

Meet the Velocity program: Faster delivery matters! 

A 300-team cross-organisational effort staffed with forward-looking principal and senior principal architects and key product leaders, the Velocity program was built to succeed where other modernisation efforts had failed. The program's northstar mission was to make software and product delivery a competitive advantage for eBay. 

In practice, this looked like a ground up reworking of how eBay thought about feature delivery. How to make it safer, easier, and faster, driven primarily by the DORA and SPACE methodologies and focusing on three main levers to find success:

Wrapping up

Feature flags allow large enterprises operating in complex environments to move with an agility that was previously unattainable. By releasing development teams from deployments that are tightly coupled to releases, engineers, QA, and product teams, among others, can move at their own pace, increasing development velocity as well as shortening feedback loops.