Dogfood Testing: Why We're Running Experimentation on Our Signup Page

We want to know if forcing a corporate email at signup helps a team adopt Flagsmith, or if it causes more friction. We had a feeling we knew the answer, but a feeling isn't a decision.

Meanwhile, we've just launched our new experimentation tool as a closed beta.

Perfect timing, as it meant we could build an experiment inside our own product, on our own signup page, using the feature we're currently testing.

We are the first users of the experimentation feature in a production environment, so we can catch anything before our users.

Why we're testing signup

We want new users to be able to test out the product before becoming paying customers, but we know that users who sign up using a public email can find it challenging to migrate to their company address once they’re ready to buy.

So we're running a small test.

During the test, half our new signups get a warning that a corporate email is preferred but can carry on regardless. The other half get the same warning, but they're blocked from proceeding until they switch away from a personal address. It's an even 50:50 split.

We don't expect signups to change much. Quite the opposite: we expect them to stay flat. If they do, that's the evidence we need to say we can safely make the corporate-email-only rule permanent, without hurting the experience for people signing up.

The only users who actually tell us anything here are the ones who tried a personal email first, since anyone already on a corporate account sees no difference either way. That's a straightforward concept, even if the honest answer to "will this work" was still just a feeling before we ran it.

The benefits of dogfooding

We didn't have to build this using our own (very new) tool; we chose to, because there's no substitute for putting yourself in the shoes of your own customer.

And, honestly, you need to multiply your own pain a bit to really understand the customer experience.

If it's a bit annoying for me, that's fine; it's part of my job. If it's annoying for a customer trying to run their first experiment, that's a bigger problem, because that experiment sits at so many crossroads for different teams: analytics, product insight, and code that's going to ship to production.

All of that has to be top-tier quality, or none of it holds up.

Mining for gold in the dogfood

Setting this up ourselves, rather than reading about it in a spec, has already taught us things we wouldn't have found any other way. It's gold, honestly.

Writing the experiment code surfaced real gaps in the SDK: the first job is deciding, in code, when your experiment starts and what counts as success or failure, and that part wasn't as straightforward as we expected.

There's a trap in the word "experiment." It can make it feel like a sandbox, like you can write it however you like because none of it matters much.

It's still code that will reach production. It has to be clean, it has to be easy to use, and it has to be isolated, so the time to production is as short as possible.

An experiment is short-lived by design; the code is supposed to disappear once it's done its job, which means it has to be self-contained enough that it never touches the critical parts of the application while it's alive.

Simplicity, at every step

The whole philosophy of the new experimentation feature is simplicity at every step.  We provide a Flagsmith-hosted warehouse (on ClickHouse) where we store experiment data. One button sets up your warehouse connection, so you're not wiring up credentials by hand before you can even start—a process that's much simpler for SaaS users who don't have a warehouse.

From there, it takes about one minute to establish what you're testing, which flag you're using, and what your initial rollout looks like.

We want a customer to be able to start running their first experiment within an hour, start to finish.

We have also been working on allowing customers to bring their own warehouse. ClickHouse is already supported, and we plan to add Snowflake, BigQuery, and Databricks soon. All your experimentation data will stay on your own server.

Experimentation built on feature flags

Our experimentation feature is built on our feature flags platform. We don't have the ambition to become a full product analytics platform; that's not the plan.

What we're doing is combining two things: the business metrics you actually care about when testing features, and a feature flag that keeps your whole application safe while you watch that metric move.

It's not about giving you twenty fascinating statistics. It's simpler than that: You want to release something, you want to release it fast, you've got a hypothesis, and you don't want to disrupt your user base while you find out if you're right.

Why AI makes this even more important

AI has unleashed a velocity in software development that we've never seen before, and feature flagging is the safety net that lets your developers—and increasingly their AI agents—accelerate without losing control of what ships.

The real question, for any engineer or any product team, isn't just "can I ship this." It's "is it worth it?" That's what experimentation, combined with feature flagging, actually answers.

You find out if it's worth it, and if it's not, you just revert back to where you were. If it is, you accelerate the rollout, because it's genuinely bringing value to your users.

What's next

This is our first time dogfooding, and we're still in it. The goal now is to start gathering insights from the first customers using this for real, on their own experiments. The more we do this, the more we learn how to get the most value out of it—for them and for us.