Development Through Experiments: Measure Every Change

Two numbers are worth knowing before you plan your next sprint. When Pendo analyzed usage data from 615 software products, it found that 80% of features are rarely or never used. And teams that measure every change find that most ideas don't work: at Microsoft, only about one in three tested ideas improved the metric it was meant to improve. The rest did nothing or made things worse.

Most companies never see this. They track their top-line goals, like revenue, churn and total signups, plus one or two obvious funnels. Very few measure whether each individual change actually worked. Without that, the changes that didn't work stay in the product, and nobody knows.

The usual product loop looks like this. You collect feedback from client surveys, reviews and the support channel. You turn the loudest themes into the next sprint. You ship. Then you go back to the same surveys and support tickets to see whether the update landed.

It works, sort of. But it has a blind spot: you never really know whether the change made things better. Feedback tells you what people feel. It rarely tells you what they did. Survey responses come late, and they come mostly from the happiest and the angriest users. Support tickets tell you what broke, not what quietly started working. "Nobody complained" is not the same as "it worked".

There is a better way: develop through experiments you can measure.

# The core rule

Every change goes through the same four gates:

1. Reason. If we want to change something, we must know what we are trying to improve and why.
2. Measure. We must know how to measure that improvement in numbers, before we build anything.
3. Change. We ship the change in a way that lets us measure it.
4. Decide. We look at the numbers and either roll back, or adopt and build on the change.

Then the loop starts again with a new goal. That's the whole method. The rest of this article covers how to run each step without fooling yourself.

Here is why every change needs a number. What a team expects and what actually happens often point in opposite directions:

In every case, the only thing that separated a good decision from a bad one was having a number defined before the change shipped.

# Step 1: Find what to improve

Keep your existing channels. Surveys, support tickets, sales call notes, churn interviews and app store reviews are still the best place to find what might be wrong. Add your own product data to the list: funnel drop-offs, abandoned carts, features nobody opens, pages where users get stuck.

What changes is the output of this step. Instead of a backlog item like "Redesign onboarding", you write a hypothesis:

We believe that shortening the signup form from 9 fields to 4 for users coming from Google Ads will increase signup conversion, because support and session recordings show people dropping off at the company details step.

A hypothesis names the change, the audience, the expected effect and the evidence behind it. If you can't fill in the "because" part, you don't have a reason yet. You have a guess. Guesses are fine, but they should be cheap to test.

# Step 2: Define success before you build

Before anyone writes code, agree on what success looks like and how you will measure it.

Take a concrete example: "We convert only 30% of users who arrive from Google Ads. We want to get that to 40%." Now you know what success is, and you can measure it properly.

A good success definition answers six questions:

Guardrails are easy to forget and very important. Making a form shorter can raise signups and also fill your pipeline with low-quality leads who never pay. If you only watch the main metric, you will celebrate a loss.

Finally, write down the decision rule in advance: "If conversion reaches 36% or more with no guardrail getting worse, we adopt. If it's below 33%, we roll back. In between, we iterate." Deciding this before you see the data stops you from moving the goalposts later.

# Step 3: Ship it so you can measure it

There are two practical ways to ship a measurable change.

# Option A: A/B test with a control group

Split incoming users at random. The control group sees the current version and the test group sees the new one. Both groups live through the same week, the same marketing campaigns and the same seasonality, so any difference between them comes from your change.

This is the most reliable method, but it needs enough traffic. A rough guide, at the usual 95% confidence and 80% power:

• To detect a jump from 30% to 40%, you need about 360 users per group.
• To detect a jump from 30% to 32%, you need about 8,400 users per group.

Small improvements need a lot more traffic to prove. If your traffic is low, test bolder changes or accept that you'll need to run the test longer.

Why: a smaller difference is easier to mistake for random noise, so you need many more users to see it clearly. The required number of users grows with the square of how small the difference is.

# Option B: Roll out to everyone and monitor

Sometimes a split isn't possible or worth it: a pricing page change on a low-traffic B2B site, an infrastructure change, a new support process. In that case, ship to everyone and compare against your baseline.

This is cheaper, but noisier. A new ad campaign, a holiday or a competitor's outage can move your numbers as much as your change does. To reduce the noise:

• Compare equal, similar periods (four weeks before vs. four weeks after, same weekdays).
• Keep a log of everything else that changed in that period: marketing, pricing, outages, releases.
• Ship one meaningful change at a time to the same funnel, so you know which one caused the effect.
• If you can, keep a small holdout (say 5–10% of users) on the old version as a sanity check.

Whichever option you choose, ship behind a feature flag. Rolling back should take a click, not a release.

# Step 4: Decide, then set the next goal

When the measurement period ends, apply the decision rule you wrote in Step 2:

• Roll back. The change didn't help, or it hurt a guardrail. This is not a failure. You learned something for the price of one sprint, and you didn't leave a harmful change in production for a year.
• Adopt. The change hit the target. Make it the new default and clean up the flag and the old code.
• Adopt and extend. It worked, and the result suggests where to go next. The 30% → 40% win on Google Ads traffic leads to the next question: does the same change work for organic traffic? Can we get from 40% to 45% by adding social login?

Each result becomes the baseline for the next hypothesis. Over time, the product improves in steps you can prove, instead of in releases you hope went well.

# It works for revenue and satisfaction too

Experiments aren't only for signup funnels. Anything you can count, you can improve this way:

• Revenue: trial-to-paid conversion, average order value, upgrade rate, expansion revenue, churn.
• Conversions: landing page conversion, checkout completion, activation (users who reach the "aha" moment in their first week).
• Client satisfaction: CSAT after a support interaction, NPS by segment, support tickets per 1,000 active users, time to first response, time to resolution.

Satisfaction is where teams usually fall back to gut feeling, and it doesn't have to be that way. "Adding in-app guides for the reporting module will reduce 'how do I…' tickets from 42 to under 25 per 1,000 active users within six weeks" is a perfectly measurable goal.

# Common traps

• Changing five things at once. If the redesign ships with a new pricing table, a new headline and a new form, you won't know which one did the work, or which one quietly hurt.
• Stopping early. Checking results every day and stopping the moment they look good is the fastest way to adopt a change that did nothing. Wait for the planned sample size or period.
• Vanity metrics. Page views and time on site are easy to move and rarely matter. Pick metrics that are tied to revenue or retention.
• Ignoring segments. A change can help mobile users and hurt desktop users, and average out to "no effect". Look at your key segments before you decide.
• The novelty effect. Users click on new things because they're new. For UI changes, let the test run long enough for the novelty to wear off.
• Rewriting the goal after the fact. "Conversion didn't move, but engagement went up, so it's a win." If engagement was not the goal, it's not a win. Write a new hypothesis and test it properly.

# What you need in place

You don't need a dedicated data science team to start. You need:

1. Event tracking you trust. If the numbers are wrong, every decision built on them is wrong. Make sure your key funnel steps are tracked consistently.
2. Feature flags. So you can split traffic, roll out gradually and roll back instantly.
3. An experiment log. One shared document or board where every experiment is recorded: hypothesis, metric, baseline, target, result and decision. In six months this becomes the most valuable record of what your users actually respond to.
4. A regular review. A short meeting every week or two to read results, make the call and pick the next hypothesis.

A simple experiment card is enough to get started:

Experiment:     Short signup form for Google Ads traffic
Reason:         38% of signup drop-offs happen at the company details step
Hypothesis:     Cutting the form from 9 to 4 fields raises conversion
Metric:         Signup conversion, Google Ads visitors
Baseline:       30% (last 8 weeks)
Target:         40%
Guardrails:     Trial-to-paid rate, support tickets / 1,000 signups
Method:         A/B test, 50/50 split, min. 400 users per group
Decision rule:  ≥ 36% and guardrails OK → adopt; < 33% → roll back
Result:         —
Decision:       —

# Summary

Feedback channels are great at telling you where to look. They are bad at telling you whether you fixed it. Developing through experiments closes that gap with one simple rule:

If we want to change something, there has to be a reason. We must know how to measure the improvement in numbers. We make the change. We measure, and then we either roll back or adopt and build on it.

Teams that work this way ship fewer changes that "felt right", and more changes that provably move revenue, conversions and client satisfaction. Over a year, that difference adds up.