Uber XP

Every launch is a bet

A ramp goes to employees first, then one percent of a city, then a quarter of a country. At every step an experiment compares the drivers who have the feature with the drivers who don’t. Uber runs more than a thousand of these at any time, across apps that carry thousands of flags.

When one goes wrong, a driver can’t sign in and doesn’t earn that night. The engineers and I worked from one question: when a launch starts hurting people, how fast can its owner know, and how sure can they be before they pull it?

Two lists, no verdict

XP began as Morpheus: two lists, every flag and every experiment, each row with an owner, a rollout bar and a sparkline too small to read. It showed what existed and who owned it. Whether any of it was hurting anyone, it could not say.

Morpheus feature flags list: a table of flag keys with author, team, last updated, platform, a rollout bar, audience size and a small sparkline, under a header reading Feature Flags, total 129, need cleanup 26 — Morpheus, the flag list XP started as.

My first pass gave launches stages and a number for their blast radius, and put both in the wrong place. The stages lived in a sidebar and the evidence in a panel, so what we shipped and what it did to drivers sat on opposite sides of the screen.

The flag is the page

The redesign was code-named Atomica. I threw out the lists and made the flag the page. It opens on a sentence, not a chart: “Treatment is hurting sign-in in latam_latest.”

Morpheus, the Feature Flags list: 129 flags, 26 needing cleanup, and two rows for Instant_Pay_Optimistic_Flag with author Tyler Woodhouse, team DEX, last updated, platform, a rollout bar, audience, a sparkline and a star. The first row carries a small warning triangle and nothing else to say what is wrong — Before, a flag in Morpheus is a row in a list: a name, an owner and a warning triangle. After, in Atomica, it opens on its status, the one action, and a sentence saying what is wrong.

Morpheus, the Feature Flags list: the first row, Instant_Pay_Optimistic_Flag by Tyler Woodhouse, with a small warning triangle and nothing else to say what is wrong — Before, a flag in Morpheus is a row in a list: a name, an owner and a warning triangle. After, in Atomica, it opens on its status, the one action, and a sentence saying what is wrong.

Under it sits what the system knows: when it started, what shipped just before, which error is up and by how much, who has been paged, and what to do. An engineer paged at 3 a.m. should know whether to act before reading a single axis.

The flag page, top to bottom. The verdict is drawn large enough to read; the evidence follows, smaller.

Cause and effect, one axis

Below the numbers is one timeline, with every ramp, the app release and the page to the team drawn on the metric’s own line. Nobody has to line up a deploy log against a dashboard by eye.

The timeline for latam_latest, taken apart into the five layers it is drawn in, each layer’s labels on its own sheet.

The flag timeline for latam_latest taken apart into five sheets of glass, stacked in the order the chart is drawn and tipped back so each can be seen. From the bottom: the grid and scale, with the days from December 27 to now along its front edge and the axis titles Successful logins / hour, Lift vs control and Rollout; the bars, with hourly exposure stepping up at each ramp and turning red in the last hours, the lift strip and the rollout steps; treatment and control, the two logins lines with their shading, treatment falling below control after the release; the events, with six markers and their names, Turned on, Rollout 25%, Rollout 50%, Driver app 3.214, Degraded and Identity joined; and on top the anomalies, the dots for the windows the system flagged, clustered after the release

AnomaliesWindows the system flagged.
EventsWhat shipped, broke, and who came.
Treatment and controlLogins an hour, shaded.
BarsExposure, lift and rollout.
Grid and scaleLogins, lift, rollout; the days.

The timeline for latam_latest, taken apart into the five layers it is drawn in, each layer’s labels on its own sheet.

Find the cause

The hardest incident to catch is the one the headline hides. Since this flag went on, successful logins are up 8.8%, and the result is significant. Over the last 24 hours they are down 5.4%. In latam_latest they are down 15.2%. So the timeline has one switch: all traffic, or the segment that is hurting.

The segments view says where. It sets every region and app version against every metric, shaded only where the difference is significant. Only latam_latest lights up. The row under it is the same region on the previous app, and those drivers are fine, which points at the release. Every dot on the timeline is a 15-minute window the system flagged. Open one and the chart makes room for a panel beside it: what that window did against what was expected, which segment the gap came from, and what shipped just before.

The segments grid: eight rows of region by app version plus an All traffic row. Each row has its share of drivers, a three-day line of lift marked at Driver app 3.214, and three metrics with intervals. Only latam_latest is shaded red, at minus 15.2% successful logins, plus 88% sign-in error rate and minus 27.0% unique trips per driver. latam_previous, the row under it, is unshaded at minus 1.1%. All traffic reads minus 5.4%, plus 31% and minus 9.2% — latam_latest carries the drop. latam_previous, the same region on the old app, is flat.

The segments grid's first three columns: each region and app version with its share of drivers and a three-day line of lift marked at Driver app 3.214. Only latam_latest's line drops, in red, after the release; latam_previous, the row under it, stays flat, and the All traffic line dips less — latam_latest carries the drop. latam_previous, the same region on the old app, is flat.

The anomaly inspector for window 12 of 20, January 2, 2:30 to 2:45 AM in latam_latest, marked outside the band, with sign-in error rate at window 2 of 3 toward Degraded and a small chart of treatment and control against the expected band. A breakdown shows latam_latest carries 97% of the gap, and that Driver app 3.214 shipped 11 hours 30 minutes before. Two buttons: Open alert and Roll back — One flagged window, opened. latam_latest accounts for 97% of the gap.

The last step is why. The performance view shows sign-in latency over time, one strip for each group. After the release, treatment grows a second band about a second out. Control never does. That band is drivers trying again after an error.

Sign-in latency over time: two heat strips across the last 48 hours, control above treatment, with p50 and p95 lines through each. Control stays one band near 0.4 seconds. In treatment a second band, labelled Retries after sign_in_error, builds past one second after Driver app 3.214 and lifts p95 from 0.84 to 1.31 seconds. A 24-hour distribution sits to the right of each strip — Sign-in latency over 48 hours. The second band in treatment is drivers trying again.

Sign-in latency over 48 hours, drawn narrower: control above treatment, each with p50 and p95 lines and a 24-hour distribution. Control stays one band near 0.4 seconds. In treatment a second band, labelled Retries after sign_in_error, builds past one second after Driver app 3.214 and the Degraded mark, lifting p95 from 0.84 to 1.31 seconds — Sign-in latency over 48 hours. The second band in treatment is drivers trying again.

Undo has a size

In my first pass, undo was a red Revert button at the bottom of a panel, with no word on what it would do. A rollback changes what hundreds of thousands of people see, so I designed it like a launch. You pick how far, and the dialog redraws what that means: 617,000 drivers back to control, or 210,000 in the one segment that is hurting while 407,000 stay in treatment. It says what happens to the experiment and to the alert. Then you type the flag’s name. A wrong click here is a second incident.

The roll back dialog with 0%, everyone on control, selected: 617K back to control and none staying in treatment, the experiment marked Stopped with its data kept, the alert staying open until sign_in_error is back under its limit, and the flag key half typed, 9 of 18 characters, with the Roll back to 0% button still disabled — Rolling everyone back.

The roll back dialog with Segment, latam_latest only, selected: 210K back to control and 407K staying in treatment, the other seven segments staying at 50%, the experiment still running everywhere else, and the flag key typed in full, marked Matches, with a red Roll back latam_latest button — Rolling back one segment. The numbers and the fine print change with the choice.

I have carried one idea from this dialog into everything since: show the consequence before the commitment. It is the rule behind AI needs receipts, and most of what designing for operators comes down to.

The policy is on the page

No one can watch a thousand experiments, so every flag carries guardrails: metrics with a limit. A breach marks the flag degraded and pages its owner, and that is all it does. Automatic rollback is a separate rule, written out where everyone can read it: sign-in errors 30% above control, on all traffic, for two 15-minute windows in a row. Automation that acts only inside rules people can read is what I am exploring now in Consentful Automation.

Guardrail history: 3 failing, 2 warning and 1 passing of 6 guardrails, drawn as six lanes over the last 48 hours in 15-minute windows, coloured pass, warn, past the limit and fail, with the Driver app 3.214 release and the Degraded mark running through all six. Sign-in error rate fails first, at 3:05 AM, then successful logins, then unique trips per driver; forgot-password rate and cold start are warning, crash-free sessions pass — Guardrail history over 48 hours. Sign-in errors broke first, 12 hours after the release.

Guardrail history, drawn narrower: six lanes over the last 48 hours with the Degraded mark through all six. Sign-in error rate, successful logins and unique trips per driver turn red and read Fail at 3:05, 3:50 and 5:10 AM; forgot-password rate and cold start warn; crash-free sessions pass — Guardrail history over 48 hours. Sign-in errors broke first, 12 hours after the release.

The sign-in error rate guardrail card: plus 88% against control, past its limit of plus 15%, latam_latest and plus 31% on all traffic, failing since 3:05 AM on a 48-hour line, and its rule: auto-rollback above plus 30% on all traffic, 1 of 2 windows — Sign-in errors carry the rollback rule: 1 of its 2 windows so far.

Every state, designed

Most of a flag’s life is quiet, and a page built for incidents makes every launch look like one. So I designed the header state by state, with one primary button for each of its five states: Increase to 50%, Increase to 100%, Roll back, Resume at 25% and Turn on.

The header in a flag’s five states. The status, the numbers and the one button change with each.

Every test ends in a decision

The old list never asked for a decision. The experiments view plots every test by how much of its sample it has collected against its lift. The upper right is ship: a win at the full sample. Below the funnel is stop: a significant loss ends a test at any sample size, because real harm should not wait for the sample to finish.

The decision map: 39 experiments across the Driver org plotted by share of planned sample collected against lift on the primary metric, the team's own with their 95% intervals. A funnel marks where the interval still spans zero, a ship zone sits at upper right past the planned sample, and a stop zone runs along the bottom. Three are labelled: driver_onboarding_carousel at plus 11.4%, significant at 103% of sample; starpower_haptics_ios at plus 1.2% with a crash guardrail; and driver_sea_magic_sign_in in the stop zone at minus 5.4% with 3 guardrails failing — Every experiment in the org on one map. Three need a decision. Point at an experiment to read its result.

One inbox for the team

Up a level, a team watches everything it owns. Alerts arrive by severity, each tied to its flag, its owner and what shipped just before. The inbox is worked from the keyboard, with J and K to move through it. Each breach is charted against its own limit.

Atomica Alerts inbox: six active alerts by severity on the left with Acknowledge, Resolve and Roll back actions and a J and K key hint at the foot, and the selected alert on the right, sign-in error rate at plus 88% against a guardrail of plus 15%, charted with markers for Driver app 3.214 and Degraded and a paragraph explaining the trigger — The alert inbox, with the sign-in error alert open.

The alert inbox list: six active alerts by severity, each with its flag, its owner and a small chart against its limit. The first, sign_in_error up 88% in latam_latest, is selected. Acknowledge, Resolve and Roll back sit above the list, and a hint at its foot reads J, K: next and previous alert — The alert inbox, with the sign-in error alert open.

What we traded

An experimentation platform can’t be made simple. The work was deciding where its complexity goes. The page opens on a sentence and one action. The views and controls for a particular anomaly or a particular rollout plan sit one step behind that, where the person who needs them will look.

Some of the choices cost something. Typing the flag’s name slows down the one person who is in a hurry, on purpose. Automatic rollback acts on all traffic only, and anything narrower stays a person’s call. And we cut all of the data visualization from the first version so the redesign could ship.

Outcome

Atomica replaced Morpheus about three months after we designed and built it. Rollbacks were the clearest win: they got faster, and they could be scoped to the segment that was hurting. In every app that used XP, experiments got easier to manage and faster to run, which made them cheaper.

For scale: in August 2018 Uber wrote that more than 1,000 experiments were running on the platform at any given time, across its driver, rider, Eats and Freight apps.

Team: One designer, frontend and backend engineers
Time: About three months
Apps: Rider, Driver, Eats, Freight, Fleet, Autonomous
Prototype: Rebuilt from the 2017 design

Every launch is a bet

Two lists, no verdict

The flag is the page

The redesign was code-named Atomica. I threw out the lists and made the flag the page. It opens on a sentence, not a chart: “Treatment is hurting sign-in in latam_latest.”

Cause and effect, one axis

Below the numbers is one timeline, with every ramp, the app release and the page to the team drawn on the metric’s own line. Nobody has to line up a deploy log against a dashboard by eye.

The timeline for latam_latest, taken apart into the five layers it is drawn in, each layer’s labels on its own sheet.

AnomaliesWindows the system flagged.
EventsWhat shipped, broke, and who came.
Treatment and controlLogins an hour, shaded.
BarsExposure, lift and rollout.
Grid and scaleLogins, lift, rollout; the days.

The timeline for latam_latest, taken apart into the five layers it is drawn in, each layer’s labels on its own sheet.

Find the cause

Undo has a size

The policy is on the page

Every state, designed

Every test ends in a decision

One inbox for the team

What we traded

Outcome

For scale: in August 2018 Uber wrote that more than 1,000 experiments were running on the platform at any given time, across its driver, rider, Eats and Freight apps.

Team: One designer, frontend and backend engineers
Time: About three months
Apps: Rider, Driver, Eats, Freight, Fleet, Autonomous
Prototype: Rebuilt from the 2017 design