Joshua Porter, VP at UXPerformable
Metrics-Driven Design
@bokardo
Doug Bowman (now Creative Director for Twitter) left Google and lashed out on his blog about his decision to leave:
Yes, it’s true that a team at Google couldn’t decide between two blues, so they’re testing 41 shades between each blue to see which one performs better. I had a recent debate over whether a border should be 3, 4 or 5 pixels wide, and was asked to prove my case. I can’t operate in an environment like that.
http://stopdesign.com/archive/2009/03/20/goodbye-google.html
For Doug, this was infuriating and it drove him away.
When a company is filled with engineers, it turns to engineering to solve problems. Reduce each decision to a simple logic problem. Remove all subjectivity and just look at the data. Data in your favor? Ok, launch it. Data shows negative effects? Back to the drawing board. And that data eventually becomes a crutch for every decision, paralyzing the company and preventing it from making any daring design decisions.
http://stopdesign.com/archive/2009/03/20/goodbye-google.html
The vast majority of designers are intuition driven - they’re instinctive, subjective and daring. Data driven design is deliberate, objective and safe. You iterate by one variable at a time and you rely on data as the proxy decision maker. What happened to trusting your gut?
Imagine that your design is a mountain. Your small mountain is your existing design - it’s on the side of the mountain. Take an engineering driven approach and you can push the design up to the top of your small mountain. This is your local maxima - you can’t go any further because there’s nothing else to test. Your goal is on top of the other larger mountain. You’re on the wrong mountain.
Optimization asks: What works best in the current model?
Design innovation asks: What is the best possible model?
What are metrics? It’s as simple or as complicated as you want them to be.
By definition: Metrics are simply numbers that measure the effectiveness of your business.
5 Reasons why metrics are a designer’s best friend:
- Metrics reduce arguments based on opinion. (They don’t remove them... they reduce them.)
- Metrics give you answers about what really works. (They can also lead you down a rabbit hole.)
- Metrics show you were you’re strong as a designer. (Also shows where you’re weak as a designer.)
- Metrics allow you to test anything you want. (Metrics empower you to try whatever you want to try. You can test the effectiveness before selling it.)
- Clients love metrics. (If you can go into a pitch and show them metrics that this worked better than this, they’ll love it.)
PRINCIPLE: Your metrics will be as unique as your business. Let’s talk about metrics:
- Acquisition metrics
- CPA - Cost per acquisition
- How much did it cost to acquire that customer?
- If your CPA is higher than your LTV (Life Time Value) then you’re in trouble.
Look at your most active users and reverse engineer their experience. Design changes to Facebook’s deactivation page accounted for 1 million members not leaving the service.
Principles of Design Metrics:
- Optimise in small steps
- Innovate w/ daring leaps
- No design survives contact with user.
Ends with post from Seth Godin:
Netflix tests everything. They're very proud that they A/B test interactions, offerings, pricing, everything. It's almost enough to get you to believe that rigorous testing is the key to success.
Except they didn't test the model of renting DVDs by mail for a monthly fee.
And they didn't test the model of having an innovative corporate culture.
And they didn't test the idea of betting the company on a switch to online delivery.
The three biggest assets of the company weren't tested, because they couldn't be.
Sure, go ahead and test what's testable. But the real victories come when you have the guts to launch the untestable.