What to Build and How to Build It

Mark Ramm: Answering questions with Python

Previous SourceForge technical lead TurboGears 2 technical lead Current Juju technical lead

Don’t waste your life

  • 9 out of 10 projects fail

  • 9 out of 10 startups fail

  • Sturgeon’s Law!

Building products - failing for technical reasons or poor execution - failing because it wasn’t actually useful

Try to build things people want!

Market Research vs Total Bullshit

Not all product managers bother to do research

Even when they do, results can be questionable

Management culled at SourceForge

XKCD: Stand back, I’m going to try: SCIENCE!

Test Driven Management

In dev, tests provide measurable progress

Can you apply the same idea to management?

3 screenshots from SF download page - updated with large download button - redesigned (ended up slightly worse) - current design is actually measured

First two redesigns had no metrics. Final redesign had metrics to track how often people made it through to get the files they wanted.

Scientific Management

Document assumptions Work with the team to learn what works Don’t just make shit up!

Scientific method - ask a question - do background research - construct a hypothesis - test hypothesis through experiment (or, observation if necessary) - analyse results and potentially reformulate hypothesis - repeat!

Science requires that your hypothesis is falsifiable. It isn’t about proving we’re right, it’s about proving we’re “Not Wrong Yet”

(Popper quote) The role of evidence is to:

Arguing based on opinions can easily go nowhere. Arguing based on objective data can be

Metrics for Pirates

Dave McClure

(these are for a public facing website)

Acquisition: users reaching your site Activation: getting visitors to go further Retention: getting vistitors to come back Referral: getting visitors to recommend the site Revenue: getting visitors to pay you!

Tools - how do you gather data? Tests - what data do you measure?


Zappo’s: selling shoes on the internet! Took photos of shoes in stores Set up the online store Bought at retail and shipped them Low overhead way to test if people would buy shoes online Return policy dealt with “What if the shoes don’t fit?” Figured all this out before expending a lot of capital

Pets.com: spent the money first, it didn’t work!

Building a great product

Traditional: - Great idea - build it - market it - profit!

Not so much: this often fails

The Structure of Scientific Revolutions

Normal progress (refining the existing models) vs revolution (rebuilding the models on a new foundation)

Optimisation (doing an existing thing better) vs Discovery (building something new)

Customer development

Searching for a busines - discover customers - validate customers (will/can they pay?)

Growing a business - create customers - scale company

Desirable, feasible, viable

  • is it worth doing?

  • can it be done?

  • is there a business model?

Figure out which of these is the biggest risk for any idea

A “Pivot” is a change in direction during the customer discovery and validation phase


More Popper: great scientists have bold ideas, but are highly critical of their own ideas. They try to find whether their ideas are right by trying first to see whether the are, perhaps, not wrong.

Q & A

Data vs knowledge - e.g. visitor -> download conversion may go down if the search page is bad - can refine an experiment (e.g. only use referrers that are almost certainly trying to download) - balancing trade-offs (e.g. ad revenue at source forge)

Low volume testing? - how do you test when you don’t have traffic yet? - initial focus needs to be on the acquisition step - need a lot of data to distinguish between 2% and 3% conversion rate - but 0% vs 1% is easier to figure out (and is where you can start)

How to explain the need for this? - jackass answer: get the hell out! - explaining the scientific method to someone that doesn’t get it is hard - in the discovery phase, “it doesn’t matter” is a common answer

My Thoughts

Wondering how you could apply this in an intranet context…

Comments powered by Disqus