At Hopscotch we were rigorous with our product. We had to be. A small team, and over 10 years of features meant that the product could easily become bloated and confusing—the opposite of our goal.

How did we keep shipping a great product over such a long time period? We developed a process based on the scientific method. Any project, whether it be a community initiative, a new feature in Hopscotch, or changing the way something worked, had to begin with a product doc that included a hypothesis, success metrics, and a plan for failure. 

Answering the questions in the doc helped us approach each change in the spirit of inquiry. It made the project an experiment, not an improvement, until proven otherwise.

The Product Document

Project name

It’s nice to have a fun codename for your project, but you also want something descriptive, maybe as the subtitle. A real example was “WavingCoffeeCake: Account Creation Optimization”.

Overview

The overview is a summary of your changes. It’s analogous to the abstract of a scientific paper. You might write this last, but it’s useful to put first for the readers (who might be you three months in the future). We would come back and add to this part a few weeks later to add the experiment’s results as well.

Company goal

This is where we started writing the actual document. It’s important to ensure that you are always working towards your company’s big goal. The easiest way to do this is to refer back to it constantly. Having it be the first thing in the product doc makes sure that all the other goals support this.

The next three sections are related, and are usually the most challenging part to get right. We’d often go through many iterations and discussions to get to something we liked.

Hypothesis

The hypothesis always had the format: If we take X action/build Y feature etc. Then some good things will happen to the company. Stating these expectations up front keeps you honest. If you start going on side quests or building things that deviate from your hypothesis, your team is going to know and call you out. A couple of examples: 

If we improve the account creation flow Then more people will get through the onboarding process to make their first project. 

If we limit the number of hashtags you can add  Then there will be more distinct and relevant projects under each hashtag. 

Success State

The success criteria is probably the single most difficult section. I hate trying to figure this out. It breaks your brain because you don’t know yet what your numbers will be, so how can you predict them? But if you don’t define your success criteria before you ship the feature and start getting data, you will inevitably find one metric that has changed for the better and call that success. You need to be realistic with yourself about what metrics would  actually convince you that you’re moving the product toward its goals and aim for those. Giving a time frame and using numbers is important. Example: After three weeks, there will be 30% more plays per project for each hashtag for that week than there were in the weeks before the feature was released. 

Plan for Failure

The success state tells you what metrics you want to hit, the failure state tells you what to do if you don’t hit those metrics. I like to list out the ways we could fail and what we will do in those specific cases.  Do you try again and do something else to hit those metrics? Do you roll back the changes? Maybe you just move on and let everything be. It’s your choice what to do, but making a plan for failure forces you to be deliberate about your changes and make sure you don’t overload your product with a million features that didn’t work. It also makes the failure much easier to handle. At least now you have a gameplan! 

Iteration and design

This is the thing you will actually do. It doesn’t have to be highly detailed and specified here, though of course, it does need to be highly specified somewhere (probably your project tracker). I often use this section to write out in prose what we’ll do before transferring it to my project management tool. It’s also nice to put images from your design file if you have one. The goal is that someone reading this doc has a high-level overview of what you’re doing. And pictures are always nice. 

Background

This is analogous to the references section of a scientific paper. It’s the research you’ve done to figure out the answers to all the questions above. Think data, graphs, links to your analytics, etc.

Conclusion

And there you have it. The Hopscotch Certified Product Doc. We’ve used this for years at Hopscotch, and I’ve started implementing it for my consulting clients as well with a lot of success. I would love to see more teams adopt a scientific process for their product development, so please reach out if you have any questions about how this could work for you!

Lastly, if all this has made you curious about Hopscotch check it out for web, iPad and iPhone here.