Running a successful beta – trickle effect and phone feedback

When you’re heavily involved with a product that you’ve been developing, you expect that as soon as you launch it, millions of people will immediately try it out. Unfortunately, unless you’re Google it is highly unlikely that will happen. Nobody knows about your product and it takes a lot of effort to promote it and sign up users. In fact, in this post I’m going to explain why it is better to get a small number of users who love your product and then build on that.

Knowing when to release a product is difficult. The maxim “release early, release often” certainly applies but that has to be balanced against making sure you have a minimum viable product and features that work well, with a minimum number of bugs. You also need to get feedback as early as possible to either completely change what you are going to do or introduce new features you would never have thought of on your own.

Customer driven development is central to how we develop and our server monitoring application, Server Density, is being built that way. We started off with an idea of what we wanted to do based on what competitors were doing wrong. Once we reached the basic feature-set, we launched. This is the best time to get real feedback because the product is small enough to be able to change yet still useful to users.

Beta Stage 1 – Very private

My background is in software development and web hosting so I already had a number of contacts I contacted the first day the beta launched. I sent them a quick e-mail asking if they would be interested in trying it out. This is important – you shouldn’t send invite codes to everyone because most won’t want them or will just forget. You need to ask the user if they want to try it and make a commitment to doing so once you send them a code. It won’t guarantee that they will but we had an 84% response rate doing it this way.

Equally, the product you are asking people to test should be relevant to their interest or job. I contacted people I knew who run web hosting companies or have servers that could actually directly benefit from the product.

Beta Stage 2 – Private

You now need to approach people who you don’t know. My main method for doing this was through my followers on Twitter. I sent out updates about how the beta was progressing, what I was working on and direct, unsubtle “server monitoring, try it out” messages. You need to keep this up over a period of time because a lot of your followers will miss your messages but you shouldn’t spam; messages should be interesting on their own and should not be the only thing you’re posting.

Even in this stage, I asked people to contact me first and to make a commitment to providing feedback. This is the whole point of the beta so you need to be sure that those who actually sign up are aware you’re expecting feedback. If they don’t want to, that’s fine – they can wait until it launches publicly.

We ran this for several weeks getting detailed feedback from users who were involved. This is the best time to build a good relationship with a small number of users who are already linked into your product development. The best possible outcome is users who start promoting for you. We were lucky to have this happen we we launched our public beta.

It is important to invite only a small number of people at a time – the trickle effect. If you have too many users you won’t be able to co-ordinate the feedback (see below).

Beta Stage 3 – Public

Opened on Monday (6th April), this is where we are at now. It was timed ready for my pitch at the TechCrunch Pitch Workshop event on Tuesday (7th April). If you are going to be at conferences or giving talks then it is a good idea to allow people to go to your site and try it out right away.

It is important to remember that people who sign up to an open beta probably won’t provide feedback. This is because they’re more likely to be testing it out to see if it’s any good and don’t actually care about helping you out. They have no involvement with the project and no prior interact with you so why should they? Even so, you will want to get in touch to ask them.

We’re still trying different tactics for getting users:

• Engaging with people on Twitter. You can use the search to find relevant keywords and @reply to people who are talking about those topics. This is quite close to spamming but is relevant and contextual – I liken it to the Google ads based on your search terms. However, it is important not to overdo it and make your message interesting and relevant to what the person you are targeting was talking about. We have had signups as a result of this.
• We are running a small budget AdWords campaign for a set period of time to look at click throughs and conversions. This can be expensive depending upon your market and keywords you are bidding on and so far, has not worked out that well for us.
• Submit your site to gallery and review websites. More information about the benefits of this are described here. We generated several hundred uniques from just one submission to feedmyapp.com. Of course, this is less targeted traffic and so is less likely to result in conversions.
• Go to meetups and pitches to talk to people about your product. I gave talked to and gave my business card out to almost everyone at the TechCrunch Pitch event and then followed up with everyone via e-mail the next day. E-mail is most likely to be considered spam so I was careful to spend some time visiting each person’s website to give them some feedback on that as well as their pitch the day before. Mentioning my company was a single line at the end of the e-mail. This is a good way to build long term relationships so if they don’t need your product now, they will still remember you in the future. This has resulted in some signups and I’m going to be meeting up with some of the guys at the next event we’re at.
• Ask HN. If your target market is technical then post an Ask HN thread on news.ycombinator.com asking for reviews of your product. This is not something I’ve done yet but do plan to. They appear quite regularly and seem to give good feedback in the comments.

Feedback – it’s all about phoning customers

If you don’t get feedback from users then there is no point in running the beta. I have tried e-mail and survey based feedback in the past but for Server Density, I have been phoning customers directly. This has proved vastly more successful in receiving quality feedback. I start with a couple of set questions I want to ask all users but the natural flow of the conversation usually leads onto discussing other areas, finding out more how the customer is actually using the product and the best bit – whether they sound genuinely excited and interested by what you’re doing.

It is incredibly rewarding to talk to people who are using your software and loving it! Even if you don’t get anything useful (we have from every call), it will give you more incentive to continue working on the product.

In addition to the benefits you get as a developer or business owner, it increases confidence in your company. People are used to dealing with faceless corporations and call centres so having the lead developer or company founder call you generates a level of trust. We’ve had feedback to this effect – people telling us how unusual it is to have a company care enough to talk to users, even when they’re not paying.

As an example, one of the first users I spoke to suggested what became the Server Snapshot feature which was mentioned in a call I had today with a completely different user as being an excellent feature that they were able to make use of in the last few days.

The steps I follow for each user are:

1. E-mail the customer to ask them if they would like to give us some feedback via telephone. I always say it only takes a few minutes and that I will call them. I ask them when would be convenient. If the call runs longer then that’s fine. Most of the calls I’ve done have been about 5-10 minutes, with one for half an hour.
2. Having arranged the time, I e-mail on the day just to remind and confirm.
3. I call the number they have provided at the time agreed, introduce myself and check to make sure it is still convenient.
4. Run through my prepared questions, taking notes and chatting to the customer off topic where appropriate. I try to stick to my structure but being flexible is important to allow the conversation to flow naturally.
5. Once finished, I send a thank you e-mail and mention a few things that were discussed. For example if the customer suggested a feature, I explain that we’ll consider it and if it is already being developed, when we expect it to be done.
6. I always make a note of every feature requested in our company wiki along with the customer who requested it so if we do implement it, I can tell the person who originally suggested it.

Questions I ask are:

• General impressions. This allows the user to talk about anything they like relevant to the product and often answers my later questions. This makes the call flow nicely and gives you the chance to talk about ideas and make notes as you’re going along.
• If relevant, ask about their specific usage of the product for example if they haven’t set up any alerts, why not.
• Ask what they were using before they started using Server Density. There are competitors that are likely to already being used so it is interesting to know who you’re up against. The important thing is to find out how your product compares to the competitor they switched from (or are using as well).
• Any suggestions for new features, if it wasn’t already discussed in the first version.
• Some statistical information about the OS / browser they use.
• Check if they want to say or ask anything else before finishing the call.

Managing users / feedback

When you are in private beta, it is important to keep following up users to get feedback and check how they’re doing. This is fine to do because they have already agreed to give feedback and given their permission for you to contact them. We usually follow up a week after signup to allow for time to use the product.

We use Highrise to keep track of all beta users and when we need to follow up. All communication is handled via FogBugz and notes are entered into a single case for that feedback call. 37signals ran a case study on our use of Highrise which goes into more detail on this.

In summary

Feedback is the ultimate goal. The strategy to achieving it is to have a small number of users you manually increase over time. Keep track of all communication and when you need to follow up with people and make sure you talk to customers on the phone.