Delivering a successful beta program can be one of the hardest things to do as a product manager.

Is it just me, or is delivering a successful beta program one of the hardest things to do as a product manager? The difficulty arises because so much is out of the product managers direct control and in the hands of the beta testers themselves. And if you cant get motivated beta customers to provide you with detailed feedback, you can significantly diminish or delay the revenue potential for your product.

Managing an effective beta program means doing what it takes to minimize the aspects that are out of the product managers direct control and, like any other program that is executed, bring as much predictability to it as possible.

I work in an enterprise software company, so our software is quite complex. It connects customers corporate databases and information systems and is considered mission critical by a large majority of those customers. It is important to tell you this because as software can range from very simple to quite complex, so can beta programs. I have managed betas for smaller consumer applications as well as web applications, and while the scale and complexity of the beta programs has varied, they were always challenging to execute well.

Think about it. A beta program involves having people spend their valuable time using your unreleased, and in many cases, buggy software. And, on top of that, you usually have a fairly short period of time to gain valuable insights on what issues need to be addressed with the product before it is released.

What a beta is not

Before I get into the details of what a beta program is and how to optimally manage it, its important to understand what a beta program is not. Its not:

• a substitute for good quality assurance
• a substitute for proper design and usability testing
• a tool for the sales team to help close deals in their funnel
• something that should be confined to the product management or product development groups in a company

First, a beta program is not a substitute for good quality assurance. Sending out an early release to beta testers and counting on them to find bugs that should be fixed is a very bad objective for a beta program. In fact, if your development or QA teams disagree with the previous sentence, then they should realize that what they are saying is that something which is fundamentally their responsibility should be outsourced to a group of external people. The next logical step would be to outsource the entire development or QA function altogether.

Related to the previous point, a beta program is not a substitute for proper design and usability testing. There are two major reasons for this. The first is simply that something that is an intrinsic part of the software development process cannot be left to beta customers to resolve. Efficient user interfaces need to be designed and implemented with an understanding of user needs. They must be based on easily understood cognitive models, support specific workflow patterns, minimize hidden or hard-to-access functionality, and in many cases, be adaptive or configurable to support a diverse pool of users. These things need thought, planning, and sufficient lead time to implement. And thus, if left until beta, almost guarantee that significant rewriting will be needed, possibly causing a delay in the products release date.

Next, a beta program is not a tool for the sales team to help close deals in their funnel. Im not saying that exposure to beta software cant help close a deal, but direct prospect participation in a beta program is a bad idea. Why not give a prospect early access to software to get them excited about what is coming down the pipeline? Well, first of all, you cannot sell beta software and there are easier ways to provide prospects information about what is coming in the next release. This can be achieved via on-site demonstrations, Webinars, or even white papers and presentations. Additionally, simple software products notwithstanding, the cost of supporting a beta site and the risk of exposing the prospect to potentially buggy functionality (it is not fully tested and debugged, after all) may outweigh any advantages or benefits in accelerating the software sale.

Finally, a beta program is not something that should be confined to the product management or product development groups. If a software company is working on a major product release, and particularly if that product generates a significant portion of the companys revenue (i.e. it is not a marginal product or a minor release), then people from across the entire company must work together to set objectives for the beta program and ensure they are met or exceeded.

After all, the beta program is almost always the first external exposure of the upcoming software product. And if this product is expected to drive future revenues for the company, then the value of, and investment in, the program must be looked at from an overall company perspective. Virtually every customer-facing group in the company should be involved in the beta program. Whether the goal is related to technical enablement for groups such as Support or Professional Services, identifying press and analyst references for Marketing, or accelerating time to revenue for Sales by developing early adopter sites; taking a broad, company-wide view of the beta will maximize the benefits of the beta program for the entire company.

Origins of beta programs

In regards to product development, the terms alpha and beta were coined by IBM in the 1960s to label product development checkpoints. Alpha meant a unit, module or component test phase and beta was the initial systems test. These terms came from even earlier A, B and C tests for hardware. The A-test was a feasibility test that was done before conducting any design or development. The B-test was a demonstration or prototype showing that the engineering model functioned according to specifications. And finally the C-test (which maps to the current usage of the beta test), was performed on early samples of the product being created.