| Beta Testing: Almost Ready for Prime Time |
|
|
|
Of all the tools available to help guide technology product managers through the often mystifying process leading to product launch, the ones that matter most are based on real people making real use of the product – even in prototype form. Beta testing, in which early versions of technology products are put into the hands of selected customers, has become a widely used and highly valued technique for finding out what you want to know – and sometimes what you don’t. The use of Beta sites by two very different technology firms illustrate the inherent flexibility of the technique.
The dirty little secret shared by technology product managers everywhere is that the product development process they preside over is at least as much an art as it is a science. While extensive research involving consultants and industry experts is often conducted, and detailed analyses of the findings are frequently performed, deep down inside, product managers often harbor feelings of uneasiness. That’s because they know that the most important test of any product comes when it hits the market and is used by real customers in an authentic work environment. At the same time, products that go to market without adequate testing can yield unwelcome, and sometimes fatal, results. With products as with people, you never get a second chance to make a first impression. So to reduce the risk inherent in product development, many product managers employ Beta testing. In fact, Beta tests have become such an important technique that most product managers consider them to be an integral part of the product development process. Take the case of McKesson Automation. There, according to Product Manager Jim Longo, the requirement is that products be reliable – right out of the gate. Longo, who has been in the field of product management for many years and has held positions with FreeMarkets, Marconi Medical Systems, and Meridia Health System, understands the importance of getting it right the first time. “Just about all our products are mission-critical,” he said. “Having failures and reliability problems significantly disrupts our customers’ operations.” That’s where Beta testing comes in. Really, really real Beta tests allow product managers to tweak their products before they go into production and to gain insight into how they’ll fare once they’re launched. Unlike launching a product based on theoretical formulations and assumptions, Beta programs allow managers to field-test their products in real situations with real customers doing real work. The lessons learned can be very real, as well. But there is more to a Beta test than simply dropping a newly developed product in the customer’s lap. There must be specific goals, objectives, time frame, and a clear definition of roles and responsibilities for both the vendor and customer. In fact, the supplier often has a specific Beta team assigned to ensure that a quality test and results are achieved. That mirror team of the Beta process, created inside the developer’s own organization to monitor test results, can be critical. “Our Beta teams are comprised of representatives from all disciplines related to the Beta offering,” Longo said. “If it’s a new version of our software, Connect-Rx, specific teams such as Development, Implementation, and Integration Testing are involved. If it’s hardware, Engineering and Manufacturing also get involved. The goal is to identify unanticipated problems and collect customer feedback on functionality before moving the offering out of Beta.” Which are the best customers to use in a Beta test? According to Longo, “those customers are almost always identified through sales channels. Most are current customers who want us to expand our product portfolio.” But there are other motivations for becoming a Beta site as well. McKesson also has offered customers a lower price, along with the new product features, to entice them into becoming Beta sites, he said. Just testing Cathy Brennan, Product Manager for IngMar Medical, Ltd., has been involved in product management since 1995 and with IngMar for a year and a half. She, too, sees Beta tests as an integral part of the development process, especially for smaller firms. IngMar, which does not have a dedicated sales force, maintains a close relationship with its customers, many of whom are R&D firms that already understand the development process. Constant communication between IngMar and its customers generates lists of features for product evolution, Brennan noted. In addition, IngMar sends out customer satisfaction survey forms twice a year to assess customer satisfaction and discover requirements for future products or enhancements. As with any test, the number of participants must be limited so the test can be controlled. IngMar uses three to four customers for each Beta while at McKesson, there may be a few as three or as many as ten customer Beta sites. This requires the product manager to work closely with the sales force to identify customers and participate in the process. It is also critical to make sure the process goes as smoothly as possible. “Salespeople are often concerned that their customer will receive something that would be highly problematic or unreliable, and that would reflect badly on them,” Longo said. Salespeople at McKesson are not compensated for Beta tests; the primary point of contact for Beta customers are project managers and strategic account specialists. Once prospective Beta customers have been identified, goals must be set, objectives identified, and lines of responsibility defined. IngMar employs an informal process where most communication is conducted via phone and e-mail. That’s because the IngMar software is mature and most of its product launches and enhancements involve “tweaking” the company’s current software. At McKesson, on the other hand, where entirely new product lines are frequently introduced, a more formal process is used. “It sometimes takes more time than we’d like to get a Beta customer signed up,” according to Longo, whose primary customers are major hospitals. “Especially when the customer hasn’t been a Beta site before; the Beta agreement causes a lot of anxiety for ‘mainstream’ customers.” So the company uses a formal document that clearly defines all aspects of the test including goals, objectives, time frames, customer responsibilities, company responsibilities, liabilities, and lines of communication between vendor and customer. At IngMar, Beta tests usually run about one month, according to Brennan. At McKesson, the tests average about six months. Unexpected findings No matter what their length may be, every Beta test experience can yield a wealth of important lessons, and they are not just limited to the product features which may have been the original basis for setting up a Beta site. Product support issues are also critical. “We always look to verify that our field service organization and call center can get up to speed and support the product so that the customer perceives us as knowledgeable and responsive,” said Longo. But the ultimate lesson may be how the customer perceives the product and how well the Beta test goes. According to Longo, success is judged according to “whether we were able to smoothly manage the Beta process, more so than if the product is ultimately released. We use an internal scorecard based on identified problems, resolution, overall feedback from the customers. A major factor in the success of the Beta test is through constant communication with the customer. But we set the expectations upfront about the potential challenges associated with being a Beta site. We manage all ‘incident reporting’ electronically so it’s relatively easy to track the frequency of problems.”
ABOUT THE AUTHOR: |
