BA 850
Sustainability Driven Innovation

Beta Testing

PrintPrint

Beta Testing

A Page From the Tech Industry

beta symbol

We are going to talk later about a derivative of Beta testing which is a bit different from those applied by the tech industry (more specifically, the software industry). Nonetheless, I would like to offer a primer on the "true" Beta testing.

"Pure" Beta, as applied in tech, tends to be more heavily weighted toward finding bugs and flaws in the product more than garnering live, revealed preference data on how the market receives the offering. Because this desire toward usability and performance testing tends to be the emphasis in the tech industry, their Betas tend to have little to no emphasis on understanding purchase behaviors, instead recruiting participants through either "closed" or "open" Betas.

Closed or "by invitation" Beta tests are as they sound: They are closed to the average Joe, and instead rely on either an outbound invitation from the organization, or for you to apply for consideration for a Beta slot. You will sometimes see this structure implemented when the offering being tested is of a sensitive or confidential nature, when a company wants to hand-pick a group of Beta testers based on past purchases or behaviors, or when they simply want to limit the number of participants.

Open Beta tests allow virtually anyone to participate, perhaps with minimal barriers to ensure the software/product will be used in the proper environment. This testing may allow free and open download of a Beta software version and simply follow up with all users as to their experiences, or in the case of limited physical products, may allow the first 500 or 1000 participants into the Beta before closing it.

As we will see, depending on where we are in our offering development process, we can apply the Beta logic in a wide variety of ways to meet our learning needs at that specific time. For example, if we were early in the offering development and had a prototype we needed early feedback for, we would likely lean toward a closed Beta with customers or organizations with which we are familiar to get some of the prototypes into the field and see how they perform. This would provide us fast feedback without having to recruit new and unproven participants, etc.

Example of a Closed Hardware Beta

As an aside, the below is from one of the most well-executed closed betas I have seen in quite some time. It was for the Steam Machine, a new gaming console from an established content provider.

The specific byproduct I would like to point out here is how Betas, especially closed Betas, can be a fantastic engagement tool for customers and prospects alike.

In the case of the Steam Machine, there were hundreds of thousands, if not millions, of people not among the 300, who were vicariously participating through hourly updates and postings as these mysterious crates began arriving at homes. While I am personally not a gamer, I followed this story in 2013, as it was a fascinating example of what a beautifully deployed Beta can do in a high-engagement group.

If it is any evidence, the video below of unboxing the Steam Machine Beta has almost 500k views. (Feel free to scrub through the following 7:23 video to see how the Beta was presented.)

Video: Steam Machine Unboxing (7:23)

<
Click here for a transcript of the Steam Machine Unboxing video.

Well, it finally came in today, the Steam Machine. Hi my name is Ellis otherwise known as Oolen on the internet, and I was one of the few lucky people selected by Valve to receive a Steam Machine. It's only in its beta state, but I'm pretty sure it's indicative of what the final version is going to be. So let's crack this open and see what we got. Here's the unit itself. Open this up. Now I'm pretty sure everyone knows what this thing is, but for those who don't it's Valve's attempt at bridging the gap between console and PC players. You play your games on your big screen TV in the living room instead of on your 20 inch monitor. Wow, here it is. It's got audio jacks, it's got USB 2.0 ports, looks like some vga seta. Basically this is a mid, mid to high-level PC, so it is expandable but like i said it's mostly console experience in the living room. It runs natively on a Linux-based operating system called Steam-OS. Now, here's the unit side by side with an Xbox. They're similar in size, but the steam machine is quite a bit heavier. Side profile right there so comparable, but a little bit bigger than the old Xbox.

Here we have the controller and definitely tell this is a beta controller just because the, the plastic feel and I, I believe the concept images had some sort of a touch screen, but here you have four buttons. It's got to concave trackpads as opposed to analog sticks and you know what it's, it's pretty comfortable. It's got a second micro USB port right there and then it has these two buttons we can click right there. I'll make a separate video about the control later. Here you see it next to a dualshock 3 controller. It's quite a bit larger. The weight, it's it's about the same it's a little bigger, so the weight is distributed a little more so the PS3 controller feels heavier. The weight is concentrated more in these wings right here. it's quite a bit lighter in the center, but there's that right there for comparison.

Here we have.. what's this? Thank you for shaping the future of Steam. Your feedback will refine Steam-OS, and this is just an outline of the system. Wi-Fi where that is, if you don't know how to use a HDMI cable this will help you out. Here's some important information just how to place it and you actually place it a horizontal not vertically like I placed it that's why it's good to read the instructions. So let's place the right way which is that. Ok, power cables, HDMI cables, no, this is USB cable, I'm sorry. USB power. I don't know what that is, but I'm sure I will find out soon. And then Steam Operating System recovery, there you go. Seems to be it as far as that goes. Well let's plug this thing in and see how it runs.

So here's what the device looks like turned on. This button right here is actually a button you push it's not a just like a touch and on the side right here it says gforce gtx. So I'm interested to know if you change out the graphics if it will indeed update or this is even customizable at all. So that is cool. Now here's our login screen to log into my account. oh geez, I'm going to have to edit this out so you don't see my password, but this is how you input text actually it's a good time to tell you. Now the controller it takes some getting used to, but I can tell that with some practice you can get some pretty good precision on there. Now here's the main screen it's a pretty streamline. It looks similar to you know, your Xbox dashboard, PS3, you know typical console. Now we can go to our library, view all games. i can see i have almost two hundred fifty games, but not every game you could install from the start. Like I was interested to see how something like Company of Heroes would play on the controller, but we can't have that yet, but I am installing a few games. Going kind of slow, I don't want to install Dota though. Actually I do because that's a good test of the controller Super Meat Boy, Serious Sam, Faster than Light, FTL, Painkiller, and Hotline Miami. A lot of these games such as these two were gifted to me by Valve to test out the machine, but that's it for this video. If you want anymore just tell me in the comments down below. If you want to know how any of the games play. I plan on doing at least one video showcasing what the controller can do on different genres such as shooters strategy games like DOTA and games that require precise platforming like Super Meat Boy.

Well, that wraps it up for this video if you liked it subscribe and favorite. If you didn't like it, subscribe anyway because I have other videos that you might like.

Credit: OolenGamingOmniverse. "Steam Machine Unboxing(link is external)." YouTube. December 17, 2013.

Six Steps of Beta

If we are considering Pure Beta as applied by myriad software companies, the following offers a simplified view of the six steps of Beta.

From the Centercode Beta Testing Process(link is external):

Step 1: Project Planning

Before beginning a beta test, the objectives of the project must be defined. It's common for the number of unique goals in a beta test to range from just a few to upwards of 20. Defining these goals in advance ensures that the appropriate number and composition of participants are selected, an adequate amount of time is available, and everyone involved understands what needs to be accomplished.

Step 2: Participant Recruitment

Beta testing begins with the selection of test candidates. The ideal candidates are those who match the product's target market and whose opinions won't be swayed by a prior relationship with the company. Most private beta tests include anywhere from 10 to 250 participants. However, this number is highly dependent on the complexity of the product, the audience involved, the time available for testing, and the individual goals you'd like to achieve.

Step 3: Product Distribution

Next, products are distributed to beta participants. The focus of a beta test is to understand the customer experience as though they purchased the product themselves. With this in mind, beta is most effective when a complete package including all appropriate materials (software, hardware, manuals, etc.) are sent to participants.

Step 4: Collecting Feedback

Once your participants begin to use the beta product, feedback needs to be gathered quickly. This feedback comes in many valuable forms including bug reports, general comments, quotes, suggestions, surveys, and testimonials. With good beta management and communication tools, you can get a lot of feedback from test participants.

Step 5: Evaluating Feedback

A beta test provides a wealth of data about your product and how your customers view it. However, that information is useless unless it's effectively evaluated and organized to be manageable. All feedback should be systematically reviewed based on its impact on the product and relevant teams.

While bugs are often the core focus of a beta, other valuable data can also be derived from the test. Marketing and public relations material, customer support data, strategic sales information, and other information can all be collected from an effective beta test.

Step 6: Beta Conclusion

When a beta test comes to a conclusion, it's important to provide closure to both the project and the beta participants. This means providing feedback to the participants about their issue submissions, updating them on the status of the product, and taking the time to thank and reward them for their effort.

Weakness of Pure Beta

As you can perhaps imagine, having a closed pure Beta with 250 participants as described would provide some extremely useful feedback from users on the software, usability, instructions, and the entire use experience.

What it would NOT help us understand is any revealed preference data on our offering, and if we were to use Beta testing to try to understand market preference, we would have quite a few issues with this methodology:

  • Betas of this type are essentially always free, or, in some cases, participants may be compensated with product or vouchers
    • Issue #1: No Dollars for us to understand real preferences and behaviors.
  • Betas of this type usually have specific guidance and instructions on how to use the product, and what feedback is desired at each step.
    • Issue #2: Minutes of participant attention/use are therefore skewed because we are setting behavior, and not letting users find their own way.
  • In the case of closed Beta, the organization will be responsible for the composition of those testing the product.
    • Issue #3: Bias may be introduced into participant selection in a variety of ways, from heavy early adopter loading to those who would likely have a favorable opinion of the organization and its products. Furthermore, it would be difficult for us to choose users in the target market if we have not yet proven what that target market is.

For those reasons, in our next topic, we will explore a philosophy which has foundations in Beta, but is better suited to our needs in testing the proposition and offering in the market quickly and inexpensively. We may call this philosophy "microtesting."

Five word summary - Hard fit when testing offerings