Putting Digital Experiences To The Test

Eight years ago, the mobile industry was a very different world. The iPhone was two years old, and the App Store was just getting off the ground.

It was also around that time that two guys in the software and tech world recognized that the millions of consumers with millions of mobile and digital devices and thousands upon thousands of apps with new and complicated functionality, all needed to work if consumers were to embrace these digital experiences. That meant bug free and a consistent experience across devices and geographies and operating systems.

That’s when these two guys came to the conclusion that the traditional way of software testing wasn’t going to cut it and created a platform, Applause, that matched companies with a desire to create a great digital experience with software testers who’d put the software through its paces to make sure that it delivered as promised.

In the most recent episode of The Matchmaker Is In, Karen Webster spoke with Applause’s SVP for Global Delivery and Customer Success, Rahul Shah, to get a better sense of how this platform got its start, how it has scaled and how it keeps the integrity of its platform by putting its testers, well, to the test before they even see a line of code.

Shah said that the Applause founders developed a two-pronged approach to getting its platform off the ground.

“First, our founders started building our platform. As they started building it, they recruited testers from their inner circle and paid them to test the Applause platform. That gave them familiarity with our own platform and how it worked. At the same time, they began talking to various startups — which resulted in 20 to 30 companies signing on,” Shah said.

With that came more testers referred by this inner circle, and with more testers came more startups. After Applause reached its 100th company, the flywheel really kicked in, and it began getting between 300 to 500 new testers each month.

Here is an edited excerpt of Webster and Shah’s conversation:

KW: You are matching testers with app developers that actually need to have their apps put through the paces before they release their apps into the wild. Sounds like an obvious problem to solve. But there must be more to it than that — tell us.

RS: Knowing that the world is moving to a more integrated intersection between technology and human behavior, optimizing those experiences has to happen through a combination of market research, usability testing and functionality testing. Most companies still seem stuck in a very traditional paradigm of developing code and handing it over to people who literally sit in an office banging away at a few devices to test that code. But that’s not the real world now.

It’s really key, we think, to test that software in the real life environments they want to enable, so that when they ultimately release that code to their consumers, they know with a very high level of confidence that it is bug- or issue- free. That’s the friction we solve for.

KW: Describe the platform.

RS: The platform sits on top of our community — we are the intermediary between the company and their need to create and deliver a great digital experience and the testing community and their ability to allow them to do that.

KW: Are these testers independent contractors or employees?

RS: Our testers are independent contractors. In many cases, they may be software developers looking to supplement their income. We give them the opportunity to test across a variety of different experiences that they otherwise wouldn’t get exposed to. We’ve found that our community of contractors are very engaged with us because we offer a way for them to exercise their professional passion in ways that are more exciting for them than doing the same thing over and over.

KW: How do you prevent conflicts — so a tester who’s given the keys to rummage around the software of their competitor?

RS: We have a very rigorous process of onboarding testers into our community, and we have an equally rigorous process of identifying and allocating testers to specific projects. When a person wants to onboard with us, while we want to make it as frictionless as possible, we do make them take a sandbox test, which makes sure they have the skills to do what our customers need. In addition to that, we make sure that with every engagement, that we select individuals where there is no conflict of interest. Through a vetting and contractual process, we make sure they don’t engage with any conflicting work. Our customer success team is responsible for these engagements and selecting those testers.

KW: How many of your 300,000+ members do this kind of software development work as independent contractors and how many are employees of big companies that are bored and need some sort of intellectual stimulation of doing something like this?

RS: I think there’s a healthy split of both which allows us to meet our customers’ needs. There are situations where our customers need full-time dedicated support with 40 hours per week, and we have access to plenty of those individuals. There are other times where our customers need … some specific experiences in very niche circumstances.

KW: How do you continue to validate the effectiveness of your testers? Are they scored or graded after they complete their task? How do you make sure your testers are performing to their optimal capacity?

RS: When a customer engages us as a company, we then go with our customer success team and find the appropriate tester to be able to execute on that customer’s need. Our community is paid on the quality of bugs that they find, which are reviewed and rated by customers. We have developed a pretty intricate algorithm that rates the quality of our testers against other variables, with the input that our customers tell us about the quality of bugs found. Based on that, we’re able to promote testers into different classifications, such as bronze, silver or gold.

KW: Is that also how they’re paid? Are they paid on the sophisticated-ness of the bugs they find?

RS: Absolutely. That creates a very healthy business model. When we introduce work into the community, it is in the best interest of the community members that are part of that particular project to be able to jump onto that exercise as quickly as possible so they can find the … highest value bugs they can so they can get paid the most.

KW: It sounds like it’s free for testers and companies must pay for access to the platform. Is that more or less how it works?

RS: Yes, absolutely. It’s a SaaS model where companies come to us and have a need. We want to build long-term relationships with our client base. The SaaS model allows us to fulfill those needs throughout the community.

KW: How has the demand on the company side changed since the beginning, and how has that changed your recruiting for testers?

RS: What we started off with was to recruit testers to fulfill the backend of the software development lifecycle. When companies came to us eight years ago, what we focused on was to be able to disrupt the way traditional software testing was done. What has evolved over time, and specifically in the last few years, our customers began pulling us up further upstream in the software and product development lifecycle. What started off as being the back of the back, we got engaged in usability and more in market research.

KW: Companies must think they’ve gotten introduced to some amazing testers through this service and want them to come in-house. What keeps you from being left out in the cold?

RS: It’s the value proposition we offer fundamentally to customers. We have continued to build our customer success team on top of our platform and community to go beyond the mechanics to provide them deeper insight, value and strategic perspective.

KW: You guys have built a very interesting matchmaker business. I’m curious to get your thoughts on what matchmaker you admire. And you can’t say Uber — since everyone usually does!

RS: For me, the other companies that are equally as large and well-known is HomeAway, VRBO and Airbnb. What they did to disrupt the vacation industry has been phenomenal. As an individual, I have taken advantage of that when I’ve taken my family on vacations across the globe to a point where our experiences are so much richer because we’re not staying in a generic hotel.

Not only on that side but on the other side — when I do go on vacation, there are times when I allow people to stay in our home using the same platform. What I’ve found is that those folks take an equal care of our home in a way that allows others to share the same joys that we share when we’re elsewhere. It’s a great example of community.