Why crowd testing is changing software testing forever

By | February 10, 2014

It was in 2003 that I started my career as a software tester at Gameloft. I had no idea then that it was going to lead me to founding one of the world’s most innovative crowd testing company.

As a tester, I strived to excel and find every bug in a software product. My first project was Splinter Cell Chaos Theory for the N-Gage. My colleagues most likely won’t admit it, but I reported over 3,000 bugs over the course of the next 6 months.

Splinter Cell Chaos Theory N-Gage


Starting out in software testing had a tremendous impact on my career and how I approach software development. In the many jobs I’ve had after being a tester, testing was always on my mind.

As a project manager, I always took the time to double-check every SKU my team was shipping. Later when I founded my own company, it was our reputation that was on the line, so quality assurance testing took on another meaning.

It was in 2011, 8 years after that first day in Gameloft’s Montreal studio that I set my sights on founding the world’s best crowd testing company.

The increasing complexity of mobile apps is what made crowd testing a necessity

In 2003, we were developing games in J2ME. They had to be adapted for thousands of different phones to ensure their availability worldwide. We had to devote hundreds of testers to ensure the quality of the different versions of every game on every device. It was a huge challenge that made it impossible for smaller, independent companies to enter the market.

When the iPhone was released it came with a huge promise from Apple: build your app once, and millions of users will be able to purchase it, without the necessity to adjust it for another platform or screen size.

And while Apple has somewhat respected that promise, Google never committed to the same philosophy and now Android is giving developers worldwide a run for their money when they attempt to build quality applications for every make and model of Android devices out there.

In 2010, my company published a fantastic Android app for Canada’s national TV station. It was one of the first app on the market that enabled users to stream a wide range of TV shows on their devices, anytime anywhere.

At the time, we had completely underestimated the challenge that fragmentation represents, especially in the mobile video space. We tested the app on the few Android devices that we had available, launched it, and then… the horrible reviews started pouring in.

The app wasn’t working for the majority of users. As a company, it made us look really bad. We had made the mistake of attempting to support the many video players available in the Google Play Store. It was ambitious and foolish, but the worst idea was launching without exhaustive testing.

Outsourced testing simply can’t keep up with crowd testing

This particular app was going to be used by millions of Canadians across the country on a huge number of different Android devices.

Traditional outsourcing companies cater to software developers by offering consulting services. Developers purchase a set number of testing hours and in return, the software testing consultancy devotes one or more employees to test the product for this period.

This methodology begs the question: what happens when you are targeting a nationwide community of Android users on several hundred different Android devices and versions of Android OS? Can a single software testing company really possess every single combination of Android hardware and software on the market?

And how much would it cost to have them test your app on these hundreds of devices if you are paying on an hourly basis?

Enter crowd testing

Software testing in real-world conditions by real-world users. Get hundreds of users to test your app for a fraction of the price of a traditional outsourcing company.

It makes so much sense.

Today, crowd testing has become a part of a healthy software testing process.

Of course, the ideal software testing process begins internally. But after the initial testing, and before launch, crowd testing has become a mandatory step to ensure the quality of web and mobile products.


Category: Quality assurance testing

About Simon Papineau

Simon is the founder of Crowdsourced Testing. After 10 years in interactive software development, he set his sights on building a world-class crowdsourcing platform to facilitate the software testing process for developers.