Search Quality Raters: the Human Eyes behind Google's Algorithms
You know how search marketing experts are always saying that websites should produce content for people and not search engine bots? Well, here's yet another reason to follow that advice: Search Quality Raters. These are individuals that are tasked to evaluate search results and websites to determine their relevancy to queries. The existence of these raters isn't exactly a secret, as there have been several reports about them before, thanks to some leaked Google documents.
Leave it up to Search Engine Land to conduct an interview with an actual Search Quality Rater. Last Friday, the popular SEO blog published a Q&A with this person that details how Quality Raters are hired, how they are trained, and most importantly, how they evaluate search results.
Check out the details of their interview below:
Who are they?
Google contracts and outsources several of its positions, and Search Quality Raters are no different. According to Search Engine Land, the position goes by a number of names, such as “Search Engine Evaluator,” “Social Search Engine Evaluator” and “Search Quality Judge.” The contracting companies that Google works with include Lionbridge, Leapforce, and Butler Hill.
According to the Lionbridge website, the position doesn’t require highly technical skills, but familiarity with Internet browsing, English speaking and writing fluency, and high-speed access to the web are mandatory. Search Quality Raters may come from different countries, so proof of English competency must be submitted. The position doesn’t require any specific background, either. Lionbridge stated that people who apply can be college students, teachers, retirees, stay-at-home parents, or professionals from any field.
Quality Raters work from home and contribute about 10-20 hours per week. Prior to gaining employment, these individuals are required to take a couple of tests to determine their competency for the position, and once they are hired, they undergo a training process that involves a webinar and some training modules.
How do they rate search results?
First of all, it’s important to note that Search Quality Raters aren’t there to rate the web and their position has nothing to do with SEO or rankings. Rather, they’re only to meant rate the effectiveness of Google’s algorithms.
The interviewee told Search Engine Land that when they rate sites, they view search results in a Side-by-Side format wherein one side shows Google’s old algorithm, while the other side shows search results that applies an algorithmic change that’s being tested.
When rating search results, they are tasked to categorize websites depending on their quality and relevancy to keywords. Search results fall into a number of categories, including “Vital,” “Useful,” “Relevant,” and “Off-Topic or Useless.” As an example, the Quality Rater said that for the keyword “Nike Women’s Running Shoes,” he rated the official Nike website as “vital,” since it would be one of the most relevant results for the given keyword. On the other hand, he labeled other sites that sold shoes as “useful.” He also saw a Wikipedia entry on Nike, and he rated this as only “slightly relevant” because he didn’t believe that most users who searched “Nike Women’s Running Shoes” were necessarily looking for the history of Nike as a company.
Needless to say, Search Quality Raters are also tasked to watch out for spammy content and sites that have hidden text, sneaky redirects, and other red flags. Aside from content though, there are also design tasks that are meant to determine “if the page has a good ratio of main content, supplemental content, and ads.” The Quality Rater furthered that this task “also asks about the overall design, is it easy to read, clear communication of information, and the like. It’s not about whether the page is beautiful or amazing, but whether or not the normal user could find what they need on the page without getting lost.”
Worried that the fate of your website is left in the hands of just one individual? Don’t fret. The Search Quality Rater noted that there about 6 raters looking at the same search results. Before a rating is officially submitted, the assigned Quality Raters discuss and deliberate websites through of the use of a commenting system. If one person rated a website differently than other Quality Raters, then they would have to debate until they reach an agreement. Of course, opinions may differ, and there are occasions when it is difficult to come into a consensus. In these events, a moderator will take over and will choose a rating based on the comments of the other raters.
Indeed, it’s nice to learn that Google isn’t relying solely on algorithms and bots to determine search results. Additionally, it is also quite comforting to know that Quality Raters are from all over the world, instead of simply coming from Google’s own Mountain View offices. The search giant has made some poor outsourcing decisions in the past, but consider contracting Quality Raters from outside of the company one of Google’s wiser choices.