appen-rater-job

Appen Search Engine Evaluator

This review is specifically for Appen’s Search Engine Evaluator Remote Job. There are a few different freelancer job opportunities from Appen including Projects, Micro Tasks, and Survey and Data Collection. The search engine evaluator positions are W2 employed positions so you will NOT be an independent contractor and will receive an hourly wage.

There are weekly minimum/maximum hours you must work but it’s still flexible where you can work on your own schedule. There are several different projects under different names which won’t be mentioned due to NDA reasons but if its a search engine evaluator position they pretty much follow the same or similar procedures.

Job Description For Appen Search Engine Evaluator

Appen’s Search Engine Evaluator job consists of several different types of tasks but in general, raters evaluate the quality and relevance of search results shown to a user that’s made a query. When a user types a query into a search engine like Google, a rater must evaluate how relevant the results are in meeting the user’s need as well as the quality of results.

For example, if a user searches for ‘best restaurants near me’, a result of a toy store would be irrelevant. Results showing restaurants in another state would also be irrelevant since it does not satisfy the ‘near me’ part of the query. If they search for ‘scores for Laker game’, results showing the most recent Laker’s game score would be more relevant than scores from a game from five years ago.

Evaluators are also responsible in rating the quality of a web page and how high it’s EAT is. EAT stands for Expertise, Authoritativeness, and Trustworthiness. For example, a website with unsubstantiated claims, click bait articles, no information given on the creator, malicious links, and a bad user interface experience will have a lower EAT rating than a high quality site from a University or .Gov site where the content is backed by evidence, provides thorough information on the author of the content or is from an already known and trusted entity.

Example of low EAT site.

Example of very high EAT site.

The tasks aren’t just limited to rating a website’s relevance and quality. There are many different search result related tasks like judging if content is safe or adult/porn. Rating the quality of an audio speech result. Rating the quality of short answers given in relation to a query. Judging how fresh or stale content is.
Usually, 3-5 different tasks are available to work on, with an estimated task completion time ranging from 25 seconds to 15 minutes per task.

Appen Search Engine Evaluator

There are always plenty of tasks available excluding the rare times where there are no tasks and you must wait (sometimes a few days) until another task dump. This usually only occurs 2-3 times a year, usually around Christmas, New Year, or other holidays. You will not get paid if there are no tasks but you also won’t get penalized for not reaching the weekly hour minimum.

All raters must work at least a minimum of 15 hours a week and a maximum of 29 hours a week (max of 8 hours a day). New raters will usually be limited to an hour or two for the first few weeks and will have their limits increased as long they’re not making too many errors. Most projects have a limit of working 8 hours per day all though some projects may differ.

How To Get A Job As An Appen Search Engine Evaluator

Requirements:

You must be 18+ years of age.
Must live and be able to work in the country you applied from.
Have a smartphone as some tasks require a mobile device.
You must pass a 3-part qualification exam.

To start the application process, go to Appen’s crowdworker page.
Scroll down to ‘Projects ‘and apply. You will need to create an Appen account and verify your email address.

Appen Search Engine Evaluator

Once logged in you will be given the option to complete your profile to gain access to all projects, long-term and short term, which will show all eligible projects based on your location. If you don’t complete your profile you can still access short-term projects (usually one time jobs) but will not see the search engine evaluator projects.

When your profile is finished, you will then be asked to verify if you have a smartphone (you can do this at a later time). Your Appen dashboard is where you will see a Projects tab where you will find all available projects that are hiring. You should see a project name, short description, project type, rate, and task workload (amount of tasks available). Search for ‘Search Evaluation’ or ‘Search Engine Evaluator’.

Select a project and it’ll show you a more detailed description of requirements and what type of work you’ll be doing as well as payment and expected hours. You’ll then have to start the application process again for each specific job you apply to. You can apply to as many projects as you like, all though some projects once started will not allow you to work on other projects.

Appen Search Engine Evaluator

After submitting the application, the waiting game begins. Like many remote, work-from-home jobs, there doesn’t seem to be a definite answer on how long it’ll be until you hear back from the company. From a collection of responses from people who applied on social media and online work forums, some received an acceptance or rejection response within a few days, some stated it took 3-6 months to 1 Year to hear back, and some never received a response at all.

When your application is approved you will get the welcome and on-boarding email as well as information on how to take the qualification exam and study materials.

Qualification Exam

Now that your application has been approved and you’ve signed all required documents you are probably excited to begin working, but there is still one additional requirement you must meet. All candidates must past the 3-part qualification exam. If you’ve already done a little bit of research on Appen projects you’ve definitely heard how these qualification exams are not easy.

The good news is you will be given all of the guidelines and study materials needed in order to pass. You can even have them open while taking the exams. The bad news? The General Guidelines they give you is over 200 pages. It is very detailed and covers topics like how to identify the intent of a query, how well a search result meets the needs of the query, the ratings for low quality and high quality sites, as well as judging the EAT of a website. It will show many examples which is something to really pay close attention to since you can actually see real sites showing what is expected when it comes to ratings.

The qualification exam consists of three tests:

  1. Theoretical – Multiple choice questions from the General Guidelines.
  2. Practical Part 1 – Examples of Quality ratings exam.
  3. Practical Part 2 – Examples of Needs Met rating exam.

Test 1 is by far the easiest and you will get the results right away letting you know if you passed or failed. You must pass each test to move on to the next one. Test 2 and 3 are very difficult as you will be doing sample ratings just as you would if you were really working. After finishing each Practical test, you will need to wait a few days for the results which will be emailed to you.

Most people are given second chances to retake any of the failed tests so long as their scores aren’t too low. It is not guaranteed though as some have also stated they were not given a second chance regardless of their score.

I consider myself pretty internet savvy and spent a lot of time studying the guidelines and still failed both Test 2 and 3 once before finally passing. You will have 10 days from when the acceptance to the project email is received to take the exam so be sure to take advantage of the few extra days to really understand the guidelines. Make sure when studying to really focus on Needs Met and Page Quality ratings.

A bit of advice, don’t go with what your instinct tells you, follow the rating system exactly how it’s stated in the guidelines. There can be discrepancies on what you think is a high quality site or how well a result meets needs than what the guidelines state. Always go with the guidelines.

Accuracy / Quality Reviews

Appen’s Search Engine Rater projects conduct monthly quality reviews of your work. They typically select 10 random completed tasks from the previous month that all raters are graded on.

There are six scores for this metric: Perfect, Excellent, Exceeds Expectations, Meets Expectations, Substandard and Unacceptable. A score of Substandard or below will limit the amount of hours you’re able to work to between 2-4 hours a day. Continuous results below Substandard can get you removed from the project. If you are able to score above Substandard the following month, your daily hour cap will be moved back to Unrestricted Status.

Quality review results are sent to your email and can be accessed on your project’s dashboard. Your status will also be shown stating how many hours per day you’re currently allowed to work.

There is a learning curve you’ll have to adjust to as the work can seem difficult at first when everything is brand new. After a few months it really isn’t difficult to keep your scores at Meets Expectations and above so long as you have a good understanding of the guidelines. It is very helpful to go back and look over any errors you’ve made on the quality reviews since there are comments left stating what is the correct rating and the reasoning. There is also a chatroom where you can talk with other raters who are generally very friendly and helpful with answering questions.

How And When Do I Get Paid?

Most projects pay bi-monthly with payments sent the following Friday after the two week period concludes. Some projects are on a monthly pay schedule with payments sent a week after the month end. Payments from Appen will be sent through Payoneer, Paypal, or bank transfer (US).

All though not too certain on workers outside of the US, for US workers Appen will pay the minimum wage for your local jurisdiction usually ranging from $11-$16 per hour pending on your location. It does seem to be a common consensus that workers outside of the US receive considerably lower pay.

Invoice

Appen Search Engine Evaluator

Most of Appen’s search engine evaluator jobs require you to keep track of your own hours and send an invoice. You will be required to install an extension on your browser that will record your work sessions and task activities as well as keep track of how long you’ve worked and how many tasks you completed. You’ll enter your total hours and task information in the Timecards section on your dashboard.
It’s best to use a time tracker like Clockify to help keep record.

After the two week pay period is over you must then submit your timecard by 12pm the following Monday. If you forget to submit your timecard on time it will defer to your next payment. In the 3-4 years I’ve worked for Appen, I never had any issues with invoicing. I’ve always received the correct amount I entered on my timecard. Sometimes more.

During busy seasons when there are a backlog of tasks, Appen will often give out bonuses for surpassing a set amount of hours. It’s often an extra $100-$200 a week for those who work 24+ hours a week. They also remove any quality restrictions in place so everyone can participate and have a chance at meeting the required hours. It’s a pretty generous deal from Appen.

PROS:

Employee paid per hour.
Pays at least minimum wage.
Can make $1000+ per month.
Long Contracts (have seen up to 8 years).
Flexible schedule.
Abundant amount of work.
Bi-monthly pay.

CONS:

Might take a while to hear back after applying.
Must pass difficult qualification exam.
Work can be repetitive and tedious.
Must meet minimum hours per week.
Reduced hours if receive bad quality review.

Conclusion

Appen’s Search Engine Evaluator position is a great remote job opportunity where you are an employee but still have the flexibility to work from home and on your own schedule. In my personal experience it has been one of the best part-time remote jobs and a great company to work for. Once you get passed the on-boarding and qualification stage you will definitely have enough work to keep you busy and and have a stable income.

Apply to be an Appen Search Engine Evaluator now.

Other Rater Jobs