Gmail & Google Calendar AI Assistant Evaluator

OpenTrain AI is looking for AI Assistant Evaluators to test how well artificial intelligence handles everyday tasks involving Gmail, Google Calendar, online bookings and real-world coordination.

Rather than building AI models, you'll interact with AI assistants as a user would: giving them instructions, checking whether they completed tasks correctly and providing structured feedback that can be used to improve their performance.

This is an entry-level opportunity, although candidates should already be highly comfortable using email, calendars and online booking services independently.

What you'll be working on

You'll evaluate AI assistants across realistic scenarios involving communication, scheduling and coordination.

Tasks may include:

• Testing AI assistants on Gmail and Google Calendar workflows.
• Working with email conversations and calendar invitations.
• Creating, changing and coordinating schedules.
• Testing online booking scenarios.
• Evaluating tasks involving multiple people or competing scheduling requirements.
• Checking whether an AI assistant completed each instruction accurately and completely.
• Rating, ranking and annotating AI responses according to evaluation guidelines.
• Testing edge cases and alternative scenarios.
• Recording results using designated AI training and evaluation tools.
• Discussing ambiguous cases with other contributors.
• Participating in quality reviews to maintain consistent evaluation standards.

What OpenTrain AI is looking for

You don't need previous professional AI evaluation experience, but you should have strong practical experience using the digital tools being tested.

You'll need to:

• Use Gmail and Google Calendar regularly.
• Be comfortable sending and managing emails, invitations and schedule changes.
• Have experience making online bookings independently, such as restaurants, appointments, travel or deliveries.
• Be comfortable giving instructions to AI tools and critically checking their work.
• Communicate clearly in written English.
• Have excellent attention to detail.
• Be organized and capable of managing contractor work independently.
• Be located in the United States.
• Have an iPhone with iMessage.
• Have an active Facebook or Instagram account.

Helpful experience

Previous work in these areas can strengthen your application, although it isn't required:

• AI evaluation.
• Data annotation or labeling.
• Prompt writing.
• User testing.
• AI feedback and response grading.
• Quality assurance.

The most important skill is the ability to judge carefully whether an AI assistant has actually accomplished what was requested rather than simply producing a plausible-looking response.

Compensation & schedule

The listed compensation is:

$15–$30 per hour

This is a remote contractor position requiring 20 or more hours per week.

The opportunity is currently limited to applicants located in the United States.

Why this role is interesting

As AI assistants become capable of interacting with tools such as email, calendars and booking systems, evaluating them requires more than checking the quality of generated text.

Evaluators need to determine whether an assistant successfully completes multi-step actions in real-world digital environments — including coordinating people, changing plans, managing invitations and responding correctly when something unexpected happens.

That makes this an accessible way to gain hands-on exposure to AI evaluation and agentic AI workflows without requiring a software engineering or machine-learning background.

How to apply

Submit your resume through OpenTrain AI and then complete the application process on the hiring platform.

About AI evaluation work

Human evaluation plays an important role in improving AI assistants. Evaluators test models against realistic tasks, identify mistakes and provide structured feedback that helps make future systems more accurate, reliable and useful.