Autonomous Agent Evaluation Engineer

Evaluate autonomous agents through rubric writing, trace debugging, and stress testing. Work remotely with Outlier as a freelance contributor on flexible schedules.

Outlier helps innovative companies improve AI agents by providing human feedback for advanced generative systems. This work focuses on teaching large language models to coordinate multi-step, real-world architectural workflows. You will assess complex system behavior and deliver high-density technical feedback. The opportunity requires two or more years in backend engineering, AI automation, or complex systems integration. Experience with modular production software, live environments, and multi-turn interactions is required. Applicants may apply from Australia, Canada, Finland, France, Italy, Norway, the United Kingdom, or the United States.

Key Responsibilities:

• Develop evaluation rubrics for agent performance
• Debug traces from multi-step agent workflows
• Stress test autonomous agent behavior
• Identify privacy leaks and authority escalation
• Review indirect prompt injection failures

Requirements:

• At least two years of relevant experience
• Strong attention to complex system details
• Clear, high-density technical feedback skills
• Production-grade modular software experience
• Live, multi-turn systems experience

Use evaluation, debugging, and systems-integration expertise to strengthen autonomous-agent performance for Outlier’s AI projects.