Autonomous Agent Evaluation Engineer

Evaluate autonomous agents with backend engineering expertise, debugging traces and stress testing systems through Outlier’s remote freelance platform.

Outlier collaborates with leading AI organizations to improve large language models through human feedback. This freelance work centers on teaching proactive, multi-step agents to coordinate complex architectural workflows. Candidates need at least two years in backend engineering, AI automation, or complex systems integration, plus production software experience.

Key Responsibilities:

• Develop rubrics for evaluating agent responses
• Investigate failures within agent traces
• Stress test autonomous agent workflows
• Assess behavior across live system interactions
• Deliver high-density technical feedback

Requirements:

• Two or more years of relevant engineering experience
• Production-grade software development and maintenance experience
• Modular separation across parsing, logic, and reporting
• Practical experience with non-mocked environments
• Ability to identify subtle system behavior failures

Use evaluation engineering, backend development, and detailed feedback to help improve autonomous agents and advanced generative systems.