AI Agent Evaluation Engineer

Evaluate AI agents remotely using debugging, stress testing, and technical analysis for Outlier’s large language model projects.

Outlier collaborates with AI organizations to improve autonomous systems through human feedback. This freelance work focuses on teaching agents to design, coordinate, and optimize complex architectural workflows. Candidates need software experience with live environments and multi-turn interactions.

Key Responsibilities:

• Develop evaluation rubrics for agent performance
• Inspect and debug detailed agent traces
• Stress test autonomous agents across workflows
• Assess complex system behaviors with precise feedback

Requirements:

• Two or more years of relevant engineering experience
• Production-grade software development with modular separation
• Experience with live, non-mocked environments
• Comfort handling multi-turn system interactions
• Strong attention to technical detail and failure modes

Use AI evaluation, trace debugging, and systems expertise to improve autonomous agent quality through Outlier projects.