Autonomous Agent Evaluation Engineer

Evaluate autonomous agents remotely with technical feedback, agent trace debugging, and stress testing for Outlier’s AI projects. Flexible scheduling is available.

Outlier helps innovative companies improve AI agents by providing human feedback on complex behaviors. As an autonomous agent evaluation engineer, you will assess proactive, multi-step systems. Your work supports large language models learning to design, coordinate, and optimize architectural workflows.

Key Responsibilities:

• Develop rubrics for assessing agent outputs
• Investigate failures within agent traces
• Test agents against demanding real-world workflows

Requirements:

• Two or more years of relevant experience
• Experience building production-grade modular software
• Background in backend engineering or AI automation
• Experience with live multi-turn system interactions
• Detailed feedback on complex system behaviors

Apply evaluation, debugging, and systems expertise while helping Outlier improve proactive agents and advanced generative systems.