AI Agent Evaluation Engineer

Evaluate AI agents for Outlier through rubric writing, trace debugging, and stress testing. Work remotely and freelance on a schedule you choose.

Outlier helps innovative companies improve AI agents by supplying human feedback for large language model development. This opportunity suits software developers and orchestration specialists with experience in complex systems. Projects teach generative systems to coordinate real-world, multi-step workflows through detailed evaluation.

Key Responsibilities:

• Create rubrics that assess agent performance and behavior
• Inspect and debug agent traces across interactions
• Stress test autonomous agents for subtle failures
• Evaluate systems operating with live tools and data

Requirements:

• Two or more years of relevant technical experience
• Maintain modular, production-grade software services
• Build for live environments rather than mocked systems
• Give high-density technical feedback on complex behaviors
• Identify privacy leaks and indirect prompt injections

Use evaluation, debugging, and security-focused analysis to improve Outlier’s next generation of autonomous AI agents.