AI Agent Evaluation Engineer

Evaluate AI agents remotely through rubric writing, trace debugging, and stress testing. Outlier offers freelance work for experienced software engineers.

This AI Agent Evaluation Engineer opportunity supports Outlier projects training Large Language Models as proactive, multi-step agents. You will assess system behavior and help improve complex real-world architectural workflows. Candidates need at least two years in backend engineering, AI automation, or complex systems integration. Practical experience with live tools, persistent state, or agent safety analysis is valuable.

Key Responsibilities:

• Create evaluation rubrics for agent performance
• Debug traces from multi-step agent interactions
• Stress test autonomous agent behavior
• Assess failures across complex system workflows

Requirements:

• At least two years of relevant engineering experience
• Maintain modular, production-grade software systems
• Build for live rather than mocked environments
• Deliver clear, detailed technical feedback
• Recognize indirect prompt injection risks

Use agent evaluation, systems integration, and technical feedback skills to help Outlier improve advanced generative AI systems.