AI Agent Evaluation Engineer

Evaluate autonomous AI agents through rubric writing, trace debugging, and stress testing while working remotely as a freelance contributor for Outlier.

Outlier helps innovative companies improve AI agents by collecting human feedback. This work focuses on training large language models to function as proactive, multi-step systems. You will assess how agents design, coordinate, and optimize complex workflows. Applicants need at least two years of relevant experience.

Key Responsibilities:

• Debug agent traces across multi-turn interactions
• Write detailed evaluation rubrics
• Stress test autonomous agent behavior
• Assess complex system behaviors with precision
• Support training for generative AI systems

Requirements:

• Experience with backend engineering, AI automation, or systems integration
• Ability to maintain production-grade modular software
• Practical experience with live environments
• Clear, high-density technical feedback skills
• Experience handling persistent state or session discovery preferred

Use technical feedback, agent testing, and systems integration experience to strengthen autonomous AI evaluation projects with Outlier.