LLM Agent Evaluation Engineer

Evaluate LLM agents with Outlier using technical feedback, agent tracing, and stress testing. Work remotely as a freelance contributor on flexible projects.

Outlier helps innovative companies improve AI agents by providing human feedback. As an LLM Agent Evaluation Engineer, you will support systems designed for proactive, multi-step workflows. The work suits software developers and orchestration specialists with production experience, attention to detail, and complex integration skills.

Key Responsibilities:

• Debug agent traces and investigate system behavior
• Write rubrics for evaluating agent performance
• Stress test autonomous agents against complex workflows

Requirements:

• Two or more years of relevant professional experience
• Experience building production-grade modular software
• Practical multi-turn system interaction experience
• Ability to deliver high-density technical feedback
• Experience identifying privacy leaks and prompt injections

Use LLM evaluation, debugging, and systems expertise to improve autonomous agent performance with Outlier.