Generative AI Evaluation Engineer EA

Develop generative AI evaluation frameworks for EA SPORTS FC from Electronic Arts’ Burnaby, BC office, working hybrid at least three days weekly.

As a Software Engineer III, you will create tools for testing AI and machine learning systems. Your work will cover LLM-as-a-judge patterns, rubric-based scoring, calibration, datasets, and error analysis. You will collaborate with engineering, design, and quality partners to clarify requirements and resolve blockers.

Key Responsibilities:

• Develop evaluation frameworks for generative AI systems
• Implement LLM-as-a-judge patterns and rubric-based scoring
• Curate datasets and define validation criteria
• Perform quantitative and qualitative error analysis
• Collaborate with engineering, design, and quality partners

Requirements:

• Hands-on experience evaluating AI/ML and LLM-based systems
• Experience with generative system evaluation frameworks
• Knowledge of prompting, fine-tuning, or retrieval-augmented generation
• Experience using quantitative machine learning metrics
• 5+ years of professional software engineering experience

Bring generative AI, NLP, evaluation framework, and LLM expertise to quality engineering at Electronic Arts.