Senior AI Quality Engineer

About Addition

Addition is R/GAs applied intelligence studio. We work with many of the worlds leading brands to solve problems at the intersection of frontier AI and human-centered design. We collaborate with our clients to implement and scale AI systems that create real-world value—improving communications, customer experiences, and internal business workflows.
In our 40-year history, we’ve changed a lot—from a culture-defining film title studio (Superman, Alien, Ghostbusters); to an award-winning product and marketing agency with design, technology, communications, ventures and IP practices. This unwavering focus helps us create meaningful and memorable experiences that live across digital platforms and often blur the line between the digital and real worlds. As our Senior AI Quality Engineer, your job is to ensure that the solutions we ship are high quality by methodically testing them and working with engineering to implement improvements. you don’t assume things are working, you understand the shape of the system and test it from every angle to ensure quality. During planning phases, you explore system architecture, write test plans, and align with the team on what “quality” looks like. As systems are built, you red-team them, code scripts to automate and scale testing processes, implement logging, and methodically analyze the results. You think like an AI agent, stress-testing the behavior of AI systems under a wide range of conditions beyond the happy-path.
This is a new role at Addition, created because the complexity of the AI systems we ship has outgrown traditional QA practices and ad-hoc evaluation. We need someone who turns quality into a new discipline for the AI systems era — evolving fuzzy notions of “quality” into measurable signals, and bringing the rigor of a structured adversarial mindset. This role sits within our engineering team, working closely with engineers, PMs, cross-functional project teams, and clients.
Test Planning:

During project planning phases, translate product and technical requirements into structured test plans — covering functional behavior, edge cases, failure modes, and AI-specific risks like hallucination, prompt injection, and output drift.
Agentic Testing:

Working with coding agents like Claude Code and local or online versions of AI System builds to write test scripts, setup logging frameworks, analyze agent behavior, user behavior, to identify points of failure and opportunities for improvement. This is automated and streamlined for scale, but always done with intentionality, human supervision, and verification along the way.
AI Red Teaming:

Probe LLM behaviors, agent decision chains, brand voice outputs, and multi-model pipelines for failure modes that functional testing wont surface. Launch Gating:

Act as the quality gate on launches — raising blockers, documenting failure cases, and holding the line on ship criteria. brand voice agents, agentic creative tools, generative UIs) to systematize and improve efficiency, speed and cost of QA and red teaming practices over time.
Cross-Project Quality Standards:

Work across our project portfolio to develop shared quality standards for AI systems — what "good enough to ship" actually means for agentic products.
Document failure patterns, red team findings, and test results in a form that engineers and PMs can act on. Thrives in fast-paced, high-ambiguity environments and turns complex challenges into structured, elegant solutions. Authentic Communicator:

Exceptional written and verbal communication skills, able to translate complex technical/AI concepts into clear, actionable, and persuasive ideas. Genuinely passionate about frontier AI, continuous learning, and exploring applied innovation.
Experience:

4+ years of software QA, test engineering, or quality engineering experience, with at least 1–2 years working on AI-powered or LLM-based systems.
Deep familiarity with failure modes specific to generative AI systems (hallucination, prompt injection, context window degradation, inconsistent persona adherence, retrieval errors, agent loop failures, non-determinism, temperature, retrieval mechanics, context windows, grounding, and context adherence).
Experience writing structured test plans and evaluation criteria, including trajectory-level assessment of multi-step reasoning and judgment (understanding when to evaluate per span vs per trace vs per session).

Hands-on fluency with AI development tools, agent frameworks, and prompt engineering, with the ability to speak the same language as the engineers whose work is being tested.
Notion, Google Workspace, Figma/FigJam, Slack) and AI capabilities.
Bonus Points

Background working in an agency, studio, or consultancy where projects run fast and quality standards need to travel across diverse client work.
Experience red-teaming production AI agents (not just evaluating model outputs in isolation) and familiarity with agentic frameworks (Claude Agent SDK, LangGraph, AutoGen, or similar) and multi-step agent pipelines.
Experience building lightweight eval harnesses or automated regression pipelines for LLM outputs, including agent-as-evaluator setups, tracing/observability for agent runs, and debugging tool/MCP integrations.
Prior work in brand safety, content moderation, or AI safety adjacent to production systems.
Shape the Future:

Lead and innovate at an AI-powered studio at the forefront of applied AI innovation.
High-Impact Work:

Build and drive technological and creative solutions that redefine how global brands operate.
Direct Leadership Access:

Collaborate closely with our CEO, leadership team, and R/GA network to help define how we scale.
Candidates must be located within commuting distance in the Tri-State area (NY, NJ, CT), Southern California, or the Bay Area, or willing to relocate. This position offers a hybrid work model, with regular in-person collaboration and team / client workshops.