QA Automation Engineer - AI

In this role you will lead QA for GenAI features within Anaplan’s AI-powered platform, designing testing strategies and evaluation metrics for LLM-enabled workflows. You’ll build automated test coverage across APIs, UIs, and performance, ensuring accuracy, safety, and reliability of AI interactions. You’ll curate test datasets and golden examples to simulate diverse user scenarios, while supporting adversarial testing and production alerting. You’ll work closely with AI teams to validate code and maintain clear documentation of issues and coverage. This is a chance to shape quality for a leading AI-driven planning platform and contribute to a culture that values customer success and bold,创新—

• Develop and execute end-to-end automated test coverage for evaluations, APIs, UIs, and performance using Python
• Implement testing strategies for GenAI features (conversational AI, agentic systems, LLM-powered workflows)
• Run and maintain automated test suites for prompt testing and regression tests to detect model behavior changes
• Support and maintain evaluation frameworks measuring GenAI performance (accuracy, relevance, safety, consistency, latency)
• Curate high-quality test datasets and golden examples representing diverse scenarios
• Assist in adversarial testing to identify hallucinations or biases and integrate production alerting systems
• Utilize internal tools and testing frameworks to validate GenAI code and assist with UAT
• Document automated coverage and testing procedures, and log known issues

• Solid QA or test engineering experience with exposure to AI/ML systems
• Hands-on Python coding experience
• Understanding of GenAI technologies including LLMs and prompt engineering
• Experience with test automation frameworks and scripting (Python, Pytest, API, UI Testing)
• Knowledge of functional, integration, regression, performance, and security testing
• Experience with API testing tools (Postman, REST Assured) and backend testing
• Familiarity with CI/CD pipelines and automated testing integration

• Python
• Pytest
• API Testing
• UI Testing
• Postman
• REST Assured