Python QA Automation Engineer

In this role you will design and implement automated testing for MCP components within EPAM's AI platform, ensuring reliability and compliance across enterprise-scale workflows. You’ll collaborate with engineering and product teams in a Classic Agile setting to establish testing standards for AI models and tools in production-like environments. This position offers impact in shaping quality practices for AI-driven automation initiatives on a transformative project.

• EPAM Employee Stock Purchase Plan (ESPP)
• Life assurance and income protection
• Private medical insurance and dental care
• Employee Assistance Program
• Pension plan
• Commuter benefits and tech perks

• Develop automated testing frameworks for MCP Servers and related AI systems
• Design evaluation strategies for LLM accuracy, safety and reliability
• Create automated tests using Python, Pytest, and BDD frameworks
• Integrate quality gates into CI/CD pipelines for continuous assurance
• Identify agentic AI failure modes (hallucination, latency, tool usage)
• Collaborate with engineering, QA, and product teams to define quality metrics and acceptance criteria
• Contribute to Agile ceremonies to align testing with sprint goals
• Prepare reporting on quality outcomes and improvement opportunities
• Maintain documentation for test cases, evaluation pipelines, and validation strategies

• Strong Python programming for test automation and evaluation
• Expertise in Pytest; familiarity with Behave or Cucumber (BDD)
• Knowledge of LLM evaluation approaches (e.g., RAGAS, DeepEval) or custom pipelines
• Understanding of agentic AI issues (hallucination, tool misuse, performance bottlenecks)
• Experience with automated testing of AI workflows, distributed systems or microservices
• Strong grasp of Agile delivery and CI/CD quality checks
• Excellent communication and problem-solving skills focusing on accuracy and reliability

• communication
• problem-solving
• team collaboration
• Python
• Pytest
• BDD frameworks (Behave or Cucumber)