Automation Tester

Description

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
  • Accuracy and relevance of responses
  • Consistency and reliability
  • Hallucinations and edge cases
• Work with LLM-as-judge patterns and automated evaluation techniques
• Define quality standards for non-deterministic AI systems
• Collaborate closely with engineering, product, and data teams

QE – Automation (API & Event-Based)

Key Responsibilities

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
  • Accuracy and relevance of responses
  • Consistency and reliability
  • Hallucinations and edge cases
• Work with LLM-as-judge patterns and automated evaluation techniques
• Define quality standards for non-deterministic AI systems
• Collaborate closely with engineering, product, and data teams

Required Skills & Experience

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
  • Accuracy and relevance of responses
  • Consistency and reliability
  • Hallucinations and edge cases
• Work with LLM-as-judge patterns and automated evaluation techniques
• Define quality standards for non-deterministic AI systems
• Collaborate closely with engineering, product, and data teams

Additional Details

Industry Background: Open (no specific industry Experience required)

3x days a week in Burgess hill office.

Skills

• java
• kotlin
• kafka
• Testing
• Automation
• Quality assurance

Job Title:

Automation Tester

Location:

Burgess Hill, UK

Job Type:

Contract