Senior AI Observability Engineer

Build AI observability systems at BMO in a hybrid role, using Python, telemetry, and evaluation methods for production models and agents.

As a Senior AI Engineer, you will design systems that trace LLM calls, monitor token usage, detect drift, and evaluate autonomous workflows. The work includes machine learning, deep learning, statistical analysis, and specialized technical support. You will help govern agent behavior and diagnose non-routine production problems.

Key Responsibilities:

• Design and maintain AI model observability systems
• Trace agent trajectories, prompts, tools, and outputs
• Build automated evaluations for relevance and hallucinations
• Monitor drift, latency, errors, and data quality
• Define guardrails and self-remediation boundaries

Requirements:

• Bachelor's degree in Computer Science or Engineering
• Seven to ten years of Site Reliability Engineering experience
• Experience in Artificial Intelligence Observability Engineering
• Python, SQL, LogQL, or PromQL proficiency
• Experience with Prometheus, Grafana, or Datadog

Apply AI engineering, observability, and evaluation expertise to support reliable agent workflows at BMO.