IT & Software

Lead Databricks Engineer

EPAM Systems

London · Greater London · United Kingdom

Overview

As Lead Databricks Engineer, you will design and steer enterprise-scale Azure Databricks data platforms to enable advanced analytics and real-time processing. You’ll lead architecture, governance and cost optimization while delivering high-quality solutions for a data-driven business. You’ll work with cross-functional teams to translate requirements into robust technical designs and drive platform performance and observability. This is a hands-on leadership role that shapes data engineering practices, production readiness, and strategic cloud initiatives.

Pay / Benefits
  • EPAP Employee Stock Purchase Plan
  • life assurance and income protection
  • private medical insurance
  • dental care
  • Employee Assistance Program
  • pension plan
Responsibilities
  • Lead architecture design and implementation of scalable Azure Databricks solutions
  • Enforce platform standards and data governance and security policies
  • Build and optimize streaming workloads (Structured Streaming, Delta Live Tables)
  • Collaborate with cross-functional teams to translate requirements into robust solutions
  • Drive performance tuning, cost optimization and monitoring for cloud workloads
  • Provide leadership across engineering lifecycle and production support
  • Mentor developers and establish best practices for Azure and Databricks
  • Participate in technical decisions and review designs for scalability and maintainability
  • Implement observability for critical pipelines and ensure QA standards
  • Engage with stakeholders to align on strategic platform initiatives
Key requirements
  • 8+ years in data engineering
  • 3+ years in tech lead capacity
  • Hands-on Databricks for enterprise-scale workloads
  • Proficiency in PySpark, Spark Structured Streaming, Delta Lake
  • Python programming and SQL performance optimization
  • Deep understanding of Azure Data Platform and cloud-native patterns
  • Experience implementing data governance, quality management and observability
  • Ability to manage complex batch and streaming data pipelines
  • Strong communication and leadership skills with distributed teams
  • strong communication
  • leadership
  • cross-functional collaboration
  • Databricks
  • Azure Databricks
  • PySpark

Reference: WJ-747_31193653

Apply now

More jobs

This role is listed on GigBlows for discovery and search. Hiring decisions and applications are handled by the employer or their chosen application system.