Lead Data Engineer (Databricks)

As Lead Data Engineer, you will own scalable Databricks-based data platforms for near real-time river water quality monitoring. You’ll build and optimise batch and streaming pipelines, enforce data governance, and mentor engineers within a cross-functional team. The role sits in the River Water Quality Programme within Digital Services, delivering trusted data for regulators, scientists and operations. You’ll shape data architecture and drive high-impact, environmentally meaningful analytics.

• Up to 10.5% annual bonus
• Pension – up to 21%
• Private healthcare
• Share buy scheme
• Wellbeing benefits and support programmes

• Design, build and optimise Databricks-based data pipelines (batch and streaming)
• Implement medallion architecture (bronze/silver/gold) for data quality and lineage
• Optimize Spark jobs to meet near real-time SLAs
• Build and maintain streaming pipelines with Structured Streaming and Delta Live Tables
• Ensure data governance, schema enforcement, validation rules and ACID compliance
• Orchestrate and automate pipelines using Databricks Jobs, APIs or cloud-native schedulers
• Collaborate with data scientists, analysts and environmental specialists to deliver well-documented datasets
• Monitor, troubleshoot and continuously improve production pipelines for reliability, performance and cost
• Champion CI/CD, testing, version control and documentation practices
• Act as technical leader and mentor to raise standards across the data engineering community

• Deep expertise in Databricks (Spark / PySpark, Delta Lake, Unity Catalog, Workflows)
• Strong experience building ETL/ELT pipelines (batch and streaming)
• Advanced Python and SQL skills
• Hands-on experience with cloud platforms (Azure, AWS or GCP)
• Solid understanding of data modelling and modern architectures
• Strong knowledge of data governance, security and compliance
• 5+ years in data engineering; 3+ years with Databricks
• Proven ability to lead technically and mentor engineers

• Collaborative mindset
• Strong communication and leadership
• Problem-solving and initiative
• Databricks (Spark / PySpark, Delta Lake, Unity Catalog, Workflows)
• ETL/ELT pipelines (batch and streaming)
• Structured Streaming and Delta Live Tables