IT & Software

Senior Manager - Principal Data Engineer

KPMG

London · Greater London · United Kingdom

Overview

In this role you will shape the technical direction for data engineering within Clara Technology & Solutions, leveraging Databricks and cloud platforms to deliver enterprise-scale solutions. You will lead a high-performing team, drive modern engineering practices including AI-assisted development, and collaborate with senior stakeholders to translate business needs into data platform strategies. You’ll enable self-service analytics and data-driven decision making through integrated BI tooling. This is a chance to influence data governance, scalability, and operational excellence at scale, within a collaborative, innovation-focused culture.

Responsibilities
  • Design and deliver enterprise-scale data solutions on cloud platforms (Azure preferred) using Databricks as the core platform
  • Build and optimize ETL/ELT pipelines with PySpark, SQL, Python; orchestrate workflows with Databricks Workflows, Delta Live Tables, Azure Data Factory
  • Implement Delta Lake storage and Unity Catalog governance for data quality, security, and compliance
  • Design scalable, performant, and operational data platforms from inception to production
  • Lead discovery and gather requirements, translating business needs into data platform roadmaps
  • Integrate data platforms with BI/analytics tools (Databricks AI/BI, Power BI) for self-service analytics
  • Establish and enforce data engineering standards, coding practices, and quality gates
  • Champion modern software engineering practices (Git workflows, automated testing, CI/CD for data workloads)
  • Drive observability, monitoring, and incident response for data pipelines
  • Promote AI-augmented data engineering with approved AI coding assistants
  • Provide technical direction and engineering accountability for data engineering initiatives
  • Coach and mentor engineers; lead Agile delivery (Scrum/Kanban) and stakeholder management
  • Foster cross-functional collaboration and represent data engineering in governance and architectural forums
Key requirements
  • Advanced hands-on Databricks experience (PySpark, Delta Lake, Unity Catalog)
  • Expert SQL for complex transformations; strong Python for data engineering
  • Deep PySpark expertise; cloud platform proficiency (Microsoft Azure preferred)
  • Experience with ETL/ELT development and orchestration (Databricks Workflows, DLT, ADF)
  • Git, GitHub collaboration, and CI/CD practices for data workloads
  • Agile software development (Scrum/Kanban) and DevOps practices for data engineering
  • Leadership experience balancing technical work with people management (minimum 3+ years Tech Lead)
  • Experience with AI-native software delivery and automated quality assurance in enterprise environments
  • leadership and mentoring
  • stakeholder management
  • collaboration across teams
  • Databricks platform (PySpark, Delta Lake, Unity Catalog)
  • SQL and Python for data engineering
  • Azure data services (Azure Data Factory, Delta Live Tables, Databricks on Azure)

Reference: WJ-747_30148611

Apply now

Continue on the employer's official application - the same link they use for every candidate.

More jobs

This role is listed on GigBlows for discovery and search. Hiring decisions and applications are handled by the employer or their chosen application system.