Senior Azure Data Engineer (Databricks)
Capco
Overview
In this role you design, build and deploy scalable data pipelines on Azure Databricks to support client data needs and business transformation. You’ll work with cross-functional teams in a world-class consulting environment to deliver secure, lakehouse-based solutions and promote best practices. You will help shape data engineering capabilities, from ingestion to serving, in a fast-paced, collaborative setting that serves Tier 1 financial institutions. This is an opportunity to impact transformation initiatives through hands-on engineering and continuous learning.
Pay / Benefits- Discretionary bonus
- competitive pension
- health insurance
- life insurance and critical illness cover
- Mental Health support
- flexible leave policies including maternity and shared parental leave
- Design and develop robust pipelines using DeltaLake, Spark Structured Streaming, and Unity Catalog
- Build real-time event-driven solutions with Kafka and Azure Event Hubs
- Apply DevOps principles to develop CI/CD pipelines using Azure DevOps, Jenkins, or GitHub Actions
- Collaborate with clients and stakeholders to translate data needs into strategic technical solutions
- Champion clean code, data lifecycle optimisation, and software engineering best practices
- Proven hands-on experience with Databricks platform and orchestration
- Strong skills in Python, PySpark, and SQL, with knowledge of distributed data systems
- Expertise in developing full lifecycle data pipelines across ingestion, transformation, and serving layers
- Experience with data lakehouse architecture, schema design, and GDPR-compliant solutions
- Working knowledge of DevOps tools and CI/CD processes
- Collaborative mindset
- Innovative and proactive problem-solver
- Client-facing communication
- Databricks platform and orchestration
- Python, PySpark, SQL
- DeltaLake, Spark Structured Streaming, Unity Catalog
Reference: WJ-747_30166379