Senior Data Engineer - Python, Databricks & Counterparty Credit Risk

Senior Data Engineer - Python, Databricks & Counterparty Credit Risk

Location: London - Hybrid

We are recruiting for an experienced Senior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supporting Counterparty Credit Risk (CCR), Credit Risk and wider risk data .

This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks , alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.

Candidates should ideally have previously worked with Counterparty Credit Risk or Credit Risk data , including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks
• Build and optimise ETL/ELT workflows using Spark and Delta Lake
• Engineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functions
• Work with datasets relating to areas such as counterparties, credit exposures, limits, collateral and risk analytics
• Design robust data models and architectures across Lakehouse / Medallion environments
• Develop and manage Databricks notebooks, jobs and workflows
• Optimise large-scale distributed data processing and troubleshoot Spark performance
• Implement strong data quality, reconciliation, governance and monitoring controls
• Build reliable logging and alerting across production data pipelines
• Integrate data platforms with Power BI dashboards and wider applications
• Support CI/CD and engineering best practices across development, testing and production environments
• Work closely with Risk, Technology, Analytics and other senior stakeholders

Required Experience

• 10+ years' experience across Data Engineering or closely related data disciplines
• Strong hands-on Python development experience
• Strong PySpark / Apache Spark engineering experience
• Extensive hands-on experience with Databricks , including notebooks, jobs/workflows and Delta Lake
• Advanced SQL and relational database experience
• Strong track record designing and building scalable ETL/ELT data pipelines
• Experience with Delta Lake, Lakehouse and/or Medallion architectures
• Experience handling banking risk data , ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domains
• Strong understanding of data modelling, data quality and production data engineering
• Cloud experience across Azure, AWS and/or GCP
• Experience with modern data platforms/warehouses such as Snowflake, BigQuery or Redshift
• Experience integrating data platforms with Power BI
• Experience with orchestration such as Databricks Workflows or Airflow

Highly Desirable

• Direct experience engineering data for Counterparty Credit Risk , including exposure/counterparty datasets
• Knowledge of areas such as PFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limits
• Previous experience within an investment bank, global bank or capital-markets environment
• Databricks Data Engineer certification
• Kafka / Structured Streaming experience
• Delta Lake CDC, Time Travel and optimisation
• CI/CD, Git and DevOps experience
• Docker / Kubernetes experience
• Experience exposing data through APIs or supporting React-based front-end applications
• Experience supporting analytics or ML workloads within Databricks

We are particularly interested in genuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.