Data Engineer
Scottish and Southern Energy
In this role you will design and operate data ingestion, transformation and serving layers on Azure Databricks to support reliable data products and insights. You will help build the Renewables Data Platform and migrate solutions from Synapse, working with data architects, analysts and product teams to deliver scalable ELT/ETL pipelines with a focus on quality, security and performance. This hands-on position emphasizes collaboration, modern data modeling and governance to enable data-driven decisions in a renewables context. You will work in a cross-functional Data & AI team to impact business outcomes.
Pay / Benefits- private healthcare discounts
- gym membership discounts
- free online GP
- 24/7 counselling
- interest-free loans for tech and transport
- Cycle to Work scheme
- Build and maintain metadata-driven pipelines in Azure Databricks for batch and streaming data
- Develop Delta Lake layers (bronze/silver/gold) for BI/analytics and Power BI integration via Databricks SQL
- Perform large-scale transformations with Databricks pipelines, PySpark and SQL, ensuring data quality and performance
- Implement data quality, monitoring and governance practices into pipelines and data products
- Contribute to Governance, Security & Compliance using Unity Catalog lineage and code-managed permissions
- Support CI/CD and version-controlled solutions in collaboration with DevOps and product teams
- Migrate and modernize data solutions from Synapse as part of renewing the platform
- Collaborate with stakeholders to refine requirements and ensure solutions meet business needs
- Champion data protection controls and contribute to SLAs, SLOs and data contracts
- Continuously develop skills with evolving tools and technologies
- Proven data engineering experience with batch and streaming pipelines
- Experience with Azure Databricks, PySpark and SQL
- Delta Lake architecture knowledge (bronze/silver/gold)
- Power BI integration and Databricks SQL experience
- Strong communication with technical and non-technical stakeholders
- Ability to manage workload in a growing business
- Awareness of governance, security and data protection practices
- CI/CD and version-controlled development experience
- communication
- adaptability
- proactive mindset
- Azure Databricks
- PySpark
- SQL
Reference: WJ-747_31159574