Data Engineer London Liverpool Street, EC2M 4TP

• Build and enhance data pipelines and automation workflows on Azure for large-scale data ingestion and processing
• Maintain and optimise existing toolkits, databases, dashboards, and API-based solutions
• Develop end-to-end automated data solutions across commodities (Gas/LNG, Power, Weather)
• Design and implement web scraping pipelines (e.g. US LNG data)

Analytics & Real-Time Systems

• Support real-time and batch analytics use cases in a dynamic trading environment
• Enable and support analytics dashboards and GUI tools with reliable backend infrastructure
• Develop real-time flow tracking solutions at critical market points
• Design and manage Snowflake data models, ensuring performance and scalability
• Integrate external data sources including ENTSOE, EPEX, KPLER, MetDesk, WoodMac, and others
• Collaborate with analytics teams to structure data for models and advanced analytics use cases
• Build and maintain data platforms supporting ML models such as demand forecasting
• LNG Sendout Optimisation Model
• Ship Tracking & Flow Monitoring Tools
• Prompt Price & Forward Curve Bootstrapping
• Web scraping pipelines for US LNG and other market data sources
• Real-time flow tracking infrastructure at critical market points
• Data platforms supporting ML and demand forecasting models

Required Skills & Experience

Technical

• Strong proficiency in Python for data engineering and pipeline development
• Hands-on experience with Azure data services (Data Factory, Databricks, Blob Storage, etc.)
• Experience designing and optimising Snowflake data models
• Proven track record building and maintaining data pipelines at scale
• Experience integrating third-party market data APIs and external data feeds
• Familiarity with SQL and data warehousing concepts

Domain

• Experience or strong interest in energy, commodities, or financial markets data
• Understanding of real-time and batch data processing patterns in a trading context
• Familiarity with market data providers such as ENTSOE, EPEX, KPLER, or similar

Nice to Have

• Experience with web scraping frameworks (e.g. Scrapy, BeautifulSoup, Playwright)
• Knowledge of LNG, Gas, or Power market fundamentals
• Experience supporting ML model pipelines or demand forecasting workflows
• Exposure to GUI/dashboard tools such as Grafana, Power BI, or Streamlit
• Knowledge of ship tracking data or AIS feeds