Software Engineer - Cloud Data Pipelines (R/Python)

Permanent/Fixed term Full time Employment

We are seeking a Software Engineer to design, build, and maintain robust data pipelines that ingest, process, and execute custom R and Python scripts at scale. You will work within our cloud infrastructure (Google Cloud Platform or Azure ) to automate data workflows, ensure high performance, and integrate advanced analytical models into production environments.

Key Responsibilities

• Execute R/Python Workloads: Containerize, deploy, and orchestrate custom R and Python scripts to run reliably in cloud-native serverless or containerized environments.
• Cloud Integration: Leverage managed cloud services on GCP (e.g., Cloud Functions, Cloud Run, BigQuery, Dataflow) or Azure (e.g., Azure Data Factory, Azure Functions, Databricks, Synapse Analytics).
• Workflow Orchestration: Implement and maintain workflow tools (such as Apache Airflow or Prefect) to schedule and monitor script execution.
• Ensure Data Quality: Implement automated logging, error handling, and testing frameworks to validate data integrity and script outputs.
• Optimize Performance: Tune data processing jobs and cloud resource allocation for cost-efficiency, speed, and scalability.

Required Qualifications & Technical Skills

• Programming: Advanced proficiency in Python and/or R , with strong experience writing clean, modular, and well-tested code.
• Cloud Platforms: Hands-on production experience with either Google Cloud Platform (GCP) or Microsoft Azure .
• Containerization & Tools: Experience with Docker , Kubernetes , and CI/CD pipelines (GitHub Actions, GitLab CI).
• Data Stores: Proficiency with SQL and relational/non-relational databases (PostgreSQL, BigQuery, Snowflake, or Azure SQL).
• Software Engineering Practices: Solid foundation in version control (Git), automated testing, and API design.

Preferred Qualifications

• Experience migrating legacy local R/Python analytical scripts into distributed cloud architectures.
• Knowledge of Infrastructure as Code (Terraform or Bicep ).