Python PySpark Data Engineer

Python and PySpark expertise define this Ottawa-based contract for a Data Engineer supporting an existing customer remotely. Convert code and optimize nightly ETL pipelines.

LeverageTek seeks a Data Engineer for a five-month contract with an Ottawa-based customer. The work requires intermediate to senior hands-on experience with Apache Spark, PySpark, and Python. You will also use GitLab or GitHub while improving existing ETL architecture, pipeline accuracy, and efficiency.

Key Responsibilities:

• Convert Python packages to PySpark
• Test, validate, and integrate packages into nightly ETL
• Identify and implement performance optimizations
• Write and validate code improvements
• Document ETL processes when time permits

Requirements:

• Intermediate to senior Apache Spark experience
• Intermediate to senior Python experience
• Experience converting and optimizing code
• GitLab or GitHub experience
• Ability to work independently remotely

Apply Python, PySpark, testing, and optimization expertise to strengthen nightly ETL pipeline performance for LeverageTek's customer.