Staff Data Engineer Pipelines

Design batch and streaming pipelines at Index Exchange using Spark, Kafka, and Flink for billions of daily events. Build reliable data systems at scale.

As a Staff Data Engineer, you will evolve the data processing and pipeline layer supporting reporting, billing, and real-time data streams. You will transform high-volume event data into clean, queryable, production-grade datasets. The work spans distributed processing, SQL, data modeling, workflow orchestration, and performance optimization.

Key Responsibilities:

• Design and operate Spark, Kafka, and Flink pipelines
• Build canonical datasets and reporting tables
• Maintain Airflow DAGs for scheduling and backfills
• Optimize runtime, throughput, compute efficiency, and scalability
• Define data processing standards across teams

Requirements:

• Strong experience in data engineering at scale
• Expertise in distributed processing frameworks, SQL, and data modeling
• Experience with Airflow, Kafka, or streaming systems
• Experience managing Kubernetes
• Ability to lead projects and debug complex data issues

Bring staff-level data engineering leadership to Index Exchange through scalable pipelines, streaming systems, and optimized data platforms.