Staff Engineer, Data Platform & Infrastructure

At Index Exchange, we’re reinventing how digital advertising works—at scale. As a global advertising supply-side platform, we empower the world’s leading media owners and marketers to thrive in a programmatic, privacy-first ecosystem.

We’re a proud industry pioneer with over 20 years of experience accelerating the ad technology evolution. Our proprietary tech is trusted by some of the world’s largest brands and media owners and plays a crucial role in keeping the internet open, accessible, and largely free.

We process more than 550 billion real-time auctions every day (in comparison, Google processes 8.5 billion searches per day) with ultra-low latency. Our platform is vertically integrated from servers to networks and runs primarily on our own metal and cloud infrastructure. This end-to-end infrastructure is designed to provide both stability and agility, enabling us to adapt quickly as the market evolves.

At the core of it all is our engineering-first culture. Our engineers tackle internet-scale problems across tight-knit, global teams. From moving petabytes of data and optimizing with AI to making real-time infrastructure decisions, Indexers have the agency and influence to shape the future of advertising. We move fast, build thoughtfully, and stay grounded in our core values.

About the role

We are hiring a Staff Data Engineer to build and evolve the data processing and pipeline layer that powers reporting, billing systems, and real-time data streams at Index Exchange.

This role focuses on designing and operating large-scale batch and streaming data pipelines , enabling reliable, scalable, and efficient data transformation across the platform.

You will work on systems that transform raw, high-volume event data into clean, queryable, and production-grade datasets.

You will work on high-scale data systems that:

• Process billions of events per day across distributed pipelines
• Power core data platform supporting consumers as (reporting, billing, MLO teams)
• Operate across batch (Spark) and streaming (Kafka / Flink) architectures
• Require careful balancing of:
  • data correctness
  • processing efficiency
  • latency vs cost trade-offs

You will solve problems such as:

• Designing pipelines that scale without exploding compute costs
• Managing data correctness at scale (deduplication, late data, joins)
• Building systems that support both:
  • historical backfills
  • near real-time updates
• Evolving pipelines from centralized processing (Hadoop) toward more distributed and efficient patterns
• Streaming pipelines and Streaming DWs.

What we’re lookingfor

• Strong experience in data engineering at scale
• Deep expertise in:
  • Distributed Data Processing frameworks
  • SQL and data modeling
  • Leading Projects end to end
• Experience with:
  • Airflow or workflow orchestration
  • Kafka or streaming systems
  • Managing kubernetes
• Strong understanding of:
  • distributed data processing
  • data modeling (large-scale datasets)
  • performance optimization
• Ability to:
  • own pipelines end-to-end
  • debug complex data issues
  • work in high-scale, evolving environments

Staff-Level Expectations

• Define data processing standards and patterns across teams
• Lead large-scale pipeline and platform initiatives
• Influence data architecture and modeling decisions
• Drive improvements across:
  • reliability
  • cost efficiency
  • scalability

Here’s what you’ll be doing

Data Pipelines (Batch + Streaming)

• Design and operate pipelines using:
  • Spark, Flink etc.
  • Kafka / Flink (streaming)
• Transform raw event data into:
  • cleaned datasets (silver layer)
  • business-ready datasets (gold / reporting tables)

Core Data Models & Datasets

• Build and maintain canonical datasets (aggregated datasets, reporting tables)
• Define data contracts and ensure consistency across pipelines
• Support evolving use cases:
  • reporting
  • billing
  • ML / experimentation

Workflow Orchestration

• Build and maintain Airflow DAGs for:
  • pipeline scheduling
  • dependency management
  • backfills
• Improve reliability and observability of workflows

Data Processing Optimization

• Optimize pipelines for:
  • performance (runtime, throughput)
  • cost (compute efficiency)
  • scalability (data growth)
• Improve:
  • partitioning strategies
  • data layout
  • job execution patterns

Streaming & Near Real-Time Pipelines

• Build pipelines that support:
  • incremental updates
  • streaming transformations
  • aggregation at scale
• Contribute to evolving patterns such as:
  • edge aggregation
  • streaming batch convergence
  • real-time data availability

Platform & System Design Responsibilities

• Define and evolve data processing patterns :