Senior Backend Engineer - Web-Scale Crawling & Ingestion

Nebius is seeking a Senior Software Engineer to build the content acquisition and crawling infrastructure for a novel agent-native search platform. You will design and operate distributed crawling systems, ingestion workflows, and scheduling to process billions of URLs at internet scale, balancing coverage, freshness, efficiency and cost.

You will collaborate with indexing and ML teams to ensure acquired content meets retrieval and ranking requirements, while enabling safe experimentation with