What the job involves

• As a Data Engineer, you'll architect and build the data infrastructure that powers everything we do—from crawling billions of pages to training our embedding models to serving real-time search
• You'll have enormous autonomy in designing systems that scale to hundreds of petabytes
• Design a lakehouse architecture that handles 100+ PB of web crawl data
• Build streaming pipelines that process billions of documents per day for real-time indexing
• Architect the data layer for our embedding training infrastructure on Ray
• Scale our ClickHouse deployment to handle analytical queries across petabytes of search logs