Who you are

• Have hands-on ML experience (training, finetuning, or evaluating models (bonus if related to embeddings or LLMs)
• Have strong engineering fundamentals and can build reliable systems (Python, Rust, distributed pipelines, GPU/cluster jobs, etc.)
• Enjoy diving into data via building eval sets, inspecting edge cases, designing creative measurement strategies

What the job involves

• We're looking for an ML evals engineer to design and build our eval stack at Exa
• The role involves investigating how to evaluate search engines in an LLM world and then building the most comprehensive, creative, and effective eval suite
• You will be deciding the future of search through the evals we choose to optimize for
• Write a manifesto of what perfect search means
• Design and implement evaluation frameworks that probe the limits of search
• Build scalable, reliable eval pipelines that track regressions, drift, and quality signals across billions of documents
• Create golden datasets, synthetic benchmarks, agentic tasks, and real-world test suites that reflect how developers, agents, and humans actually use Exa
• Partner closely with ML researchers, data engineers, infra engineers, and product to shape the feedback loops that improve our search models