Qualifications:

• Ability to hold a position of public trust with the U.S. government.

• Bachelor’s degree and 8 years of experience

• 5+ years of experience evaluating machine learning, NLP, or generative AI systems, with strong familiarity with LLMs and retrieval-based architectures.

• Deep understanding of evaluation metrics, statistical testing, dataset construction, experimental design, and model validation methodologies.

• Hands-on experience with Python and libraries such as PyTorch, Hugging Face, LangChain, scikit-learn, and evaluation tooling (LLM-as-a-judge, rubric-based evaluators, or custom harnesses).

• Proficiency in AI evaluation frameworks such as Ragas

• Demonstrated experience designing automated evaluation pipelines and integrating them into CI/CD or LLMOps workflows.

• Strong understanding of AI governance, responsible AI principles, bias detection, fairness metrics, and risk identification.

• Experience working with structured and unstructured datasets across multiple modalities (text, tabular, documents).

• Familiarity with vector databases, RAG architectures, and multi-step LLM workflows.

• Familiarity with OWASP LLM Top 10 Risks

• Excellent analytical, written, and verbal communication skills, with the ability to translate evaluation insights into clear technical recommendations.

• Proven ability to collaborate with cross-functional engineering and product teams while independently driving evaluation strategy.

• Experience working in agile or iterative development environments and documenting scientific processes clearly.

About steampunk:

Steampunk relies on several factors to determine salary, including but not limited to geographic location, contractual requirements, education, knowledge, skills, competencies, and experience. The projected compensation range for this position is $135,000 to $170,000. The estimate displayed represents a typical annual salary range for this position. Annual salary is just one aspect of Steampunk’s total compensation package for employees. Learn more about additional Steampunk benefits here.

Identity Statement

As part of the application process, you are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud.

Steampunk is a Change Agent in the Federal contracting industry, bringing new thinking to clients in the Homeland, Federal Civilian, Health and DoD sectors. Through our Human-Centered delivery methodology, we are fundamentally changing the expectations our Federal clients have for true shared accountability in solving their toughest mission challenges. As an employee owned company, we focus on investing in our employees to enable them to do the greatest work of their careers – and rewarding them for outstanding contributions to our growth. If you want to learn more about our story, visit http://www.steampunk.com.