Resaro provides independent AI assurance and testing to ensure AI systems are responsible, safe, and robust. It achieves this through technical evaluation, stress-testing, and advisory services for mission-critical AI workloads. The company uses AI testing tools, data science expertise, and proprietary testing protocols to assess performance, safety, and security. Headquartered in Singapore and Europe, Resaro serves enterprises deploying mission-critical AI and helps clients meet regulatory requirements.
AI AssuranceAI GovernanceResponsible AIRobust AISafe AIStress-TestingTechnical EvaluationThird-Party Testingresaro.ai
Resaro
Resaro provides independent AI assurance and testing to ensure AI systems are responsible, safe, and robust. It achieves this through technical evaluation, stress-testing, and advisory services for mission-critical AI workloads. The company uses AI testing tools, data science expertise, and proprietary testing protocols to assess performance, safety, and security. Headquartered in Singapore and Europe, Resaro serves enterprises deploying mission-critical AI and helps clients meet regulatory requirements.
AI AssuranceAI GovernanceResponsible AIRobust AISafe AIStress-TestingTechnical EvaluationThird-Party Testingresaro.ai
HQSG
Team Size28
Open Jobs19
Total Funding-
Latest FundraiseUnknown
Join the Team
Senior LLM Scientist (m/f/d)
HybridGreater Munich Metropolitan Area, DE
Hybrid • Greater Munich Metropolitan Area, DE
Resaro was founded on the belief that AI will change the world in ways we cannot even imagine, but every new technology needs safeguards to advance.
We are looking for a senior LLM scientist, based in Munich, with experience in deep learning based language models to work with a category-defining AI assurance venture that will help companies test and audit their AI systems. You will help evaluate and stress-test AI models to make sure they are fit for purpose and safe to be deployed. We value strong technical ability and real world experience and there will be room to solve challenging problems and adopt cutting edge technology into business applications.
YOU WILL
Teeming tracks opportunities at over 24,000 AI startups, then works with you to find (and land) the one you'll love.
Product Designer
Part-timeTel Aviv
Part-time • Tel Aviv
DevOps Engineer
Full-timeCambridge, GB
Full-time • Cambridge, GB
Backend Developer
Part-timeJerusalem
Part-time • Jerusalem
DevOps Engineer
InternshipNew York, US
Internship • New York, US
DevOps Engineer
ContractLondon, GB
Contract • London, GB
Product Designer
Part-timeRotterdam, NL
Part-time • Rotterdam, NL
Lead the design, implementation, and execution of robust frameworks to evaluate the performance of generative AI systems, including text and multi-modal models. This will be for foundation models, fine tuned models as well as end-to-end systems
Establish and refine metrics and benchmarks for model quality, including output fidelity, diversity, creativity, and bias detection.
Perform technical AI evaluations, benchmarking and “red-team” tests on large language models, including assessing them for robustness in performance, embedded biases, vulnerability to jailbreaks and prompt injection attacks.
Work with clients and more junior team members to design custom evaluation approaches using the latest scientific research that address the client’s needs
Work with the product management team to develop a suite of technical and analytical AI evaluation frameworks and tools that are backed by scientific research and methods. These should assess the robustness, explainability, fairness, privacy, safety and security of AI and machine learning systems, with a strong focus on large language models.
Lead the design and implementation of evaluation frameworks for
Large Language Models (LLMs)
, including but not limited to GPT-based models, BERT, T5, and other state-of-the-art architectures.
Define and refine metrics for evaluating model performance, such as perplexity, BLEU, ROUGE, accuracy, coherence, factual consistency, and bias detection.
Lead efforts in curating and managing large, high-quality datasets for evaluating LLMs, ensuring data is representative, unbiased, and ethically sourced.
Mentor junior data scientists, guiding them in best practices for LLM evaluation and the latest advancements in NLP.
Stay up-to-date with the latest advancements in
Natural Language Processing (NLP)
and
LLM evaluation
, applying cutting-edge methods and tools to improve model performance.
YOU ARE ABLE TO
Think from first principles and want to tackle the most challenging technical problems from a multi-disciplinary approach e.g., design, engineering and social science.
Lead by example regardless of whether you are a manager or individual contributor. You want to work with passionate and talented individuals and people want to work with you.
Communicate in an open, frank and respectful manner.
Thrive in a fast-paced environment.
Navigate uncertainty by being willing to explore, while remaining laser focused on the mission at hand.
YOU HAVE
Extensive experience as a data scientist training or deploying deep learning based natural language models/large language models in real-world contexts. About 5-8 years of working experience or a relevant postgraduate degree with 2+ years of working experience building and deploying LLMs.
Strong experience in evaluating LLMs using metrics such as perplexity, BLEU, ROUGE, and human-centered evaluation techniques.
Proven track record of managing and analyzing large, complex language datasets, including text preprocessing and tokenization.
Excellent written and verbal communication skills, with the ability to clearly explain complex technical concepts to diverse audiences, including non-technical stakeholders.
Solid programming skills in Python and experience building automated pipelines for continuous model evaluation.
Passion and interest in applied research on the safe and responsible use of AI and with large language models.
NICE TO HAVE
Published research in the field of generative AI or model evaluation.
Hands-on experience with model explainability tools and methods.
Familiarity with cloud-based platforms (e.g., AWS, GCP) for scalable model evaluation and deployment.
ABOUT US
Resaro is a global AI Assurance company, pioneering the field of AI testing and evaluation. We are a team of AI experts, engineers, and data scientists. Our mission is to ensure an AI market worthy of trust.
Resaro is an Equal Opportunity Employer. We respect each individual and support the diverse cultures, perspectives, skills and experiences within our teams.
