ML Researcher – Benchmarks & Evaluation

Location: Hyderabad / Bangalore

Experience: 2+ years

About Deccan AI

Deccan AI is a high-growth, venture-backed AI model training and evaluation company headquartered in the Bay Area. Founded by alumni of IIT Bombay, IIM Ahmedabad, and ex-Google , we partner with the world’s top AI frontier labs including Google DeepMind, Snowflake , and several cutting-edge research groups. We are backed by Prosus Ventures , and our India office is based in Hyderabad.

We’re not just participating in the AI race we’re building the infrastructure that powers it.

With 1M+ global experts, advanced automation, and vertically integrated platforms, we deliver the gold-standard data that world-class AI models rely on. The AI data annotation market is exploding set to quadruple by 2032. The opportunity? Massive, and you can help define the future.

Role Overview

DeccanAI is seeking a Machine Learning Researcher – Benchmarks & Evaluation to conduct deep AI research and design innovative benchmarks and evaluation datasets . This role focuses on end-to-end research , translating cutting-edge academic insights into practical evaluation systems that advance AI capabilities and real-world applicability.

Key Responsibilities

1. Research & Literature Review

Track and analyze the latest AI research papers, conferences, and emerging trends.

Conduct deep literature reviews in areas such as:

• Agent frameworks
• Reasoning systems
• NLP, CV, and multimodal AI
• Identify gaps and opportunities for new benchmarks and evaluation datasets.

2. Benchmark & Evaluation Design

Propose novel AI benchmarks addressing real-world and research-driven challenges.

Design evaluation datasets for both coding and non-coding domains.

Define meaningful, scalable evaluation metrics aligned with industry needs.

Ensure benchmarks push the state-of-the-art while remaining practical.

3. Documentation & Deliverables

Create detailed benchmark and dataset proposal documents covering:

• Purpose and motivation
• Design overview
• Evaluation metrics
• Testing and validation guidelines
• Novelty compared to existing benchmarks
• Ensure documentation is clear and implementation-ready.

4. Collaboration

Work closely with the ML Lead, MLEs, and project managers .

Incorporate feedback from implementation teams and stakeholders.

Support refinement of benchmarks based on execution results.

5. Continuous Improvement

Iterate on existing benchmarks using research updates and real-world feedback.

Suggest new metrics or evaluation methodologies where existing ones fall short.

Contribute to internal knowledge sharing and best practices.

Deliverables

Benchmark proposal documents

Evaluation dataset designs

Iteration and feedback reports post-implementation

Optional research summaries or whitepapers

Timeline Expectations

Initial benchmark proposal within 1–2 weeks

At least one benchmark and one evaluation dataset per month

Ongoing iterations and monthly research updates

Required Skills & Experience

Strong foundation in AI/ML research , including deep learning, NLP, CV, and agent systems.

Hands-on experience in benchmark or dataset design .

Ability to synthesize academic research into practical evaluation frameworks.

Excellent written communication and documentation skills.

Comfortable working independently and collaboratively.

Impact

Continuous pipeline of innovative benchmarks

Strong research-driven evaluation standards

Enhanced credibility with clients and AI research partners