Responsibilities

• Automation Pipelines: Build Python-based pipelines for automated quality testing of AI responses.
• Integrate LLMs into automated evaluation frameworks (e.g., using GPT-based evaluators, embeddings, or custom scoring).
• Automate regression and stress testing for conversational AI flows.
• Quality & Evaluation: Define evaluation metrics (relevance, factuality, coherence, safety, empathy).
• Implement both rule-based and AI-driven quality checks.
• Monitor model drift, bias, and hallucinations using automated workflows.
• Integration & Deployment: Work with APIs, SDKs, and CI/CD pipelines to embed automated AI evaluation in production.
• Develop monitoring dashboards to visualize conversation quality.
• Collaborate with ML engineers, product managers, and QA teams to close the feedback loop.
• Optimization & Innovation: Experiment with prompt engineering and automated prompt-testing frameworks.
• Explore reinforcement learning, self-critique models, or human-in-the-loop automation for continuous improvement.
• Automate compliance and policy adherence checks for enterprise AI systems.

Requirements

• Strong 2-5 Years of experience in Python development (automation, scripting, data handling).
• Experience with LLMs/NLP frameworks.
• Understanding of MLOps / AI deployment pipelines.

Netomi is an equal opportunity employer committed to diversity in the workplace. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, disability, veteran status, and other protected characteristics.