Responsibilities

• Design and maintain evaluation rules: Define clear, testable pass/fail criteria for Mia’s conversational behavior by translating customer workflows and success metrics into evaluation standards. Apply leadership-defined severity frameworks to prioritize what matters most.
• Build and manage test suites: Create structured test suites to proactively surface failure modes before launch. Organize tests by risk category and document explicit success criteria so quality can be validated consistently.
• Review simulations and make launch decisions: Review 100+ simulated conversations prior to each bot launch. Identify recurring failure patterns, group issues by type, quantify impact, and block launches when failure rates or severity thresholds are too high.
• Run regression testing: Re-test after model, prompt, workflow, or product updates to detect regressions introduced by changes. Escalate issues before they reach customers.
• Monitor and correct rule drift: Review dashboards weekly to spot drift or unexpected rule behavior. Use MCP/Cursor to recalibrate rules, and partner with Customer Success to prioritize fixes based on customer impact.
• Support quality reporting: Collaborate with the Business Analyst to define metrics and requirements for pre-launch quality reports, including failure modes, severity summaries, readiness status, and recommended next steps.

Qualifications

• 3+ years of experience in software QA, test case development, conversation review, trust & safety, or a closely related role
• Exceptional attention to detail and consistency—you catch edge cases others miss
• Strong written communication skills with the ability to clearly explain what’s broken and why it matters
• Strong pattern recognition skills—you identify themes across large volumes of conversations, not just individual failures
• Comfort using modern AI tools (e.g., Cursor) in day-to-day workflows
• Proactive ownership and risk awareness—you flag issues early rather than waiting for deadlines

Nice to Have

• Experience writing guidelines, rubrics, rules, prompts, or evaluation criteria that define pass/fail outcomes
• Background reviewing voice or chat interactions at scale
• Adversarial testing experience (e.g., fraud detection, cybersecurity, abuse testing, or edge-case analysis)
• Healthcare or medical practice experience
• Startup experience building QA or quality processes from scratch

What Success Looks Like

• Bots go live without customers discovering preventable issues
• Evaluation rules catch 80%+ of failure modes during simulation rather than after launch
• Regressions introduced by bot updates are identified and resolved before production
• You can clearly explain failure patterns in any bot you’ve reviewed without needing to look them up
• Rules remain calibrated—drift is detected and corrected weekly before compounding
• Pre-launch quality reports clearly communicate what is ready, what is not, and why

About Hello Patient Hello Patient is transforming healthcare communication through advanced conversational AI. After spending 3.5 years scaling Carbon Health's technology during its hypergrowth, our founding team witnessed firsthand how much clinical time gets wasted on basic communication—great clinicians buried in administrative work, patients frustrated by missed connections and phone tag.

Our enterprise-grade AI platform proactively engages patients through perfectly timed calls and texts, naturally resolving complex interactions end-to-end—from patient acquisition and scheduling to retention. Built with security and compliance at its core, our technology integrates seamlessly with existing healthcare infrastructure to deliver 24/7 patient engagement while enabling staff to focus on what matters most: providing exceptional care.

Backed by Scale and 8VC, we're building the future of healthcare communication, one conversation at a time.