The Role

This is a hybrid role. You will handle traditional Automated Testing for our platform UI and APIs, but your primary challenge will be AI Evaluation (Evals). You’ll be reporting to the Head of Engineering to define what "Good" looks like for an AI-driven retention moment and building the systems to measure it.

Key Responsibilities

• AI Evaluation Frameworks: Design and run "Evals" to catch hallucinations, logic errors, or "regressions" in our AI agents' decision-making processes.
• End-to-End Automation: Build and maintain a robust test suite (using Playwright or Cypress) that covers the entire journey from data ingestion (Databricks/Snowflake/Shopify) to message delivery (Klaviyo/Braze).
• Data Integrity Testing: Since our AI is only as good as the data it eats, you will build automated checks to ensure customer order histories and behavioral signals are being processed correctly.
• Edge Case Discovery: You are the "Professional Skeptic." You’ll hunt for scenarios where the AI might over-message a customer or provide a nonsensical cross-sell recommendation.
• Performance & Load Testing: Ensure our decision engine can handle the "Black Friday" level traffic spikes our enterprise clients face without breaking a sweat.

The Toolkit We’re Looking For

• Testing Mastery: 3+ years in QA/SDET roles. You’ve built automation frameworks from scratch (Python or JavaScript preferred).
• AI-Curious/Literate: You understand that LLMs and predictive models aren't deterministic. You’re familiar with concepts like RAG evaluation, prompt sensitivity testing, or model drift.
• Integrations Expert: You have experience testing third-party SaaS integrations and APIs. If you’ve worked with Shopify’s API or MarTech stacks, that’s a massive plus.
• SQL & Data Savvy: You can write complex queries to validate that the "ground truth" in the database matches what the AI is claiming.
• Startup Agility: You prefer a "Move Fast and Ship" environment. You can balance the need for perfect coverage with the reality of a rapidly evolving product.

Why Replenit?

• Shape the Standard: There is no "industry standard" for testing autonomous e-commerce agents yet. You get to help define it.
• Direct Impact: Reporting to a Co-founder means your quality reports aren't ignored—they shape the roadmap.
• Future-Proof Your Career: Transitioning from "Manual/Automation QA" to "AI QA" is the most valuable move you can make in the 2026 job market.

Ideal Candidate Profile

• Experience with Playwright, Pytest, or K6.
• Familiarity with LLM eval tools (e.g., LangSmith, Ragas, or WhyLabs).
• A "Product-First" mindset: You don't just find bugs; you suggest how the feature should actually work to better serve the user.

Job Type: Full-time

Pay: 8,935.10zł - 12,506.18zł per month

Work Location: Remote