Product Manager, AI Quality & Evaluation

San Francisco, CA · Hybrid (Financial District)

About the company

An early-stage company building AI infrastructure for Enterprise IT. Its platform gives enterprise teams AI agents that operate from a living, continuously refreshed model of how their systems actually behave collapsing discovery and documentation work that normally takes weeks into hours, without giving up quality or control. The team is small, fast-moving, and assembling the group that will set the standard for how Enterprise IT runs in the AI era.

The role

Enterprise buyers will not hand an AI agent real authority over their systems unless the output can be trusted, and trust here is an engineering problem before it is a sales one. This Product Manager will own how the company measures agent quality: the evaluation tooling, the standards, and the process that lets the team ship quickly without eroding customer confidence.

The work spans the full evaluation lifecycle curating datasets, building the harnesses, iterating on them, and putting them into production. As frontier models ship on a near-monthly cadence, this person makes swapping in and assessing a new one cheap, fast, and routine rather than a fire drill. The role sits directly with the founders and the engineering team, and the person in it will define a measurement layer that becomes a durable advantage rather than a checkbox.

Responsibilities

1. Set the strategy and roadmap for evaluation across the product, and own the outcomes it produces
2. Build the shared measurement infrastructure that lets anyone on the team define what "good" means, run experiments, compare models, and monitor live behavior
3. Own the standards for LLM-as-judge setups, rule-based evaluators, human annotation, and production monitoring — including a clear point of view on where each one belongs and where it doesn't
4. Give the team a repeatable path to evaluate any newly released frontier model within days, plus a defensible framework for when a fine-tuned or specialized model earns its cost over a well-prompted general one
5. Keep the evaluation system model-agnostic so it stays an impartial referee as the underlying stack changes
6. Define the quality gates that run from early prototype through general availability and steady-state monitoring — which are hard blocks, which are advisory, and who owns the non-negotiable floors such as critical-error rates
7. Work across data ingestion, agent orchestration, and customer-facing surfaces, translating what breaks in production into what gets built next

What the company is looking for

1. 5+ years in product management with substantial ownership of ML-powered products or platform systems
2. Deep working knowledge of how model quality gets measured and improved evaluation frameworks, annotation pipelines, benchmark design
3. Strong technical fluency across ML, data pipelines, and distributed systems
4. Experience partnering closely with ML engineers and researchers to move things into production
5. Judgment to weigh long-term architectural investment against near-term quality wins
6. Clear communication, with the ability to turn dense technical material into decisions people can act on
7. A record of shipping in domains where accuracy, reliability, and trust are non-negotiable

Bonus points

1. Built evaluation platforms, ML observability systems, or quality measurement pipelines before
2. Worked in regulated or enterprise environments with a high bar for accuracy and auditability
3. Experience with domain-specific model adaptation or post-training
4. Familiarity with context ingestion frameworks, retrieval systems, or agentic workflows
5. Shipped large-scale ML products with human-in-the-loop workflows
6. Enterprise IT background in ERP, ITSM, systems integration, or large-scale migrations

Why this role

1. Ground-floor ownership of consequential decisions at an AI company going after a very large market
2. A direct line to the founders and impact visible from week one
3. Competitive compensation with equity, plus benefits including 100% employer-paid health plan coverage for employees