Back to results
Technology & IT

Lead Product Data Scientist, Agents at Klaviyo, Boston, US

On-site · Boston, Massachusetts Lead USD 172k–258k / yr Posted just now International
Apply on company site ↗
Not ready to apply yet?
Free ATS check

Check your CV before you apply

See your score the way hiring software reads it. If it needs work, build a stronger resume in the same place.

About the role

Klaviyo is hiring a Lead Product Data Scientist, Agents in Boston, the United States. At Klaviyo, we value the unique backgrounds, experiences and perspectives each Klaviyo (we call ourselves Klaviyos) brings to our workplace each and every day. We believe everyone deserves a fair shot at success and appreciate the experiences each person brings beyond the traditional job requirements. If you’re a close but not exact match with the description, we hope you’ll still consider applying. Want to learn more about life at Klaviyo?.

Key responsibilities

  • We're looking for a Lead Product Data Scientist, Agents to own and advance Klaviyo's measurement and data science strategy across Composer and Customer Agent, combining deep technical expertise with strong product and business judgment to influence some of our most important AI investments.
  • Own agent measurement end to end across Composer and Customer Agent, developing comprehensive measurement frameworks that connect adoption, engagement, agent performance, customer value, and revenue impact
  • Define what successful agent performance looks like by partnering with Product and Engineering to establish meaningful measures of agent effectiveness, task completion, customer outcomes, failure rates, and the trade-offs between quality, cost, and business value.
  • Build scalable agent evaluation frameworks by combining conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification to assess quality, identify failure patterns, and understand how changes to models and agent behavior influence product performance.
  • Measure causal impact and incremental value through experimentation and advanced statistical methods, including quasi-experimental approaches such as difference-in-differences, matched comparisons, staged rollouts, and synthetic controls, to determine how AI agents influence customer behavior, retention, and revenue.
  • Connect agent performance to business strategy by partnering with Finance and Product teams to evaluate usage-based pricing, packaging, credit economics, model costs, conversion, and the financial implications of product investments
  • Establish trusted metrics and analytical standards by defining consistent measurement methodologies, governed data models, and evaluation practices that enable teams to make informed decisions as AI products and underlying technologies evolve

Requirements

  • You have 10+ years of experience in data science, product analytics, applied statistics, or a related quantitative discipline, including recent hands-on experience measuring AI, machine learning, or LLM-powered products in production
  • You bring deep expertise in product measurement, experimentation, statistical modeling, and causal inference, with the ability to select and defend rigorous analytical approaches when traditional randomized experiments aren't feasible
  • You have experience evaluating AI systems in production, including designing evaluation frameworks, analyzing agent interactions or conversations, leveraging human labeling or LLM-as-judge methodologies, and understanding how model, prompt, and tool changes influence product outcomes.
  • You're highly proficient in SQL and Python or R, with experience developing production-quality analytical solutions and familiarity with dbt and modern analytics engineering practices
  • You take an AI-first approach to your work, actively leveraging coding agents, LLMs, and emerging tools to accelerate analytical workflows while understanding their limitations and ensuring the accuracy and reliability of their outputs
  • You've demonstrated technical leadership by setting direction on complex analytical initiatives, mentoring other data scientists, reviewing methodologies, and influencing technical standards across teams

Nice to have

  • Experience measuring and optimizing AI agents, conversational AI, or LLM-powered products in a high-growth B2B SaaS or product-focused technology company
  • Hands-on experience with agent observability and evaluation platforms such as Braintrust, LangSmith, or Arize, including working with agent traces, tool-call data, and evaluation pipelines
  • Experience with usage-based or consumption-based pricing, credit models, AI infrastructure costs, or LLM unit economics

About Klaviyo

Klaviyo's Product Data Science team plays a critical role in shaping how we understand, measure, and improve product and business performance. We partner closely with Product, Engineering, Finance, and Go-to-Market teams to turn complex data into actionable insights that inform product strategy, customer experience, and long-term growth.

Location and schedule

  • Based in Boston, the United States (onsite role)
  • Base Pay Range For US Locations

Salary and benefits

The posting states a base salary of USD 172,000 to 258,000 a year.

How to apply

Apply on the company's own site.

Applying for a ai and machine learning engineer role?See the resume keywords 218 real job posts asked for.See keywords →
Klaviyo
Listed via Employer Career Pages
View companies

Source: Employer Career Pages · The Briefed never charges candidates.

The Weekly Brief

New roles, in your inbox.

One email a week — the best hand-picked roles, salary insights and career reads. No spam, unsubscribe any time.