Senior Software Engineer - Agentic AI & Python

At Klaviyo, we believe the future of software lies not only in tools that help people work efficiently, but in intelligent systems that can take action, learn from outcomes, and improve customer experiences over time. Serving over 167,000 customers and processing billions of consumer profiles and conversion signals, Klaviyo provides a unique environment for building state-of-the-art AI systems at scale.

We are looking for a Senior Software Engineer, Agent Runtime & Customer Workflows to join the Customer Agent team—Klaviyo’s AI-native conversational platform. In this role, you will design and build scalable backend systems and AI-driven product experiences that enable autonomous agents to reason, retrieve context, execute tools, and carry out reliable multi-step actions on behalf of customers.

Key Responsibilities

• Agentic Infrastructure & APIs: Design and build reliable, high-throughput backend services and APIs powering AI-driven customer workflows, tool use, orchestration, and context retrieval.
• Applied AI & Hallucination Mitigation: Solve complex applied AI challenges across RAG, context grounding, model selection, hallucination reduction, and hybrid/deterministic logic vs. LLM execution.
• Evaluation Frameworks: Develop automated and human-in-the-loop evaluation strategies, evaluator calibration frameworks, regression suites, and production signal metrics.
• Production Guardrails & Reliability: Implement production guardrails, fallbacks, retry logic, observability, and failure-handling mechanisms for agentic systems.
• Distributed Workflows: Build asynchronous and distributed processing workflows (e.g., event queues, task systems) capable of processing high-volume AI workloads.
• Customer & Cross-Functional Collaboration: Partner directly with external customers, product managers, and cross-functional teams to translate feedback and workflow friction into production technical solutions.
• Operational Excellence: Participate in the team’s on-call rotation to diagnose, mitigate, and conduct postmortems on production incidents.

Requirements

Experience & Education

• 5–7 years of professional software engineering experience with a strong foundation in backend systems and distributed cloud applications.
• Hands-on, practical experience building and deploying generative AI or agentic AI applications into production for real users.

Technical Capabilities

• Strong proficiency in Python and modern backend application frameworks (e.g., FastAPI, Django).
• Hands-on experience with asynchronous processing, event-driven architectures, and distributed task queues (Celery, Kafka, AWS SQS, RabbitMQ, or Redis).
• Deep understanding of database design, data modeling, REST/gRPC APIs, and persistence patterns.
• Comfort operating within cloud-native environments (AWS, Docker/containers, Kubernetes, infrastructure-as-code, CI/CD pipelines).
• Proven ability to evaluate tradeoffs across quality, latency, cost, reliability, and architectural complexity.

Preferred Qualifications

• Experience training, fine-tuning, distilling, or adapting machine learning and language models for production workflows.
• Hands-on experience with reinforcement learning or feedback-driven optimization techniques.
• Demonstrated background engineering AI evaluation platforms (e.g., LLM-as-judge systems, benchmark suites, quality metrics platforms).
• Experience optimizing inference cost, latency, or throughput for AI systems operating at significant scale.