Senior Distributed Systems Engineer - AI Runtime

At Databricks, we run the world's largest AI platform. We believe the next breakthroughs will come from models trained on domain-specific enterprise data. That is why we built AI Runtime—a serverless product that makes GPU training of any size feel simple, allowing enterprise customers (including Rivian, Robinhood, and Merck) to train custom foundation and domain models directly alongside their enterprise data in Databricks.

As a Senior Software Engineer, you will work on the core distributed systems behind AI Runtime. You will solve technical challenges across GPU reliability, fault-tolerant model training, custom fused kernels, sequence packing, and agent-driven experiment launches, helping shape how Databricks' serverless AI training infrastructure evolves.

Key Responsibilities

• Platform Architecture & Features: Build core components of AI Runtime and contribute to Databricks' technical direction in serverless GPU training infrastructure.
• Reliability & Observability: Develop reliability, failure detection, and observability systems for a growing fleet of multi-tenant GPU clusters.
• Multi-Node Training: Solve multi-node training challenges, including distributed orchestration, GPU scheduling, high-throughput data loading, and fault-tolerant checkpointing.
• Performance & Efficiency: Optimize training speed and reduce compute costs by raising efficiency across deep learning model architectures, accelerator hardware, and communication layers.
• Developer & Agent Experience: Improve AI Runtime's developer experience for human engineers and autonomous AI agents.
• Hardware Preparedness: Help prepare AI Runtime for next-generation GPU hardware, specialized accelerators, and new ML workload types.
• End-to-End Ownership & Mentorship: Partner with research, product management, and customers on upcoming features, owning deliverables from design through production while mentoring team members.

Requirements

Experience & Education

• 5+ years of professional experience in large-scale distributed systems, with experience in GPU training, high-performance computing (HPC), or ML systems.
• Bachelor's degree in Computer Science or a related quantitative field (Master's or PhD preferred).

Technical Capabilities & ML Systems

• Hands-on experience training large-scale models using distributed parallelism frameworks (FSDP, DeepSpeed, or Megatron).
• Working knowledge of GPU performance, including accelerator architecture, high-speed interconnects (NVLink, InfiniBand), and custom kernels.
• Strong understanding of training resilience patterns, including asynchronous checkpointing, failure detection, and automatic recovery.
• Background in building and operating multi-tenant cloud platform products.

Engineering & Delivery

• Demonstrated track record of owning complex technical features end to end and driving them to customer impact.
• Clear technical communication skills in writing and in person.

Benefits & Total Rewards

• Compensation: Competitive base salary, performance bonus eligibility, and Databricks equity grants.
• Healthcare & Wellness: Comprehensive medical, dental, and vision insurance options.
• Flexibility & Perks: Generous paid time off, retirement savings plans, and local office perks across global Databricks hubs.