Staff Engineer - AI Training Infrastructure & GPUs

At Databricks, we run the world's largest AI platform. We believe the next breakthroughs will come from models trained on domain-specific enterprise data. That is why we built AI Runtime—a serverless product that makes GPU training of any size feel simple, allowing enterprise customers (including Rivian, Robinhood, and Merck) to train custom foundation and domain models directly alongside their enterprise data in Databricks.

As a Staff Software Engineer, you will work on the core distributed systems behind AI Runtime. You will solve high-stakes technical challenges across GPU reliability, fault-tolerant model training, custom fused kernels, sequence packing, and agent-driven experiment launches, shaping Databricks' long-term AI training infrastructure architecture.

Key Responsibilities

• Architecture & Technical Direction: Drive major components of AI Runtime's architecture and help shape Databricks' long-term technical direction in serverless GPU training infrastructure.
• Reliability & Observability: Design reliability, failure detection, and observability systems for a rapidly growing, multi-tenant fleet of GPU clusters.
• Multi-Node Training: Take on complex multi-node training challenges, including distributed orchestration, smart GPU scheduling, high-throughput data loading, and fault-tolerant checkpointing.
• Performance & Efficiency: Optimize training speed and reduce compute cost by raising efficiency across deep learning model architectures, accelerator hardware, and communication layers.
• Developer & Agent Experience: Lead improvements to AI Runtime's developer experience for human engineers and autonomous AI agents.
• Next-Gen Hardware Strategy: Evaluate and choose technical bets preparing AI Runtime for emerging GPU generations, specialized accelerators, and new ML workload types.
• Cross-Functional Leadership & Mentorship: Partner with AI research, product management, and enterprise customers to define product roadmaps while championing engineering excellence and mentoring senior engineers.

Requirements

Experience & Education

• 10+ years of professional experience in large-scale distributed systems, with significant depth in GPU training, high-performance computing (HPC), or ML systems.
• Bachelor's degree in Computer Science or a related quantitative field (Master's or PhD preferred).

Technical Capabilities & ML Systems

• Hands-on experience training large-scale models using distributed parallelism frameworks (FSDP, DeepSpeed, or Megatron).
• Expertise in GPU performance optimization, accelerator architecture, high-speed interconnects (NVLink, InfiniBand), and custom kernel execution.
• Deep understanding of distributed training resilience patterns, including asynchronous checkpointing, failure detection, and automatic job recovery.
• Proven track record building and operating multi-tenant cloud platform products at enterprise scale.

Leadership & Delivery

• Demonstrated history of leading cross-team technical initiatives from architectural design through customer impact.
• Exceptional technical communication skills with a reputation as an approachable mentor and technical leader.

Benefits & Total Rewards

• Compensation: Competitive base salary, performance bonus eligibility, and Databricks equity grants.
• Healthcare & Wellness: Comprehensive medical, dental, and vision insurance options.
• Flexibility & Perks: Generous paid time off, retirement savings plans, and local office perks across global Databricks hubs.