Senior Software Engineer, Spark Platform (San Francisco)

DoorDash connects customers with local businesses for delivery, from restaurants to grocery, retail and its own DashMart stores. The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem — the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve.

Key responsibilities

• Set the multi-year technical direction for an in-house Spark-on-Kubernetes platform — runtime, shuffle, scheduler, reliability —
• Own the deepest distributed-systems problems on the team: shuffle architecture, multi-tenant scheduling, runtime performance
• Partner with the Engineering Manager on technical roadmap, hiring, interview design, and team shape as the team continues to grow
• Uplevel the rest of the team through design reviews, mentorship, and raising the bar on what we ship
• Represent Spark Platform in cross-team architecture forums and shape how data, analytics, and ML workloads land on the platform

Requirements

• B.S., M.S., or PhD in Computer Science or equivalent
• 6+ years of industry experience designing and operating distributed systems at scale
• Deep, hands-on experience with Apache Spark — internals, query execution, shuffle, the executor/driver model — at platform scale on Amazon EMR
• Remote/external shuffle systems (Celeborn, Magnet, Cosco, or similar), batch/big-data schedulers (YuniKorn, Volcano, Kueue
• Strong fluency operating workloads on Kubernetes in production — operator patterns, executor pod lifecycle, network topology
• Familiarity with data lake table formats such as Apache Iceberg or Delta Lake, and with the query and SQL engines that read them

Location and schedule

• Onsite in San Francisco, California
• You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team
• Set the multi-year technical direction for an in-house Spark-on-Kubernetes platform — runtime, shuffle, scheduler, reliability — and make the architectural calls that compound for years

Salary and benefits

US base salary $159,800 to $235,000 a year. DoorDash also offers benefits; the full list is on the posting.

How to apply

Apply on the company's own site.