Software Engineer, Spark Platform, DoorDash (San Francisco)

DoorDash connects customers with local businesses for delivery, from restaurants to grocery, retail and its own DashMart stores. The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem — the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve.

Key responsibilities

• Build and operate an in-house Spark platform that runs at company-wide scale, spanning runtime, scheduler, reliability, and user-facing tooling
• Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement that let a small team serve dozens of consumer teams
• Own pieces of cluster lifecycle automation — provisioning, upgrades, capacity changes
• Build the observability and incident automation that make the platform debuggable end-to-end and keep on-call sustainable as the team
• Partner with senior engineers on shuffle, runtime, and architecture work, and grow into deeper ownership of those areas over time

Requirements

• B.S., M.S., or PhD in Computer Science or equivalent
• 2+ years of industry experience operating production distributed systems
• Experience operating Apache Spark at scale on Amazon EMR, Databricks
• Hands-on experience operating production systems on Kubernetes — controllers, operators, custom resources
• Familiarity with batch or big-data schedulers (YuniKorn, Volcano, Kueue, or equivalent) and/or with the Spark-on-Kubernetes operator
• Familiarity with observability stacks (Prometheus, OpenTelemetry, distributed tracing, structured logging) and with defining SLOs

Location and schedule

• Onsite in San Francisco, California
• You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team
• Build and operate an in-house Spark platform that runs at company-wide scale, spanning runtime, scheduler, reliability, and user-facing tooling

Salary and benefits

US base salary $130,600 to $192,000 a year. DoorDash also offers benefits; the full list is on the posting.

How to apply

Apply on the company's own site.