Sr. Software Engineer - Data Compute Platform | Roblox

Roblox’s data platform processes petabytes of data daily, powering analytics, machine learning, and core product decisions. As a Senior Software Engineer on the Data Platform Compute team, you will build and operate the compute layer beneath this ecosystem: the Kubernetes (EKS) clusters running all Apache Spark workloads, as well as the software enabling engineers, analysts, and ML practitioners to submit and execute jobs reliably and cost-effectively. You will own the systems sitting between internal users and the cluster—including Spark job submission APIs, internal libraries, interactive notebook compute (Spark Connect), auto-tuning engines, and cluster lifecycle automation.

Key Responsibilities

Compute Services & Interactive Infrastructure

• Core Compute Platform Services: Design, build, and operate the software backing Spark job submission, including API design, scheduling integrations, and shared user libraries.
• Interactive Spark at Scale: Deliver Spark Connect to power notebook workflows, provisioning and managing EKS compute infrastructure with robust isolation, quotas, and cost controls.
• Auto-Tuning Engine Development: Develop automated job tuners that analyze Spark history and event logs to recommend or apply configuration changes that optimize performance and compute spend.

Cluster Automation & Batch Scheduling

• EKS Fleet Management Automation: Build lifecycle automation for EKS clusters, executing blue-green rollouts to validate new AMIs and Kubernetes versions with zero disruption to active workloads.
• Batch Scheduling Ownership: Operate and upgrade YuniKorn-based batch scheduling, overseeing queue architectures, tenant quotas, fairness policies, and version migrations.
• Emerging Technology Evaluation: Evaluate next-generation compute technologies (e.g., native Spark acceleration via Apache Comet), executing benchmarks on real petabyte-scale workloads to drive adoption.
• Reliability & Cost Optimization: Define SLOs, build observability pipelines, and reduce operational toil, incidents, and platform spend across multi-tenant clusters.
• Cross-Functional Alignment: Partner with Data Engineering, ML, Analytics, and Infrastructure teams to streamline workflows and migrate users to optimized infrastructure patterns.

Requirements

Technical Depth & Platform Engineering Foundations

• Experience Track: 5+ years of professional experience building and operating production-grade distributed systems or core infrastructure platforms.
• Education: Bachelor of Science in Computer Science or equivalent practical experience.
• Spark on Kubernetes Expertise: Hands-on experience operating Apache Spark on Kubernetes (EKS, GKE, or self-managed) at the platform layer—including driver/executor behavior, resource tuning, and debugging failed or slow jobs.
• Kubernetes Depth: Strong operational experience with production Kubernetes clusters at scale (cluster upgrades, node lifecycle management, autoscaling, operators/controllers).
• Backend Language Proficiency: Strong proficiency in Go, Java, Scala, or Clojure with a track record of writing clean, maintainable backend code (Python proficiency is a plus for tuning tools).
• Cloud Infrastructure: Experience with AWS (EKS, EC2, IAM, S3) or GCP equivalents, alongside Infrastructure-as-Code (IaC).
• Operational Ownership: Experience taking systems from architecture through production, including on-call rotations, incident response, and SLO/metric observability.

Preferred Qualifications

• Experience with batch schedulers on Kubernetes (YuniKorn, Volcano, Kueue).
• Experience with interactive and gateway layers (Spark Connect, Livy, Kyuubi).
• Experience with Spark Operator, Karpenter, Helm, Terraform, or Argo.
• Experience building automated tooling using Spark event logs/metrics for cost attribution, tuning, or anomaly detection.
• Track record of maintaining internal Spark builds or executing Spark version upgrades.
• Familiarity with native execution engines (Comet, Gluten, Velox) or JVM performance tuning.
• Experience migrating legacy data workloads from YARN/Hadoop to Kubernetes.
• Active open-source contributions to the Apache Spark or Kubernetes ecosystems.
• Experience operating platforms at consumer-internet scale (100M+ users).

Total Rewards & Benefits

• Compensation & Equity: San Mateo base salary range ($243,290 – $295,250 USD) + competitive Roblox equity compensation.
• Healthcare & Wellbeing: Comprehensive health, dental, and vision insurance packages for employees and eligible dependents.
• Onsite Perks & Campus: Modern San Mateo headquarters featuring daily catered meals, team events, and state-of-the-art technical setups.
• Time Off & Retirement: Generous vacation policies, paid company holidays, paid parental leave, and 401(k) matching programs.