Staff Software Engineer - Observability | Reddit

As a Staff Software Engineer on the Observability (OBS) team, you will work at the intersection of infrastructure systems engineering and software development. You will own, scale, and evolve Reddit’s telemetry platform—which ingests billions of data points per minute across metrics, logs, and distributed traces. Operating one of the world's largest deployments of Prometheus and Thanos, you will lead performance engineering for distributed query engines, scale logging platforms based on Vector and ClickHouse, and architect OpenTelemetry (OTEL) tracing pipelines. In this high-impact role, you will build Kubernetes operators, automate event-driven workflows, contribute upstream changes to core open-source projects, and empower thousands of internal engineers to debug and optimize their production services.

Key Responsibilities

Telemetry Platform Architecture & Scale

• Monitoring & Query Performance: Engineer and optimize Reddit’s large-scale Prometheus and Thanos monitoring deployment—processing billions of metrics per minute and solving complex distributed query performance challenges.
• High-Throughput Logging: Operate and enhance the enterprise logging platform built on Vector and ClickHouse, processing millions of log streams daily for debugging, deployment tracking, and compliance.
• Distributed Tracing Evolution: Architect and expand OpenTelemetry (OTEL), ClickHouse, and Grafana-based distributed tracing pipelines, building new visualization and analytics capabilities.
• Open-Source Upstream Contributions: Actively contribute upstream bug fixes, feature enhancements, and performance optimizations to core open-source telemetry projects (Prometheus, Thanos, Vector, OTEL).

Automation, Kubernetes & Operational Excellence

• Kubernetes Platform Engineering: Develop and deploy custom Kubernetes controllers and operators to automate the provisioning, lifecycle, and scaling of observability components.
• Event-Driven Automation: Automate critical stages of the event-driven development process, reducing operational toil for product engineering pods across Reddit.
• Reliability & On-Call: Share on-call responsibilities, driving post-incident root-cause analysis and systemic reliability improvements for core telemetry infrastructure.

Requirements

Distributed Systems & Telemetry Expertise

• Experience Track: 7+ years of professional software engineering experience developing internet-scale software, specifically within infrastructure or platform engineering domains.
• Distributed Systems Architecture: Proven experience building, scaling, and operating complex, fault-tolerant distributed systems.
• Kubernetes Development: Deep experience developing applications on top of Kubernetes or similar cloud-native container orchestrators (direct experience developing Kubernetes operators or controllers is a major advantage).
• Troubleshooting Depth: Exceptional troubleshooting and debugging capabilities spanning both application software code and Linux infrastructure layers.
• Self-Directed Execution: Demonstrated track record of scoping large technical projects, tracking execution milestones, and acting as a self-starter across organizational boundaries.
• Communication & Collaboration: Excellent written and verbal communication skills to collaborate effectively within a service-oriented engineering culture.

Preferred Qualifications

• Direct hands-on experience with core observability software stacks: Prometheus, Thanos, Grafana, Vector, ClickHouse, OpenTelemetry (OTEL), or Loki.
• Experience implementing custom query parsers, TSDB storage engines, or distributed caching layers for telemetry databases.
• Active maintainer or contributor status within CNCF (Cloud Native Computing Foundation) observability projects.

Total Rewards & Benefits

• Compensation & Equity: US Remote base salary range ($217,000 – $303,900 USD) + competitive Restricted Stock Unit (RSU) equity grants + potential commission eligibility.
• Healthcare & Wellbeing: Comprehensive health, dental, and vision insurance packages; income replacement programs; mental health coaching benefits; and gender-affirming care coverage.
• Retirement & Financial: 401(k) retirement savings plan with generous employer matching.
• Family Planning & Support: Comprehensive family planning support, generous paid parental leave, and caregiving assistance programs.
• Workplace & Lifestyle: Fully remote flexible-first work environment, home office/workspace stipends, professional development allowances, flexible PTO, and paid volunteer time off.