Check your CV before you apply
See your score the way hiring software reads it. If it needs work, build a stronger resume in the same place.
About the role
Ramp is hiring a Research Engineer, Benchmarks & Evaluation in New York, the United States. Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends. We automate how over $275B in annualized spend flows in and out of 85,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books.
Key responsibilities
- Every model decision at Ramp Labs runs through a benchmark. Which model to route to. Whether a post-training run worked. Whether an agent is ready to post a corrective journal entry to a customer's ledger without a human in the loop
- You'll own Ramp's benchmarking effort. You'll turn real finance work (expense policy review, invoice coding, month-end close, spreadsheet Q&A) into benchmarks and RL environments, build the graders, and tell the team what is actually improving
- Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment
- Own the benchmarking effort: decide what we measure, run benchmark suites across models and agent versions, and report results the team makes decisions on
- Design graders: rubrics, programmatic checks and llm-as-judge, calibrated against human agreement, with a clear view of where each one breaks
- Close the loop with post-training: turn evals into reward functions and RL environments, and catch reward hacking and contamination before they ship
- Publish external benchmarks and write-ups on how models perform at real finance work
- Hands-on experience building LLM benchmarks or evals that people used to make real decisions
- Strong Python and software engineering; your eval code is reproducible and fast
- Statistical judgment: variance, confidence intervals, sample size, inter-rater agreement, and knowing when a delta is noise
About Ramp
Ramp is building the smart infrastructure for finance teams, embedded in the transaction flow of every dollar a business spends. We automate how over $275B in annualized spend flows in and out of 85,000+ companies: authorizing payments, flagging risk, categorizing spend, and closing books.
We hire people with high agency and high urgency. We look for slope over intercept. We care less about where you trained and more about what you’ve built. At Ramp, everyone is a builder who owns problems end to end and makes consequential decisions that shape the outcome.
Location and schedule
- Based in New York, the United States (hybrid role)
- Build benchmarks from real workflows: turn agent traces, customer edge cases and expert labels into versioned benchmark suites for long-horizon agents that run for hours across tools and systems, plus tool use, multi-step reasoning and financial judgment
- Relocation expense coverage to NYC or SF (if needed)
Salary and benefits
The posting states a base salary of USD 204,400 to 352,000 a year. Benefits and what the employer offers, as the posting lists them:
- Centralized home-office equipment ordering
- Health and wellness stipend
- Budget for intra-office travel
- Weekly coffee stipend
- 100% medical, dental & vision insurance coverage for you, with partial coverage for dependents
- One Medical annual membership
How to apply
Apply on the company's own site.
Source: Employer Career Pages · The Briefed never charges candidates.
Similar roles
More Technology & IT roles near this one