Smartsheet is seeking a Senior Software Engineer II - Applied AI and Evaluations to own agent quality end-to-end. This role involves diagnosing agent failures, designing systems to catch regressions, and driving measurable improvements across the orchestrator and subagent fleet. The ideal candidate will have experience with LLMs, prompt engineering, and evaluation frameworks.

Company: Smartsheet

Role: Senior Software Engineer II - Applied AI and Evaluations

Location: Remote, USA

Experience:

8+ years of software engineering experience
At least 2 years working directly with LLMs in production

Key Skills:

LLMs
Prompt engineering
Context engineering
RAG architectures
Retrieval-augmented generation
Python
Databricks
MLflow

Qualification:

BS or MS in Computer Science, a related field, or equivalent industry experience

Role Focus:

Own agent quality end-to-end: diagnosis, improvement, and validation
Identify failure modes across quality dimensions
Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
Extend and mature evaluation framework
Close the feedback loop to ensure measurable quality signal

Nice to have:

Experience with MLflow or similar experiment tracking platforms
Familiarity with CI-integrated evaluation pipelines
Experience with multi-agent orchestration frameworks
Prior work in an Applied AI or LLMOps function within a product company

Other:

Employer subsidized medical/vision and dental coverage
401k Match (50% of contribution up to 6% of eligible pay)
Monthly stipend for work and productivity
Flexible Time Away Program, plus Sick Time Off
Life insurance, short-term, and long-term disability plans
12 paid holidays per year
Up to 24 weeks of Parental Leave
Personal paid Volunteer Day
Opportunities for professional growth and development
Company Funded Perks (counseling membership, local retail discounts, Smartsheet account)
Teleworking options from any registered location in the U.S.
Competitive base salary range: $175,000 - $245,000 USD

Senior Software Engineer II - Applied AI and Evaluations

Skills Required

Description