Smartsheet is seeking a Senior Software Engineer II - Applied AI and Evaluations to own agent quality end-to-end, focusing on LLM evaluation, prompt and context engineering, and retrieval-augmented generation.

Company: Smartsheet

Role: Senior Software Engineer II - Applied AI and Evaluations

Location: Remote, USA

Experience:

8+ years of software engineering experience
At least 2 years working directly with LLMs in production

Key Skills:

LLM evaluation
Prompt engineering
Context engineering
RAG architectures
Python
Databricks
MLflow

Qualification:

BS or MS in Computer Science, a related field, or equivalent industry experience
Legally eligible to work in the U.S. on an ongoing basis

Role Focus:

Own agent quality end-to-end: diagnosis, improvement, and validation
Identify failure modes across quality dimensions
Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
Extend and mature evaluation framework
Close the feedback loop

Nice to have:

Experience with MLflow or similar experiment tracking platforms
Familiarity with CI-integrated evaluation pipelines
Experience with multi-agent orchestration frameworks
Prior work in an Applied AI or LLMOps function within a product company

Other:

Employer subsidized medical/vision and dental coverage
401k Match
Monthly stipend
Flexible Time Away Program
Sick Time Off
Life insurance
Short-term and long-term disability plans
12 paid holidays per year
Up to 24 weeks of Parental Leave
Personal paid Volunteer Day
Opportunities for professional growth and development
Company Funded Perks
Teleworking options
Competitive base salary range: $175,000 - $245,000 USD

Senior Software Engineer II - Applied AI and Evaluations

Skills Required

Description