Experience:

• 3+ years in software engineering, ML/NLP workflows, or applied LLM prompt engineering

Qualification:

• Strong writing precision for structured instructions, role prompts, and multi-step reasoning constraints
• Experience with LLM evaluation methods such as rubrics, pairwise ranking, golden sets, and adversarial testing
• Familiarity with RLHF concepts, preference data, and human feedback loops
• Working knowledge of Python and basic tooling for experimentation (notebooks, scripts, JSON, APIs)

Role Focus:

• Craft and iterate high-performing prompts for LLM-based applications
• Build reusable prompt templates
• Define evaluation criteria to improve reliability, groundedness, and safety
• Run prompt experiments (A/B tests) and analyze outputs
• Create preference/ranking data for RLHF-style training
• Partner with engineering to integrate prompts into products such as chatbots, copilots, agents, and RAG systems
• Support model performance improvement, training data quality, and consistent behavior across domains
• Own prompt lifecycle management including writing, debugging, versioning, and library maintenance
• Design prompt evaluation plans using rubrics for accuracy, completeness, instruction following, tone, and policy compliance
• Produce high-quality preference data and rationales for RLHF and instruction tuning workflows
• Collaborate with stakeholders to translate product requirements into system, developer, and user prompts
• Improve grounding and reduce hallucinations using retrieval-augmented generation (RAG) strategies and citation/traceability constraints
• Create and maintain annotation guidelines compliance for evaluators including edge-case handling and escalation paths
• Monitor quality metrics such as pass rate, inter-annotator agreement, and regression failures across releases
• Support content safety labeling and red-teaming prompts to prevent policy violations, sensitive data leakage, and jailbreak behaviors

Additional responsibilities:

• Collaborate with engineers and researchers to write system prompts and build prompt libraries
• Run A/B tests and define rubrics to measure model performance improvement across accuracy, helpfulness, and content safety
• Manage QA evaluation workflows at scale
• Collaborate with annotation vendors or BPO teams

Nice to have:

• Experience with RAG pipelines, vector search, embedding evaluation, and grounded answer validation
• Knowledge of content safety labeling, policy taxonomy, and prompt-based guardrails
• Exposure to named entity recognition, information extraction, or domain-specific ontologies for structured outputs
• Familiarity with LLM observability and evaluation tooling such as unit tests for prompts, regression suites, and eval harnesses

More skills:

LLM Evaluation, NLP

Other:

• Work supports domains such as customer support, coding help, research summarization, and enterprise knowledge search
• Salary range USD 63,360 to 126,720 per year
• Employment type: Full-time
• Industry: Technology
• Job function: Engineering
• Experience level: Mid-Senior
• Role suitable for candidates from AI labs, tech startups, enterprises, BPOs, and annotation vendors
• Applications reviewed for prompt clarity, evaluation rigor, and ability to improve training data quality and end-user reliability