Experience:

• 4–10+ years of engineering experience
• 2–4 years focused on AI/ML

Qualification:

• Bachelor’s or Master’s degree in Computer Science, Engineering, AI/ML, or a related field

Role Focus:

• Champion the adoption of AI-powered testing tools and practices across the QA Guild
• Identify gaps in current QA processes and introduce AI-driven solutions
• Design and maintain evaluation frameworks for LLM outputs
• Build automated validation pipelines for LLM outputs
• Validate RAG architectures and autonomous agent workflows
• Develop and maintain scalable UI and API automation frameworks

Additional responsibilities:

• Serve as the internal subject matter expert on AI quality standards
• Collaborate with QA leads, engineering managers, and product teams
• Create synthetic and benchmark datasets for AI testing scenarios
• Evaluate latency, token usage, model cost, throughput, and scalability of AI systems
• Integrate automation and quality gates into CI/CD pipelines

Nice to have:

• Experience with Robot Framework and Playwright
• Knowledge of LangChain, LangGraph, LlamaIndex, CrewAI, or Semantic Kernel
• Familiarity with MCP (Model Context Protocol)
• Hands-on experience with developer AI tools
• Exposure to MLOps workflows, performance testing, security testing, or AI governance and compliance requirements
• Experience with contract testing, chaos testing, or resilience engineering
• AI Red Teaming or LLM security testing experience

More skills:

AI/ML, LLM testing, prompt engineering, context engineering, AI Evals, RAG systems, AI agents, CI/CD pipelines, API testing, microservices testing, UI automation

Other:

• Three work environment scenarios: work-from-home, hybrid, or work-from-the-office
• Work-life balance
• Diversity and inclusion programs
• Excellent benefits: OSDE 310 Health Care, Monthly Salary Reviews, Tuition Reimbursement, Internet Reimbursement, In-company English Lessons, LinkedIn learning, Referral Program Bonus, F5 extra days off, Annual Bonus