Description
Quality Engineering & Delivery • Own end-to-end quality assurance for AI/GenAI engagements including test strategy, planning, execution, defect management, reporting, and exit criteria • Lead and mentor QA pods of 3–6 engineers across automation frameworks, AI testing practices, and delivery execution • Design and execute functional, API, backend, integration, regression, performance, and AI evaluation test cases • Build and maintain scalable automation frameworks for Web, Mobile, and API testing • Drive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation tools AI Application Testing & Evaluation • Define AI-specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool-call correctness, latency, bias, and safety testing • Design and execute testing strategies for RAG architectures, multi-agent workflows, OCR systems, and Speech-to-Text applications • Build synthetic datasets covering edge cases, adversarial scenarios, persona-based workflows, and multimodal inputs • Implement LLM-as-a-Judge evaluation pipelines using customizable scoring rubrics and multi-model evaluations • Produce detailed AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metrics Automation, CI/CD & Tooling • Integrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelines • Build regression dashboards, quality gates, and automated reporting frameworks • Perform API testing, backend validation, database testing, and event-driven workflow validation • Conduct load and performance testing using modern performance engineering tools • Support cross-browser, parallel, and scalable automation execution environments Client & Stakeholder Management • Act as the client-facing QA lead for enterprise engagements • Conduct quality reviews, risk assessments, and stakeholder reporting sessions • Translate technical quality metrics into business-focused outcomes and recommendations • Collaborate with engineering, AI/ML, product, and leadership teams to ensure successful delivery outcomes