Description
Role Description▍ About the RoleWe're hiring a QA Engineer to test our Agent-as-a-Service conversational AI platform for Arabic-speaking customers, focused on Kuwaiti and Najdi dialects. This isn't traditional QA — expected output isn't an exact string match. You'll judge whether the agent understood intent, replied in natural correct dialect, followed business rules, held context, retrieved accurate info, and stayed safe, across real conversations where correctness is linguistic and contextual, not binary.Arabic LLM research consistently shows MSA fluency does not predict dialect fluency — dialect, culture, code-switching, reasoning, and safety fail independently and must be tested as separate dimensions, not assumed to travel together.▍ Responsibilities1. Dialect & Linguistic QATest Kuwaiti/Najdi responses for: dialect authenticity (no unprompted MSA drift), grammar, vocabulary, tone/register, naturalness, spelling/punctuation, Arabic-English code-switching, numeral style, Arabizi. Flag answers that are technically correct but don't read as native.2. Conversational QATest full multi-turn conversations, not isolated prompts: context retention, pronoun resolution, corrections, topic switching, long threads, angry/emotional customers, typos, voice-transcribed Arabic, mixed Arabic/English. Confirm the agent never loses context or contradicts itself.3. Intent & Instruction FollowingVerify correct handling of booking/reschedule/cancel, pricing/service questions, complaints, escalation triggers, multi-intent and implicit/ambiguous messages. Verify business rules hold even under user attempts to override them.4. RAG & Knowledge QACheck answers against source knowledge; catch hallucinations, missing/conflicting/outdated info, and ambiguous-document confusion. Distinguish a correct+grounded answer from a correct-looking but unsupported one. Include dialectal queries against MSA knowledge sources.5. Cultural & Regional QACheck local terminology, Kuwaiti/Saudi conventions, names/places, dates, currency, formality register, local expressions/idioms — as its own pass, not a grammar check.6. Safety & Adversarial TestingRun prompt injection, jailbreaks, instruction override, system-prompt extraction, abusive/harmful input, social engineering, malicious links — in Kuwaiti/Najdi Arabic, not just English. Arabic-specific safety blind spots don't show up in English-only testing.7. Regression SuiteEvery confirmed bug (dialect, hallucination, context, intent, RAG, safety, prompt-injection) becomes a permanent test case. Run full suite after every prompt/model/pipeline change, diff against prior version.▍ Example of the BarUser: ممكن تحجزلي موعد باجر العصر؟Not is the answer correct but: did it parse باجر / العصر correctly, keep dialect, sound native, ask for missing info instead of inventing availability, follow the booking workflow, and hold that context next turn▍ Evaluation DimensionsEach conversation is scored across:DimensionQuestionTask SuccessDid it accomplish the user's goalIntent AccuracyDid it understand the requestDialect FidelityConsistent Kuwaiti/Najdi, no driftNaturalnessNative-soundingContext RetentionRemembers prior turnsFactualityInformation correctGroundednessSupported by knowledge baseInstruction FollowingObeys system/business rulesCultural AppropriatenessRegionally correctSafetyHandles adversarial inputRobustnessSurvives typos/slang/ambiguityEscalationHands off correctly when required▍ Required• Native/near-native Arabic, strong Kuwaiti and/or Najdi dialect fluency• Experience testing chatbots, conversational systems, or LLM applications• Strong exploratory + adversarial test design• Understands LLM failure modes; judges by semantic correctness, not string match• Good written English for bug reports; Jira/Linear or equivalent▍ Strongly Preferred• LLM evaluation / LLM-as-a-Judge experience• RAG system testing• Prompt engineering understanding• Hands-on with OpenAI/Anthropic/Gemini or similar• API testing (Postman/cURL); basic Python for test automation• Experience building regression/eval datasets; Arabic NLP backgroundQualificationsDemonstrated expertise in Quality Assurance and Quality Control, with the ability to design and implement robust QA processes.Strong background in Quality Management, including developing standards, procedures, and continuous improvement initiatives.Experience with Good Manufacturing Practice (GMP) or comparable structured quality frameworks and compliance systems.Proficiency in Quality Auditing, including planning, executing, documenting, and following up on internal and external audits.Relevant bachelor's degree in business, engineering, quality management, or a related field; professional certifications in quality (e.g., CQE, CQA, Six Sigma) are an advantage.Solid analytical, problem-solving, and documentation skills, with attention to detail and accuracy.Ability to collaborate effectively with multidisciplinary teams and communicate clearly with stakeholders at all levels.Experience in professional services, advisory, or financial environments in Kuwait or the wider region is preferred.