AI Quality & Knowledge Analyst
Dealls (YC W22)
- Kontrak
- On-site • Jakarta Selatan
- Rp6.000.000 – 8.000.000
Tidak Perlu Sign Up!
Deskripsi Pekerjaan
This role combines two main functions:
- AI Quality Analyst: Review AI responses, define the expected answer, identify gaps or failure patterns, and recommend improvements to the knowledge base, instructions, or system behavior.
- Knowledge Base Editor: Write, organize, and update knowledge base content based on findings from the AI response review process.
This is not primarily a software engineering role. It is more suitable for someone who is analytical, detail-oriented, structured, and able to translate complex information into clear rules and documentation. Key Responsibilities:
A. AI Response Review
- Review AI-generated responses to user or client questions.
- Compare AI responses against the available source documents, policies, guidelines, or expected business logic.
- Determine whether each response is correct, partially correct, incomplete, misleading, unsupported, or incorrect.
- Define the expected answer for each reviewed case.
- Identify important information that the AI missed.
- Identify answers that appear correct but are not sufficiently supported by the available knowledge base.
- Detect inconsistencies across similar AI responses.
- Review both the final answer and the reasoning or evidence used to produce it, where available.
B. Root Cause Analysis
Investigate why the AI produced an incorrect or incomplete response.
C.Knowledge Base Improvement
- Write and update knowledge base articles, rules, examples, and decision guidelines.
- Convert informal, incomplete, or ambiguous information into clear and explicit documentation.
- Add missing definitions, conditions, exceptions, examples, and edge cases.
- Ensure that each knowledge base entry can be understood consistently by both humans and AI systems.
- Structure information so that the AI can retrieve and apply the correct rule.
- Remove duplicated, outdated, or conflicting information.
- Clarify which rule should take priority when multiple rules apply.
- Create question-and-answer examples for situations that may be interpreted in different ways.
- Maintain version history and document the reason behind each knowledge base update.
D. Quality Assurance and Testing
- Retest previously incorrect cases after the knowledge base has been updated.
- Identify high-risk questions that require stricter review or human approval.
- Escalate recurring system issues to the relevant product or engineering team.
Kualifikasi
Core Qualifications
- Strong analytical and critical-thinking skills.
- High attention to detail.
- Strong written communication skills in English and Indonesian.
- Ability to identify factual errors, logical gaps, inconsistencies, and ambiguity.
- Ability to convert complex information into clear, structured, and explicit documentation.
- Ability to understand rules, conditions, exceptions, and decision logic.
- Comfortable reviewing large amounts of text and repetitive cases.
- Able to make consistent judgments based on documented standards.
- Comfortable working with spreadsheets, documentation tools, dashboards, and internal software.
- Able to work independently while knowing when to ask for clarification.
- Able to explain why an answer is correct or incorrect, rather than relying only on intuition.
Preferred Backgrounds
Candidates may come from one or more of the following backgrounds:
- Business analyst.
- Operations analyst.
- Quality assurance analyst.
- Customer support quality analyst.
- Knowledge management specialist.
- Knowledge base writer or editor.
- Content quality analyst.
- Policy analyst.
- Compliance analyst.
- Research analyst.
- Process improvement analyst.
- Medical claims analyst.
- Insurance operations analyst.
- Legal researcher.
- Regulatory researcher.
- Auditor.
- Risk analyst.
- Fraud review analyst.
- Case review specialist.
A software engineering background is not required.
Preferred Experience
- Experience reviewing the quality of customer service, operational, policy, legal, healthcare, insurance, or financial decisions.
- Experience creating standard operating procedures, policies, decision trees, or knowledge base articles.
- Experience identifying gaps in documentation or operational processes.
- Experience creating test cases or quality evaluation frameworks.
- Experience handling information that contains multiple rules, exceptions, and approval conditions.
- Basic understanding of how AI assistants and large language models work.
- Experience with healthcare, insurance, claims, finance, legal, compliance, or other document-heavy industries.
- Experience reviewing customer conversations or support tickets.
What This Role Is Not
This role is not primarily responsible for:
- Developing machine learning models.
- Building backend systems.
- Writing production software.
- Managing cloud infrastructure.
- Training foundation models.
- Designing complex AI architecture.
- Fixing technical retrieval, database, or integration problems directly.
Technical issues will be escalated to software engineers or AI engineers. This role is responsible for identifying, documenting, and explaining the issue clearly.
Candidate Profile
The ideal candidate is someone who:
- Enjoys investigating why something is wrong.
- Does not accept an answer simply because it sounds convincing.
- Can separate facts from assumptions.
- Can explain complex rules in simple language.
- Naturally thinks about exceptions and edge cases.
- Is comfortable making structured judgments.
- Is patient with repetitive review work.
- Can maintain consistency across many cases.
- Takes documentation quality seriously.
- Understands that small wording differences can change the meaning of a rule.
Tips Menjaga Diri
Perusahaan dan Lowongan di Dealls tidak meminta data pribadi, informasi rekening, atau pungutan ketika melamar. Hindari juga lowongan Google Form / Grup Telegram tanpa keabsahan yang jelas.
Dealls is Indonesia's largest job portal & mentoring platform. We help people easily find jobs to top Indonesian companies for internship and full-time roles.
As you might have already heard about us, we are revolutionizing how Indonesian engage with employers. Many Indonesian students are talented, ambitious, but never found a better opportunity for themselves. Our team is driven by the vision to democratize information and knowledge, thus opportunities. We think what we build will allow Indonesian students to have a chance to self develop and have a better future.
Our founder is a Harvard graduate, and has gathered a strong team with burning passion to help Indonesian discover opportunities they deserve! 🚀 In 1 year, we amassed 2mio+ users, got funded by YCombinator, and awarded the Best App of 2021 by GooglePlay, beating 3 unicorns and tens of others.
We are looking for passionate and committed individuals who look forward to working in a fast-paced entrepreneurial environment. Join us to have the opportunity to make decisions to drive a huge difference and impact for Indonesia youths! Here in Dealls, you’ll work closely with our board members. We are preferably looking for applicants who are proficient in both Bahasa and English.
