Published Date: 21 December 2023
Author: Mohammad Galib Shams, Nabil Mosharraf Hossain
At Greentech Apps Foundation (GTAF), we deeply value user feedback. It helps us refine our Islamic apps and serve the Ummah better. But as our user base grew, so did the volume of feedback—especially on Google Play Store. With tens of thousands of reviews across multiple apps, we needed a way to analyse, categorize, and reply to them efficiently. That’s where our AI-powered auto-reply system comes in.
Objective: Automate Review Analysis & Response
Our goal was to perform sentiment analysis on user reviews to:
- Identify praises, feature requests, and issues.
- Detect reviews where users gave low ratings mistakenly.
- Generate automatic, context-aware replies.
We began with the Al Quran app as our first test case.
Motivation: Respond at Scale
Engaging with users matters. Replying to feedback:
- Shows users we care.
- Helps clarify confusion (e.g., mistaken ratings).
- Surfaces critical product issues faster.
But doing this manually for tens of thousands of reviews wasn’t feasible. We needed scalable AI-driven support.
TL;DR
- Built an AI system to classify and auto-reply to reviews.
- Used LLMs (Llama-2, Mistral-7B), FastText, and SetFit.
- Tackled dataset imbalance using synthetic data generation.
- Reached 99% accuracy using dual-model conflict filtering.
- Replied to over 61,000 reviews.
Results So Far
- Thanks for praise: 55,205
- Rating mistake detected: 702
- Feature request/issue found: 6,016
- Reviewers who updated rating: 29
- Reviewers who replied back: 43
- New issues/requests raised after reply: 25
Challenge: The Dataset Problem
Creating a usable dataset was the hardest part:
- Praise reviews vastly outnumbered complaints or suggestions.
- Star ratings weren’t reliable indicators of review sentiment.
Solution 1: Generative AI for Prompt-Based Classification
We used Llama-2 to classify reviews by prompting. But Llama-2 was heavy and expensive, so we moved to quantized versions.
Challenges:
- Lost accuracy with smaller models.
- JSON output formatting errors.
- Frequent misclassifications.
Mitigating with Prompt Engineering
We created separate prompts for detecting issues and feature requests, and introduced:
- Clear distinction between praise and issues.
- Format enforcement.
- Anti-hallucination constraints.
- Translation to focus on English.
Eventually, Mistral-7B (OpenOrca) gave us 95% classification accuracy.
Solution 2: Synthetic Dataset with FastText
We generated 3,000+ synthetic reviews with specific label combinations (e.g., praise + feature request). Over time, we grew the dataset to 7,000+ with better diversity.
FastText was chosen for its lightweight deployment. Accuracy reached 92%.
Solution 3: Pseudo-Labelling with LLMs
We used our LLM prompts to pseudo-label real-world reviews, detected conflicts with FastText predictions, and further improved the dataset.
Solution 4: SetFit + Conflict Filtering
SetFit, a transformer-based classifier, gave us 97% accuracy using only a portion of our dataset.
To push reliability to 99%, we deployed both FastText and SetFit, and only used predictions when both agreed.
Benefits:
- Reduced bias.
- Conflict data used to strengthen future training.
Deployment
We containerized the models using Docker and deployed them on our server. The pipeline:
- Pull latest reviews.
- Preprocess.
- Classify.
- Generate reply.
- Send to Play Store.
Remaining Work
- Strengthen dataset using conflict data.
- Extend to other apps.
- Push SetFit to 99% accuracy to replace FastText.
Conclusion
Our auto-reply system boosted user engagement, improved satisfaction, and allowed us to respond to 61K+ reviews in a thoughtful way. By combining AI, Islamic values, and user-centred design, we hope to continue improving the user experience and helping the Ummah better connect with authentic Islamic knowledge.
Have thoughts or want to collaborate on AI + Islamic tech? Reach out at https://gtaf.org.
Leave a Reply