The context
The platform wanted to show each member posts close to their interests. To do that, every post had to be assigned to a topic. The community also had to be protected: hate speech filtered out, automated accounts detected.
What we built
- Topic classification. 52 topics, from finance to cooking, each defined by around thirty reference phrases. Every post is compared with these phrases by meaning, not by keywords.
- Moderation. A filter that detects hate speech.
- Bot detection. Behavioral signals to spot accounts that are not run by people.
- Recommendations. Collaborative filtering, which suggests content based on the preferences of similar members.
The hard part: classification accuracy
The first version assigned between 60 and 70% of posts correctly. That is not enough: a wrong topic shows up immediately in a member’s feed.
The gain did not come from a bigger model. It came from two things. First, the decision rule: instead of looking only at the average similarity with a topic, the score takes the strongest matches into account. Second, the topics themselves: the list and its reference phrases were reviewed by the product team, who flagged gaps and overlaps. Accuracy rose to between 90 and 95%.
The results
- Classification accuracy raised from about 65% to over 90%.
- Moderation, bot detection and recommendation components built into the product.
- A documented topic list that the product team can maintain on its own.
In classification problems, the gain often comes from how the categories and the decision rule are defined, before the choice of model.
What we take away
A topic list the product team can read and correct is worth as much as a good model. It is what makes it possible to improve the system without starting over.
Need to classify, filter or recommend content at scale? Let’s talk.
Book a callMore notes