How moderation works
The four moderation surfaces, what each one detects, the result shapes, and how to tune thresholds.
The four surfaces
One license key unlocks four ways to moderate:
| Surface | What it does | Method |
|---|---|---|
| Image | Moderate a single photo | analyzeImage |
| Video | Sample and aggregate a recorded clip | analyzeVideo |
| Text | Moderate a chat message or comment | moderateText |
| Live camera | Real-time per-frame moderation | startStream and frames |
What it detects
Objects. Each visual check returns the best confidence per class for weapon, drug, cigarette and alcohol. A class is flagged when its confidence meets that class’s threshold.
Adult / NSFW. A whole-image verdict:
safe: clean.warn: suggestive or borderline. Advisory only, and it does not flag the content by itself.block: explicit. This makes the content unsafe.
Text. A message is classified as safe, profanity, toxic, hate, harassment or threat. Text runs in two layers: a fast keyword blocklist first, then an AI classifier for anything the blocklist misses. The result tells you which layer decided.
Results
A visual check (image or live frame) returns a result with:
| Field | Type | Description |
|---|---|---|
isUnsafe | bool | True if any object is flagged, or NSFW is block |
detections | list | Flagged objects, each { category, confidence } |
nsfw | enum | safe, warn or block |
nsfwScores | object | safe, sexy, explicit (0 to 1) |
rawScores | object | Best object score per class, for tuning |
A text check returns { blocked, layer, category, score, matchedWord }.
A video check returns { isUnsafe, categories, flags, framesAnalyzed, nsfw }, where flags lists the timestamps and categories that were flagged.
Thresholds
Every object class and both NSFW bars are adjustable at runtime. Lower is stricter. The defaults are tuned to balance catching unsafe content against false positives.
setThreshold(weapon, 0.70) // object class bar
setNsfwThreshold(explicit, 0.45) // NSFW block bar
setNsfwThreshold(sexy, 0.55) // NSFW warn bar
TIP
Start with the defaults. Tighten a threshold only if you see misses, and loosen one only if you see false positives for your content.
Live performance
On the live camera, the lighter NSFW check runs on every frame while the heavier object detector runs on a cadence you choose (favor responsiveness or battery). This keeps the camera preview smooth across a wide range of devices. See the platform guides for the controls.
