# How do you filter harmful content with OpenAI's moderation endpoint?

Date: 2022-08-21 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/filter-harmful-content-openais-moderation-endpoint/

**Answer:** You send each piece of text to OpenAI's Moderation endpoint, a free classifier released on August 10, and it tells you whether the text is flagged for sexual, hateful, violent or self-harm content. Your app then decides what to do: pass it on to GPT-3, hold it for a person to review, or block it.

If your app lets the public type text that GPT-3 then reads or answers, you need a way to catch the worst of it before it causes harm. On August 10, OpenAI released a tool for exactly that: the **Moderation endpoint**. An endpoint is simply one address in the API that does one job. This one takes a piece of text and returns a verdict on whether it breaks OpenAI's content policy. It replaces the company's earlier content filter, and OpenAI describes it as faster and more accurate.

Under the hood it is a set of classifiers, models trained to sort text into categories, built on GPT models. OpenAI says developers can use it free of charge to moderate content generated with its API, so you don't have to build and maintain your own filters. Screening other traffic, such as a forum that never calls GPT-3, is in a private beta that will carry a fee.

## What it checks for

The categories come from a paper OpenAI researchers posted on August 5, which describes how the classifier was built. Its main categories are:

- **Sexual** content, with a separate, stricter category for sexual content involving minors.
- **Hate**: abuse aimed at people because of who they are, with a severe subcategory for hate that calls for violence.
- **Violence**, with a severe subcategory for extremely graphic violence.
- **Self-harm**: graphic descriptions or encouragement of self-harm.

The paper also defines harassment, but notes that this category is still being improved and is not yet offered by the endpoint.

## A worked example: screening comments on a recipe site

Picture a community recipe site. Visitors leave comments, and the site uses GPT-3 to write a short, friendly reply to each one. You don't want an abusive comment to reach GPT-3, and you don't want GPT-3's reply to go out unchecked either. The flow has four steps:

1. A visitor submits a comment.
2. Your server sends the comment to the Moderation endpoint.
3. If it comes back flagged, the comment is held for a person; if not, it goes to GPT-3 for a reply.
4. The reply also goes through the endpoint before it is published.

With OpenAI's Python library, step 2 is a single call:

```python
import openai

openai.api_key = "YOUR_API_KEY"  # keep this on your server, never in the page

comment = "Made this last night. Way too salty for me, I'd halve the soy sauce."
result = openai.Moderation.create(input=comment)
```

The library also accepts a list of strings, so you can check several comments in one request. You can name one of two models, `text-moderation-stable` or `text-moderation-latest`; leave the model out and the endpoint uses its default. The response says whether the text is flagged overall and gives a verdict for each category, so your code can treat a flag for violence differently from one for hate.

Here is how the site might handle four comments. The verdicts are illustrative, not real API output:

| Comment (paraphrased) | Flagged? | Action |
|---|---|---|
| Too salty, halve the soy sauce | No | Send to GPT-3 for a reply |
| A threat against another commenter | Yes (violence) | Hold for review, do not reply |
| "This cake is killer, I could die happy" | Possibly | Hold for review |
| An insult aimed at a religious group | Yes (hate) | Block and log |

The third row is the interesting one. Words like "killer" and "die" can trip a classifier even when the meaning is harmless, which is why a flag should usually mean "a person looks at this", not "delete automatically".

> **Try it:** Write ten sample comments for a site you know, including two harmless ones that use violent-sounding slang. Run them through the endpoint and sort the results into "clearly fine", "clearly not" and "needs a person". The third pile is where your review rules should focus.

## Where should a human still review?

Context is the first reason. The OpenAI paper gives the example of a sentence like "I will kill you": the taxonomy labels it violent, yet in a piece of fiction it may be perfectly acceptable. A classifier sees words, not the situation around them. If your site hosts stories, role-play or discussions of history and news, expect flags that a person needs to judge.

Bias is the second. The authors report that their model can give higher hate scores to text that mentions gay people, and higher sexual scores to text containing "her", because the training data came from the internet. They also warn that such tools can produce disproportionate false positives when people discuss groups that are often targeted by hate. In practice, that means members of a community talking about their own experience may get flagged more than anyone else. Human review of flagged posts, and an easy way to appeal, protects those users.

Language and rare cases are the third. Only about 5% of the training samples were in languages other than English, and the authors say performance on other languages has not been rigorously evaluated. If your audience writes in French or Spanish, check a sample of results by hand. The same goes for the gaps in coverage: harassment is not yet a category, so bullying that avoids the listed categories may pass through.

A sensible pattern is to let the endpoint handle the clear cases automatically, route every flag and every borderline case to a person, and spot-check a small random sample of unflagged posts each week. The classifier makes moderation faster; deciding what your community accepts stays with you.

## Sources

- New-and-Improved Content Moderation Tooling: https://openai.com/blog/new-and-improved-content-moderation-tooling/
- A Holistic Approach to Undesired Content Detection in the Real World: https://arxiv.org/abs/2208.03274v1
- openai-python v0.22.1: openai/api_resources/moderation.py: https://github.com/openai/openai-python/blob/v0.22.1/openai/api_resources/moderation.py