Skip to content

About

Identify potential Wikipedia AI cleanup tasks

Resources

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

WikiTomte-LLM

This tool is an experiment in auto-identifying potential undetected LLM-generated text in Wikipedia articles. It produces prioritized lists of suspicious articles for editors to review. The goal is to help editors who want to identify and remove LLM content, especially the dedicated volunteers of WikiProject AI Cleanup, by speeding up their work and making it more fun.

What you can do with this tool:

  1. Search to create a candidate article list. Run a command to create a list of articles with potential LLM-generated text by searching all articles for random combinations of AI vocabulary. You get a CSV of search results and a plain-text list of article names. (You can import CSVs into Google Sheets or another spreadsheet application to sort, filter, and make decisions.)
  2. Scan candidate articles to find suspicious passages. Run another command to scan each article on the list for significant LLM vocabulary overall and particularly suspicious passages. You get a CSV with all of the data for review. Make sure you're familiar with WikiProject AI Cleanup/Guide before taking action on articles.

You can skip directly to scanning if you want to provide your own list of articles to scan, such as a list you made using Petscan.

The process is based on User:Gnomingstuff's guide to finding AI-generated text. It uses the vocabulary lists in Wikipedia:Signs of AI writing, supplemented with additional vocabulary lists.

This method tends to find articles with promotional content in general, including some articles with LLM-generated text and other articles that were written by people with conflicts of interest.

This tool was made by User:Dreamyshade, with the help of Wikipedia-AI-Skills by User:Fuzheado, using Cursor. Check out WikiProject AI Tools if you'd like to learn about using LLMs for making tools. It is called WikiTomte-LLM because a tomte in Nordic folklore is a small person-like creature who lives in your house, a bit like a gnome, who is mostly helpful but not always.

Web app:

Screenshot of web app

Search - CSV output:

Screenshot of triage

Scan - CSV output:

Screenshot of output

Hosted web app

GitHub Codespaces

A fast way to try WikiTomte-LLM in a browser:

  1. Click Open in GitHub Codespaces.
  2. Add a Codespaces secret named WIKITOMTE_USER_AGENT with your contact info, for example:
 WikiTomte-LLM/1.0 (User:YourUsername, you@example.com) WikiTomte-LLM/1.0
  1. Rebuild or recreate the codespace if you add the secret after creation.
  2. The Streamlit app opens on port 8501 — search for candidates, review results, then scan.

The web app runs random search only. Era-builder, freeform queries, and other CLI options are available below.

Local setup

Download and install the code:

git clone https://github.com/brittag/WikiTomte-LLM.git

cd WikiTomte-LLM

pip install -r requirements.txt

Set up a unique User-Agent, per Wikimedia's User-Agent policy. Copy config.example.json to config.json and replace the placeholder with your email address or Wikipedia username. In Codespaces, use the WIKITOMTE_USER_AGENT secret instead.

cp config.example.json config.json
# Edit config.json to add your contact info

Local web app

To run the web app:

streamlit run app.py

Command-line interface

Start

Step 1: Search. Find candidate articles by searching Wikipedia for random combinations of AI vocabulary words, and review the list of search results with excerpts. Returns a maximum of 100 results by default.

python3 assets/search_triage.py \
  -o triage.csv --write-articles candidates.txt

Review triage.csv. Edit candidates.txt to remove obvious false positives and any other articles you don't need to scan in depth, such as articles already flagged for AI cleanup. See docs/search-triage.md for column meanings.

Step 2: Scan. Scan remaining candidates for suspicious passages across AI vocabulary lists from multiple eras (GPT-4, GPT-4o, GPT-5, generic AI vocabulary list).

python3 assets/ai_detector.py candidates.txt -o report.csv

See docs/csv-output.md for scan report column meanings.

Scan a list of articles (skip search)

If you already have a list of articles that you want to scan:

python3 assets/ai_detector.py articles.txt -o report.csv

See examples/articles.txt for the input format (one title per line; # for comments).

Advanced options

Custom search queries

By default, the search picks a random era, then either three words or a phrase plus two words from the AI vocabulary. If you want to search using your own CirrusSearch strings, use --query with --era:

python3 assets/search_triage.py \
  --query '"crucial role" emphasize underscore' --era gpt4o \
  -o triage.csv --write-articles candidates.txt

More query modes and examples: docs/search-triage.md.

Scan options

Restrict to one era band (gpt4, gpt4o, gpt5, or generic):

python3 assets/ai_detector.py articles.txt --era gpt4o -o report.csv

JSON output and other flags (--min-score, --delay): docs/csv-output.md.

About

Identify potential Wikipedia AI cleanup tasks

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages