# AI crawlers and data-collection bots, gathered from the Krogza list
# and NOT banned by default: an outright ban on an AI assistant's
# crawler is unjust to the sites hosted here — it removes them from AI
# search results — and the polite ones honour robots.txt anyway.
#
# Kept as data for operators who DO want them banned (add these names
# back into bad_user_agents.regex.txt and regenerate), and for the
# rate-limiting rules we may build when one of them turns virulent.
# What that will need is a mechanic the evaluator does not have yet:
# counting occurrences PER USER-AGENT rather than per address.
#
# One name per line, as the upstream list writes them.
Ai2Bot
Bytespider
CCBot
ChatGPT-User
ClaudeBot
FacebookBot
GPTBot
ImagesiftBot
Timpibot
allenai.org
anthropic-ai
cohere-ai
imagesift.com
omgili
openai
openai.com