The world’s top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.
OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.
TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.
OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.
However, according to TollBit’s findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to “bypass” robots.txt in order to retrieve or scrape all of the content from a given website or page.
A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.” A spokesperson for Anthropic did not respond to emails seeking comment.
Robots.txt is a single bit of code that’s been used since the late 1990s as a way for websites to tell bot crawlers they don’t want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.
If the models are random then we shouldn’t be trusting them to do anything, let alone serious applications. If any other type of software told us that it’s based on partially random results we’d say “get that shit out of here, I want my software to work first time, every time”.
“Statistically good enough” works for some applications but not for others. If a LLM finds a formula that has an 80% chance to be the cure for cancer or a new magical fuel or some amazing new material that’s cool, we’re not going to look the gift horse in the mouth.
But using LLM to polute the web with advertising texts that are barely inteligible, and using it as a pretext to break copyright in the process, who does that help? So far the only readily available commercial application for LLMs has been to spit out semi-nonsense so that a bunch of bottom-crawling parasitic industries can be enabled to keep on pinching pennies and shitting up everything they touch.
Which, ironically, it will help them to hit bottom all the faster, so in a strange way it’s a positive return, but the problem is they’re going to take down a lot of useful things with them.