Major publishers – including Reddit, Yahoo, Medium, and Quora – today announced support of a new licensing standard, Really Simple Licensing (RSL). In theory, RSL gives websites leverage to demand compensation when AI companies scrape and train on their data.

Why we care. Until now, publishers could only block or allow bots via robots.txt. RSL upgrades that model, letting publishers embed licensing and royalty terms directly into robots.txt files, online books, videos, and datasets.

How it works. Publishers can set subscription, pay-per-crawl, or pay-per-inference fees (compensation when an AI uses their content in responses).

  • Bots for non-training purposes (like search indexing or archiving) can crawl as usual.
  • RSL relies on AI companies’ cooperation. Without buy-in, enforcement will be a challenge.

The big picture. The RSL Collective, led by RSS co-creator Eckart Walther and former Ask.com CEO Doug Leeds, is pushing the standard.

  • They see it as a scalable business model for the web, modeled after rights groups like ASCAP in music.
  • Fastly is working with RSL to build “gatekeeper” tech that can admit or block bots based on licensing compliance.

Supporters. Brands that are supporting RSL include:

  • Reddit
  • Quora
  • Yahoo
  • Medium
  • Adweek
  • Fastly
  • Internet Brands
  • The MIT Press
  • O’Reilly
  • People Inc.
  • wikiHow
  • Ziff Davis

Between the lines. Some big media players (e.g., The New York Times, News Corp, Vox Media) have cut direct licensing deals with AI firms. But RSL aims to simplify the process for any publisher, big or small.

The catch. AI model builders have a history of ignoring robots.txt. Without enforcement teeth, RSL’s success hinges on whether major AI players adopt the standard.

Not the first. Cloudflare, which powers about 20% of the internet, is also cracking down on AI bots. It blocks AI crawlers by default and is testing a Pay Per Crawl system that charges AI companies for access – a parallel effort to RSL that could reshape how AI firms get training data.

Really Simple Licensing. You can dig deeper on the RSL website.

Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.

Danny GoodwinDanny Goodwin

Danny Goodwin is Editorial Director of Search Engine Land & Search Marketing Expo – SMX. He joined Search Engine Land in 2022 as Senior Editor. In addition to reporting on the latest search marketing news, he manages Search Engine Land’s SME (Subject Matter Expert) program. He also helps program U.S. SMX events. Goodwin has been editing and writing about the latest developments and trends in search and digital marketing since 2007. He previously was Executive Editor of Search Engine Journal (from 2017 to 2022), managing editor of Momentology (from 2014-2016) and editor of Search Engine Watch (from 2007 to 2014). He has spoken at many major search conferences and virtual events, and has been sourced for his expertise by a wide range of publications and podcasts.