Reddit scraping data AI startups
News

Reddit blocks free access for most crawlers

Here’s the scoop:

  • Reddit’s Playing Hardball: Reddit is blocking search engines and AI bots from accessing their latest content unless they pay a fee.
  • Google Gets Special Treatment: Currently, only Google can crawl new posts, likely due to a reported $60 million deal.
  • Human Answers Harder to Find: This move makes it harder to find discussions by real people on search engines besides Google.

The Deal with the Block

Reddit is tightening its grip on its data. Recently, they’ve updated their rules for web crawlers (programs that search websites) to restrict access to fresh content. This means search engines like Bing, DuckDuckGo, and others will only show older Reddit posts in their results.

This change isn’t totally out of the blue. Reddit wants new revenue streams and to appease investors. They’ve already hiked up fees for some developers who access their data and reportedly threatened Google with a cut-off if they didn’t pay to train their AI with Reddit content.

Why it Matters

With a growing number of AI chatbots generating questionable content online, finding genuine human discussion has become more valuable. Many people, like myself, rely on adding “Reddit” to searches to locate human-written responses. This new policy makes that method less effective, especially for folks who prefer search engines other than Google.

The Future of Reddit Search

It’s unclear what exactly the agreements with search engines like Google entail. Whether they involve just access to recent content or broader data sharing remains to be seen. One thing is clear: Reddit is setting a precedent for controlling how its content is searchable on the internet. This could potentially change the way we access information and discussions on the platform.

//Source//

Reddit blocks free access for most crawlers

Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

To Top