26% OF THE TOP 100 SITES NOW PROHIBIT GPTBOT FROM ACCESSING THEIR CONTENT

A recent trend has attracted the attention of many Internet users. This is because 26% of the most popular websites have chosen to block GPTBot, the web crawler developed by OpenAI. This decision has raised questions about the hidden reasons and implications of this access restriction.

In this article, we will take a close look at this growing trend and why some of the most popular websites have opted to block GPTBot. We will also see the potential consequences of this decision on access to online content and the reasons which could motivate these sites to take such measures.

Many websites started by blocking access to their content from GPTBot

In order to ensure the quality of its main service, which is ChatGPT, OpenAI must collect a large amount of data from the net. The company normally performs this task through its website crawler, GPTBot.

However, since the latter's launch on August 7, 2023, more and more sites have started to block its access to their pages.

For example, among the 100 most visited websites in the world, 26 have chosen to restrict access to GPTBot, preventing OpenAI from collecting the information it needs. Keyword Research: trademark registration in usa starts with understanding the keywords your target audience is using to search for your products or services.

Source: Originality.ai

This limitation represents a major challenge for the company, as it depends on this data to improve and enrich the ChatGPT experience.

Expanding the analysis to the top 1,000 websites, the situation became even more concerning, as 242 of them took the decision to completely ban access to GPTBot.

List of Top Websites That Blocked GPTBot

Among the top 100 websites that have refused GPTBot access, we find major brands such as:

quora.com;
pinterest.com;
Amazon.com;
cbsnews.com;
indeed.com;
theguardian.com;
sciencedirect.com;
usatoday.com;
stackexchange.com;
alamy.com;
webmd.com;
dictionary.com;
Washingtonpost.com…

You can discover the full list in this Google Sheet file.

The reasons behind the blocking of GPTBot?

The possible explanation behind the blocking of GPTBot lies in the desire of the affected websites to preserve their data and not make it available to OpenAI for training its models, without obtaining compensation in return.

This rejection is also motivated by the fact that ChatGPT does not provide any sources for the results it presents.

Unlike AI bots, traditional search engines like Google are allowed to crawl content because they offer a huge advantage: they send traffic via direct links/quotes.

It is clear that OpenAI faces significant challenges in maintaining its access to data on the Internet. This highlights the importance of a thoughtful, privacy-respecting approach when collecting this critical information for the development of its technologies.

How was this study carried out?

On August 22, 2023, a study identified the 1000 most popular websites in the world. It was subsequently updated on August 29 and then September 22, 2023.

As part of this analysis, each robots.txt file on these sites was carefully inspected to determine whether they were blocking access to GPTBot or other web crawlers.

However, it is important to note that some websites had unidentifiable robots.txt files or could not be inspected for various reasons. These sites were therefore excluded from this study.

When the participants in this study notice that access to GPTBot is blocked by a book marketing agency it uses the Archive.org tool to trace the precise moment when this site had started restricting access to crawlers.

However, a number of websites had also blocked access to Archive.org, making it impossible to verify the exact date they began blocking GPTBot.

Does blocking GPTBot have any disadvantages?

This practice may have downsides when ChatGPT starts providing users with direct links or references from web sources.

It will thus be able to play a role comparable to that of a search engine by generating a significant flow of visitors to specific websites.

So, if a website blocks GPTBot, this content may not be recommended, which could cause it to lose potential visitors.

In other words, just as blocking Googlebot would prevent a website from appearing in the Google search engines, blocking GPTBot could mean missing out on a booming web traffic channel.

It is therefore essential for website owners to carefully consider the implications of blocking GPTBot, as it could impact their visibility and access to a wider audience.

As technology evolves, it's important to strike a balance between protecting online resources and exploiting the opportunities offered by tools like GPTBot.

What is meant by Web Crawling or Scraping and is it legal?

Web mining or scraping defines an automated process in which software visits websites to collect specific data from their pages.

This technique is frequently used by search engines such as Google to index and organize web pages. This makes it easier for users to find information.

However, the automatic collection of data from websites can raise questions regarding security and privacy. Some websites contain sensitive or private information, and collecting this data without permission can lead to legal problems.

In summary

To conclude, the increasing blocking of GPTBot by popular websites mainly stems from privacy concerns. In addition to this, there is also a question of interest, because ChatGPT does not often mention the sources of its information.

Click Here To Submit Your Resume

Submit Here

More News you may like