In recent news, Google has put forth a proposal known as the "Web Environment Integrity Explainer", authored by four of its engineers. On the surface, it
You probably benefit from bots without realizing it. Even if you don’t code or install a bot yourself, you might visit a site that runs with scraped content. I discovered that one of the real estate sites that I scraped was itself showing scraped content. Most likely at some point you’ve bought airfare from a consolidation site where the data was not only fed by the shared db but also showed flights from some of the more protectionist small airline sites.
I wish the general public were more aware of this. The masses unwittingly support the anti-bot measures without realizing the harvested data is often made available to everyone via another (improved) UI.
You probably benefit from bots without realizing it. Even if you don’t code or install a bot yourself, you might visit a site that runs with scraped content. I discovered that one of the real estate sites that I scraped was itself showing scraped content. Most likely at some point you’ve bought airfare from a consolidation site where the data was not only fed by the shared db but also showed flights from some of the more protectionist small airline sites.
I wish the general public were more aware of this. The masses unwittingly support the anti-bot measures without realizing the harvested data is often made available to everyone via another (improved) UI.
Cloudflare works hard to demonize bots.