In recent news, Google has put forth a proposal known as the "Web Environment Integrity Explainer", authored by four of its engineers. On the surface, it
a bot that scrapes real estate sites & public transport sites, then finds you a home within your parameters that’s a short commute to work/school on public transport. (I did this)
a bot that scrapes travel sites and finds you the cheapest itinerary, which could even combine buses, trains, planes, and bla bla car. Most travel sites don’t even give a quote calendar to quickly show you which days the cheapest fare is available – and those that do often lack a way to specify criteria like time of day ranges. They pretty much all suck.
a bot that scrapes the site of your snail mail service so you never have to login manually… grabs envelopes, orders scans, grabs PDFs, etc.
a bot that scrapes dating sites & finds your ideal mate. A woman who rejected the narrow search criteria of a dating site did this and her own algorithm found her a date who eventually ended up becoming her husband.
a bot that scrapes 2nd-hand goods sites and finds what you’re looking for
a bot that scrapes your bank & logs in on a monthly basis to fetch your statements with no manual labor on your part.
a bot that manages your email forwarding service by resetting counts of allowed msgs for particular addresses.
a bot that scrapes bike sharing networks and uses openstreetmaps to work out which portion of your journey is flat or downhill so you don’t have to pedal a heavy rental bike uphill.
(edit) And generally, any website that has a lousy or useless search criteria needs the help of a beneficial bot. Any site that is manual human labor intensive needs a beneficial bot.
You probably benefit from bots without realizing it. Even if you don’t code or install a bot yourself, you might visit a site that runs with scraped content. I discovered that one of the real estate sites that I scraped was itself showing scraped content. Most likely at some point you’ve bought airfare from a consolidation site where the data was not only fed by the shared db but also showed flights from some of the more protectionist small airline sites.
I wish the general public were more aware of this. The masses unwittingly support the anti-bot measures without realizing the harvested data is often made available to everyone via another (improved) UI.
What beneficial bots are there?
Countless bots. E.g.
a bot that scrapes real estate sites & public transport sites, then finds you a home within your parameters that’s a short commute to work/school on public transport. (I did this)
a bot that scrapes travel sites and finds you the cheapest itinerary, which could even combine buses, trains, planes, and bla bla car. Most travel sites don’t even give a quote calendar to quickly show you which days the cheapest fare is available – and those that do often lack a way to specify criteria like time of day ranges. They pretty much all suck.
a bot that scrapes the site of your snail mail service so you never have to login manually… grabs envelopes, orders scans, grabs PDFs, etc.
a bot that scrapes dating sites & finds your ideal mate. A woman who rejected the narrow search criteria of a dating site did this and her own algorithm found her a date who eventually ended up becoming her husband.
a bot that scrapes 2nd-hand goods sites and finds what you’re looking for
a bot that scrapes your bank & logs in on a monthly basis to fetch your statements with no manual labor on your part.
a bot that manages your email forwarding service by resetting counts of allowed msgs for particular addresses.
a bot that scrapes bike sharing networks and uses openstreetmaps to work out which portion of your journey is flat or downhill so you don’t have to pedal a heavy rental bike uphill.
(edit) And generally, any website that has a lousy or useless search criteria needs the help of a beneficial bot. Any site that is manual human labor intensive needs a beneficial bot.
Ok yeh some good examples, thanks.
I’m not a fan of content scrapers like that, but I get why people like them.
You probably benefit from bots without realizing it. Even if you don’t code or install a bot yourself, you might visit a site that runs with scraped content. I discovered that one of the real estate sites that I scraped was itself showing scraped content. Most likely at some point you’ve bought airfare from a consolidation site where the data was not only fed by the shared db but also showed flights from some of the more protectionist small airline sites.
I wish the general public were more aware of this. The masses unwittingly support the anti-bot measures without realizing the harvested data is often made available to everyone via another (improved) UI.
Cloudflare works hard to demonize bots.