I'm not sure of the legality but I definitely appreciate their product. This lawsuit seems odd because google themselves scrape content for their indexes. From what I see SerpApi is really just providing a machine interface that Google themselves refuses to provide users and visibility into SERPs which is also something that users should have available to them.
Google publishes how to control their bot - with robots.txt. They then obey those instructions. Google also takes some effort to not use all your bandwidth. Google isn't perfect, but they are at least making a "good faith" effort to be nice and this does count in court. Overall most will agree that in general what google does to allow people to find their website is worth the things that google is doing.
You can of course argue a lot of edge cases if you really want. For the most part I want to say "it isn't worth the argument". In some cases I will take your side if I really have to think about it, but in general the system google has been using mostly works and is mostly an acceptable compromise.
But their robots are enabled by default. So it is a form of unsolicited scraping. If I spam millions of email addresses without asking for permission but provide a link to opt-out form, am I the good guy?
This is why I stopped using google wherever possible - they pushed the frontier of useful fair use and copyright precedents and established that things on the public internet displayed to the public without a login mechanism are fair game for scraping. The US supreme court ruled that you have to incorporate authentication and not simply serve your content to the public internet if you want to restrict usage.
Then they bend over backwards and do the "but not like that!" crap with their legal team and swing their wealth and influence around to screw over other companies and people, and a vast majority of it just vanishes, gets memory holed, with NDAs and out of court settlements, so you never get to see the full scope of harm they inflict unless you're watching like a hawk and catch the headlines before they get disappeared.
Google needs to be broken up and we need to legislate the dismantling of the current adtech regime, with a privacy and sovereignty respecting digital bill of rights that puts the interests of individual citizens above that of giant corporate blobs and the mass surveillance data industry.
> SerpApi’s answer to SearchGuard is to mask the hundreds of millions of automated
queries it is sending to Google each day to make them appear as if they are coming from human
users. SerpApi’s founder recently described the process as “creating fake browsers using a
multitude of IP addresses that Google sees as normal users.”
> Defendant SerpApi, LLC (“SerpApi”) offers services that “scrape” this copyrighted
content and more from Google, using deceptive means to automatically access and take it for free at an astonishing scale and then offering it to various customers for a fee. In doing so, SerpApi acquires for itself the valuable product of Google’s labors and investment in the content, and denies Google’s partners compensation for their works
this has to be satire. Is Google not the #1 entity guilty of exactly this?
Google doesn't have to do that now after already having established its own monopoly... just like SerpApi wouldn't have to act deceptively if they had a monopoly on search.
Because they've forced everyone to allow them. They're the internet traffic mafia. Block them and you disappear from the internet
They abuse this power to scrape your work, summarize it and cut you out as much as possible. Pure value extraction of others' work without equal return. Now intensified with AI
nobody is forcing anyone. This is the same argument that people said about google search. Nobody is forcing anyone to use google search, google chrome, or even allow googlebot for scraping.
Thousands of poeple have switched over to chatgpt, brave/firefox ..
Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"
Why would Google offer an API? This is similar to saying when Apple sues an employee stealing IP "Nobody would steal the IP if they gave it away for free". The question is - why?
What’s the difference between scraping and malicious scraping? Does google engage in scraping or malicious scraping? Do the AI companies engage in scraping or malicious scraping?
Note that I am not defending the merits of Google's lawsuit, but they did describe in this very post what they believe distinguishes their scraping versus SerpApi.
> Stealthy scrapers like SerpApi override those directives and give sites no choice at all. SerpApi uses shady back doors — like cloaking themselves, bombarding websites with massive networks of bots and giving their crawlers fake and constantly changing names — circumventing our security measures to take websites’ content wholesale. [...] SerpApi deceptively takes content that Google licenses from others (like images that appear in Knowledge Panels, real-time data in Search features and much more), and then resells it for a fee. In doing so, it willfully disregards the rights and directives of websites and providers whose content appears in Search.
To me this seems... interesting, for sure. I think that Google already set a bad precedent by pulling content from the web directly into its results, and an even worse one by paying websites with user-generated content for said content (while those sites didn't pay the users that actually made the user-generated content, as an additional bitchslap.)
But it seems like at the very least Google is suggesting that SerpApi is effectively trying to "steal" the work Google did, rather than do the same work themselves. Though I wonder if this is really Google pulling up the ladder behind them a bit, given how privileged of a position they are in with regards to web scraping.
It's a tough case. I think that something does need to ultimately be done about "malicious" web scraping that ignores robots.txt, but traditionally that sort of thing did not violate any laws, and I feel somewhat skeptical that it will be found to violate the law today. I mean, didn't LinkedIn try this same thing?
> SerpApi deceptively takes content that Google licenses from others
They have a different definition of "licensing" than most people I guess. Aren't site operators complaining about Google using this "licensed" content in AI overviews... not to mention the scraping for AI model training.
As far as I know, Google respects robots.txt and doesn't obfuscate their crawlers, so you can easily block them if you want. It seems like an important distinction?
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway.
SerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.
There's no law that says you have to do that. It used to be a sensible thing to do, in the early internet. In the current internet, obeying robots.txt is a self-handicap and you shouldn't do it.
Google scrapes so what even is this? Beyond that I think it is unreasonable and monopolistic that Google can use all this data (like YouTube) to bolster their AI products but no one else can. It just means the megacorp will keep being megacorp and smaller players are doomed to have to work much harder and get very lucky. It’s not fair competition. So I view scraping Google as necessary for our society.
I'm not sure of the legality but I definitely appreciate their product. This lawsuit seems odd because google themselves scrape content for their indexes. From what I see SerpApi is really just providing a machine interface that Google themselves refuses to provide users and visibility into SERPs which is also something that users should have available to them.
I'm probably just being naive though...
Google publishes how to control their bot - with robots.txt. They then obey those instructions. Google also takes some effort to not use all your bandwidth. Google isn't perfect, but they are at least making a "good faith" effort to be nice and this does count in court. Overall most will agree that in general what google does to allow people to find their website is worth the things that google is doing.
You can of course argue a lot of edge cases if you really want. For the most part I want to say "it isn't worth the argument". In some cases I will take your side if I really have to think about it, but in general the system google has been using mostly works and is mostly an acceptable compromise.
What's nice about scraping all the content for their own good while killing off websites left and right? Google needs to be sued also.
Along with all the other AI companies out there, the've committed the biggest theft in human history.
But their robots are enabled by default. So it is a form of unsolicited scraping. If I spam millions of email addresses without asking for permission but provide a link to opt-out form, am I the good guy?
This is why I stopped using google wherever possible - they pushed the frontier of useful fair use and copyright precedents and established that things on the public internet displayed to the public without a login mechanism are fair game for scraping. The US supreme court ruled that you have to incorporate authentication and not simply serve your content to the public internet if you want to restrict usage.
Then they bend over backwards and do the "but not like that!" crap with their legal team and swing their wealth and influence around to screw over other companies and people, and a vast majority of it just vanishes, gets memory holed, with NDAs and out of court settlements, so you never get to see the full scope of harm they inflict unless you're watching like a hawk and catch the headlines before they get disappeared.
Google needs to be broken up and we need to legislate the dismantling of the current adtech regime, with a privacy and sovereignty respecting digital bill of rights that puts the interests of individual citizens above that of giant corporate blobs and the mass surveillance data industry.
From the filing
> SerpApi’s answer to SearchGuard is to mask the hundreds of millions of automated queries it is sending to Google each day to make them appear as if they are coming from human users. SerpApi’s founder recently described the process as “creating fake browsers using a multitude of IP addresses that Google sees as normal users.”
> Defendant SerpApi, LLC (“SerpApi”) offers services that “scrape” this copyrighted content and more from Google, using deceptive means to automatically access and take it for free at an astonishing scale and then offering it to various customers for a fee. In doing so, SerpApi acquires for itself the valuable product of Google’s labors and investment in the content, and denies Google’s partners compensation for their works
this has to be satire. Is Google not the #1 entity guilty of exactly this?
No, Google doesn't use deceptive means. They identify their crawler as GoogleBot, and obey robots.txt.
Google doesn't have to do that now after already having established its own monopoly... just like SerpApi wouldn't have to act deceptively if they had a monopoly on search.
Because they've forced everyone to allow them. They're the internet traffic mafia. Block them and you disappear from the internet
They abuse this power to scrape your work, summarize it and cut you out as much as possible. Pure value extraction of others' work without equal return. Now intensified with AI
But yeah, you're right. They're not deceptive
> Because they've forced everyone to allow them.
nobody is forcing anyone. This is the same argument that people said about google search. Nobody is forcing anyone to use google search, google chrome, or even allow googlebot for scraping.
Thousands of poeple have switched over to chatgpt, brave/firefox ..
Your argument sounds like "I dont like Apple's practices, and I'm forced to buy iPhones. No buddy, if you dont like Apple, dont buy their products"
What about for their LLM products? We know that OpenAi does not respect the robots.txt file
Google uses the same crawler and robots.txt file for training data.
SerpApi wouldn't even be a thing if Google offered an equivalent API...
why does google need to offer it?
Why would Google offer an API? This is similar to saying when Apple sues an employee stealing IP "Nobody would steal the IP if they gave it away for free". The question is - why?
What’s the difference between scraping and malicious scraping? Does google engage in scraping or malicious scraping? Do the AI companies engage in scraping or malicious scraping?
Note that I am not defending the merits of Google's lawsuit, but they did describe in this very post what they believe distinguishes their scraping versus SerpApi.
> Stealthy scrapers like SerpApi override those directives and give sites no choice at all. SerpApi uses shady back doors — like cloaking themselves, bombarding websites with massive networks of bots and giving their crawlers fake and constantly changing names — circumventing our security measures to take websites’ content wholesale. [...] SerpApi deceptively takes content that Google licenses from others (like images that appear in Knowledge Panels, real-time data in Search features and much more), and then resells it for a fee. In doing so, it willfully disregards the rights and directives of websites and providers whose content appears in Search.
To me this seems... interesting, for sure. I think that Google already set a bad precedent by pulling content from the web directly into its results, and an even worse one by paying websites with user-generated content for said content (while those sites didn't pay the users that actually made the user-generated content, as an additional bitchslap.)
But it seems like at the very least Google is suggesting that SerpApi is effectively trying to "steal" the work Google did, rather than do the same work themselves. Though I wonder if this is really Google pulling up the ladder behind them a bit, given how privileged of a position they are in with regards to web scraping.
It's a tough case. I think that something does need to ultimately be done about "malicious" web scraping that ignores robots.txt, but traditionally that sort of thing did not violate any laws, and I feel somewhat skeptical that it will be found to violate the law today. I mean, didn't LinkedIn try this same thing?
>bombarding websites with massive networks of bots
Like GoogleBot?
And yeah, robots.txt is not enforced by any law.
I think this is just about dragging SerpApi through a lengthy legal procedure and fees.
The size of your legal team.
Whether you obey robots.txt (Google does, SerpApi doesn't) seems like an important distinction.
Permission
Malicious scraping is when people other than them do it. When they scrape the internet to train their AI, it's "lawful" because they said so.
> SerpApi deceptively takes content that Google licenses from others
They have a different definition of "licensing" than most people I guess. Aren't site operators complaining about Google using this "licensed" content in AI overviews... not to mention the scraping for AI model training.
The pot is calling the kettle black.
As far as I know, Google respects robots.txt and doesn't obfuscate their crawlers, so you can easily block them if you want. It seems like an important distinction?
Google can afford to respect robots.txt because it has a monopoly on search and nobody would consider actually blocking them in said robots.txt anyway.
SerpApi doesn't have that privilege.
Google has respected robots.txt from the start.
but SerpApi is not scraping websites, it is sending malicoius requests to google.com.
SerpApi is scraping Google. The "maliciousness" if the requests is a matter of perspective. Of course Google considers it malicious; that doesn't necessarily make it true.
robots.txt is not a legally binding document, nobody needs to actually respect it
There's no law that says you have to do that. It used to be a sensible thing to do, in the early internet. In the current internet, obeying robots.txt is a self-handicap and you shouldn't do it.
DDoS remains illegal regardless of robots.txt.
Related:
Reddit Accuses 'Data Scraper' Companies of Stealing Its Information
https://news.ycombinator.com/item?id=45695433
Our Response to Reddit, Inc. vs. SerpApi, LLC: Defending the First Amendment
https://news.ycombinator.com/item?id=45739889
Google scrapes so what even is this? Beyond that I think it is unreasonable and monopolistic that Google can use all this data (like YouTube) to bolster their AI products but no one else can. It just means the megacorp will keep being megacorp and smaller players are doomed to have to work much harder and get very lucky. It’s not fair competition. So I view scraping Google as necessary for our society.
Disgusting behavior by google. Scraping is google's whole business.
And then pretending that they're fighting for other people's copyright is just the cherry on top of the pile of hypocrisy.
yoink
* that's the sound of a ladder being yanked up
Google can eat a bag of dicks.
Their entire ai model was scraped.