Google does not hate us... it is worse than that - it is indifferent to us. Hate requires some sort of recognition. I mean this single incident may not mean anything but overall google is heading to an _interesting_ place. In short, it was state of the art but in 20 years it became just another conglomerate sacrificing quality for shareholder gain, I think?
As a search engine, it does not work for me. I see promoted links above the thing I actually search for. Moved to Kagi and didn't look back.
As an AI it does not work for me. I am seeing an arbitrary usage limit, refreshing in 5 hours and a weekly quota given in a percentage. That is as opaque as it gets. Again, to give Kagi as an example I look at my usage details and I see how much is remaining in a clear way. Not working for Kagi by the way, I am just a happy customer.
As a cloud storage, it does not work for me. Probably some shared folder I am working with others has a spam user and/or a hacked account and they periodically spam x-rated notifications. And that's not only me (https://www.reddit.com/r/techsupport/comments/1azf25v/myster...). Moved to apple iCloud and done with it.
Mail is fine. After 22 years of usage, I kind of delegated it to a non-important stage in my life. The important bits have relocated to European providers anyway.
I feel like a crazy person, but I've been using Yandex as the last resort and having positive results in finding stuff that I know is out there but Google has decided to stop letting me see. (I tried DDG but for my use it's been worse than Google).
Nah you're not crazy. I also felt crazy when i discovered that some obscure censored russian search engine gives me overall better search results in 2026 than google.com
Worse, they often aren't even relevant: we searched "passport renewal" and you had to go the the second page to even get the government site that renews passports, and not ad scams masquerading as the real thing. Optimized for engagement, presumably.
Edit: come to think of it, I don't know why I still use Google. I don't care if they track me. But when they have been actively try to prevent me from finding the information I'm looking for, and instead try to scam me?
Even after that, for whatever reason, the next tranche of links is a mixture of AI slop and shopping links. If I'm looking for information about something and not a product to buy, I often have to, gasp, go to the 2nd page of results.
The 'mail is fine' is an impending apocalypse that most people don't think too much about. Google can dump you at any time for any reason or no reason. Your chances are small, but if it happens its incredibly disruptive. I don't know the durable answer, but I definitely need to complete that step of degoogling, the job just seems huge.
Source: a small wiki I help manage, for an obscure game with <10k players, recently had to disable new signups, because the spam was so bad (and it was stuck on an old version of MediaWiki, which didn't have CAPTCHA-support).
On a popular wiki, and it sounds like this one was fairly popular, I imagine even CAPTCHA's won't be enough to stop wiki spammers. If those spammers were posting more than just "buy my penis pill" garbage (e.g. they were putting links to malware sites), Google probably, and somewhat legitimately, saw them as a source of such malware.
I imagine the fix for the OP is a thorough audit/cleansing of all malicious content on the wiki, followed by some sort of appeal to Google (which will no doubt take months, if they even respond at all, because ... Google).
Really OP's only hope is that the Google team responsible for this has an Italian Pokemon fan; otherwise they are probably screwed.
We have very good anti-bot system set up with a good number of Cloudflare fine-tuned rules, limited permissions for newly created accounts, and a very dedicated team of volunteers that patrol the recent edits constantly. I cannot exclude that somewhere on a rarely visited page (out of 37k+) there is a spam link, but I doubt it’s the reason for the deindexing. I think this would also appear on the Google Search Console.
I'd still recommend doing searches for common spam topics to see if you have "bad" stuff. On our wiki everything looked fine until you searched for (say) "finance" (which most users never would) ... and then you'd find a mess of spam finance stuff.
As for whether it's responsible or not, obviously I don't know. What I do know is that, without all the info, "Google saw malicious content on your wiki" is a far more logical theory than "Google just decided to hate us out of the blue".
I saw a comment on here a few days ago and the user mentioned that they use a Captcha AI bot in their day to day life because a solve costs $.003. So even if you had the captcha-enabled new version it might not have helped!
If your project is popular enough to the point where tailored automation make sense there no way to fight spam really.
If its small enough you can usually avoid all the spam bots by adding any none-standard flow in registration procedure. E.g static picture or audio of something only your audience know with like drop down option to click on picture saying "I'm not a bot". Or add one more email verification for first post or edits. Or make users watch large YouTube video at certain timespamt with correct answer, etc. Anything non-standard works.
Breaks 99.9% of automation and SERP spammers wont bother create unique one for your wiki / forum / etc.
If your site is very popular you're fckd obviously and it's just arm race. This is where you can use Hashcash or something that will burn lots of CPU / GPU / RAM / etc single time so spammers will just blacklist you.
Captcha does nothing against the spammers. I have found that blocking email domains from signups works pretty well. My list is at https://www.rejectionwiki.com/index.php?title=MediaWiki:Emai... (this is a built-in feature of Media Wiki and should work ok with most versions)
Do you have any basis for saying that this wiki is overrun with spam, or are you just hand-waving? They were explicit in their Twitter thread about not being full of AI slop, and that they checked their list of pages that were marked as 'crawled but not indexed' and found no abuse.
I understand that you were taken aback by spam attacks on the wiki you help manage, but it's not reasonable to generalize from yours to theirs.
AI slop wouldn’t be on my top ten list of annoying wiki spam, having been the one dealing with such things in the past. You can be free of slop and still overflowing with spam.
How old a version? I've been running a much more obscure game (<150 players, down from ~1k in 2010) for some time, and it was using QuestyCaptcha back in...2008 or so, I think? Certainly at least 15 years ago. It's almost always been sufficient: just put in a couple of questions based on knowledge of the game itself.
Also running a wiki. Similar. Had a sign-up based on in-game knowledge. LLMs now crack it and I had to turn off signups about a year ago. Now people email me directly if they want an account.
Social sites should have all have a tree-based invite system. This would allow wiping out spammers and their enablers in a single hit. It would allow vetting of good actors too.
I feel like the dream solution is more like tree-based content: you see content that is vouched for by people you vouch for; if someone's account is compromised then their vouches get updated to not matter anymore, cutting their whole tree off at the root to make it invisible. Spammers should end up in largely disconnected components of the trees.
Pretty much what xwitter / bsky is on the following page.
The algorithm layer atop twitter was pretty good connecting me with people/content before daddy elon came along. And this algorithm layer is actually needed (in my view) to make the social network thing work, otherwise there is no critical mass
Thankfully bsky is not that good, so I don't get hooked by it at all. But i miss it
Same tree that gave them credentials gives them weight that is used to spread their content.
New User problem has been around for a minute though - wikipedia and stack overflow both faced it, as does every social media platform nowadays.
Reality is, though, new visitors are getting the same blast radius as early adopters got when they started, just that the early adopters now have blast radiuses that are much bigger
- You can't vouch for downstream invites, so the tree aspect isn't useful.
- It's not your fault if someone's account gets taken over by a spammer.
- Just because you vouched for someone once doesn't mean you vouch for them in the future.
- What should the punishment be if you accidentally invite a bad actor?
- Your community has to be large and desirable enough for people to bother. The vast majority of sites will die before anyone cares about jumping through hoops.
Addressing issues like these ends up kinda defeating the ideals of the proposal and regresses it into a mechanic that simply makes it harder to register. Which might be useful wrt anti-spam, but it has its own issues, like people having to constantly grovel for invites, shutting out earnest contributors, etc.
In the "real" world we keep our relationship bubbles to a small number for just this reason - we have our close relationships, the people we trust, and then we have gradually less and less close relationships - people we know to greet, or know have done x or y or z
We know that people who are on our outer orbits should not be shared with in certain ways.
Our communities are in fact lots of overlapping bubbles, x knows y, and y knows z, but x sees them as a stranger.
The internet changes that dynamic, and we don't yet know how to manage it - we cannot all live in the town square, and we know to be very careful there, pickpockets, thieves, and robbers abound - and we have no idea who is who
Again our historical approach has been places like universities, where we have "trusted" advisors (teachers) who guide us on subjects being discussed openly - but who also ensure that we avoid pitfalls, like heated debates where people abandon logic and instead use rhetoric or violence, and who ensure that commercial interests are managed - that is, some advertisements are allowed, but unauthorised advertising is forbidden
That approach (moderation) has its own set of problems
You still need criteria to handle reputation: does an account invited years ago and now spamming affects the reputation of the inviter, how much? What about the hacked accounts?
For small platforms it makes a lot of sense, for larger the potential for abuse is still there in different forms.
Yes Lobste.rs is great but much more limited in conversational scope. I don’t think the content on each site is directly comparable. The sites are not equal in audience and intention. For example, Lobste.rs doesn’t allow rampant evangelism or want to attract start-ups and thus doesn’t attract a more “spammy” crowd.
An organization I'm involved with has had to add Anubis (https://github.com/TecharoHQ/anubis) because of the recent wiki attacks from LLM scrapers. It's finally fixed our outages.
To be honest it's probably just jank on Google's end.
There's a lot of delayed cause and effect in search, and it's much easier to make a minor mistake that excludes 0.1% of websites from crawling or indexing than it is to detect that it's happened except from affected websites telling you about it.
Like in marginalia I've had a bug that affected websites in the condition that if the root path didn't support HEAD, but did support GET with a `Range` header, and it correctly responded with a HTTP 206, then the website wouldn't be indexed because some code that was testing the root document for issues as an initial probe handled that as an error state. Most websites that support range requests also support HEAD (as this usually means the document isn't generated). Except a handful of Caddy-based configurations, about 0.3% of servers.
Why would Google need to direct traffic to the website when they've already scraped and trained their models on the data? Content creators and legitimate websites were wham-bammed and thank-you-ma’amed.
Personifying Google in this way is not realistic. The search team alone at Google is made of thousands of people who are all working on different things with an over-arching mission of making the web MORE accessible, not less. Any release from any of those people could have created a side effect of this kind. Is there a chance it was an intentional policy implementation? Sure. But the odds are heavily against it.
This seems akin to saying that humans shouldn't be personified because their brains are made of millions of neurons that are all doing different things. But the actions or motives of individual processing units are hardly relevant, especially at the scale of The Google. We don't need to speculate how non-malevolent individuals cause harmful side effects. It doesn't even matter what The Google "thinks". The system is what it does, and what it does is consistently operate in ways that are not for the mere benefit of users of the Web. The conceptual model that The Google hates (or is callously indifferent to) us makes far better predictions than a model presuming thousands of people make mistakes while trying to make the Web more accessible. It doesn't matter if the former model isn't a technically perfect reflection of reality. We are less likely to be victim to The Google when we act as if it is a hostile force. Diffusing the results of its actions across thousands of nameless humans increases the risk that one finds themself posting on HN or X about how The Google spontaneously locked them out of their entire life.
Someone at Google are ultimately responsible for the overall direction. Saying that a company is made up by thousands of people and they should be judge, perhaps not individually, but at least not as one gigant whole, is asking the employees to absorb moral responsibility, while the corporate is excused of any wrong doing.
> over-arching mission of making the web MORE accessible, not less
Right, that's why they pushed AMP and upranked AMP pages in their results. That's also why they decided to severely neuter/remove as blocking extensions for Chrome. That's also probably why google search results are getting worse by the month with more and more ads and spam being upranked to the top.
It's because google has a mission of making the web more accessible. Okay bud.
It's laughable to assume good intentions at this point, this predatory monopolist makes every decision against a free and open internet and in favor of monetization, authoritarianism, and enshittification.
So same thing ad-block users have been doing for 20 years now?
Edit: You can downvote, but you can't tell me the difference, can you?
Edit 2: Funny how when you call out ad block users for denying creators revenue, they go on about how the internet was fine in '96, how no one should expect anything for putting content online, or how it's their computer so they can chose what loads on it. Where did those arguments go?
Users take part and improve wikis, it's the whole model. If they don't take the adverts, they still can contribute. Googlebot isn't making edits, not even giving signal to the site about what is useful allowing the owners to hone the site.
Two ways in which issues who have adblock are better than bots.
Users will promote organically, which can win more credence than even a higher listing in SERPs. Depends if your wiki is part of building a community.
Because while your argument sounds nice, if you break out the numbers, it becomes largely meaningless. In fact you find that the average internet user, especially in the tech/gaming space, usually contributes nothing, while watching/loading no ads and self congratulates themselves for doing so while encouraging others to do the same.
What bizarre and absurd line of reasoning. Users who care about their privacy and opt out of downloading ads and malware are 'denying creators revenue'?
Are you denying creators revenue by not reading reading/observing every ad that comes your way and making purchases based on them? Maybe you should read/comment on HN less and focus on consuming more ads instead?
When you don't want the ads and privacy invasion, you don't visit the website. There are still honestly free things on the internet one can enjoy.
Like if a video game is too expensive for your liking, you simply don't buy it. Going and pirating it is not a valid response. You get the game and creator gets nothing. You can just stick to playing honestly free games, there are plenty out there.
This idea that digital data is worthless is stupid child logic born from when kids ruled the internet. Obviously it has value, as evidence by the very top level post I responded to.
(Also, as an aside, it's only heavy ad-block/privacy tool users who get malware and scam ads, because they have no profile and only bottom feeders bid on their views. Regular users get Tide and Chevy ads.)
> When you don't want the ads and privacy invasion, you don't visit the website.
First of all, I can and will visit any website I want, and I will use an ad blocker while doing so. Second - how do you know what ads and privacy invasion a website might have before you visit it? Makes no sense.
> Like if a video game is too expensive for your liking, you simply don't buy it. Going and pirating it is not a valid response
In either case the creator gets zero $. It could be argued that pirating might actually benefit the creator more - since it would increase overall usage/adoption/prevalence of the product/game. So your argument is kinda backwards.
> This idea that digital data is worthless is stupid child logic born from when kids ruled the internet.
You keep mentioning 'kids' and 'teenagers' across your comments seemingly as a way to imply that you have some kind of greybeard wisdom and special knowledge. You don't and your arguments don't make sense - your own realization of that is probably what triggers you to call everyone who disagrees with your kids and teenagers LMAO.
And for the record - intellectual property is a made up scam, the only purpose of which is to stifle competition.
>First of all, I can and will visit any website I want, and I will use an ad blocker while doing so.
And so can LLMs, so I don't see why anyone should be upset about "stealing content"
>In either case the creator gets zero $. It could be argued that pirating might actually benefit the creator more - since it would increase overall usage/adoption/prevalence of the product/game. So your argument is kinda backwards.
So how do you decide (I'm asking you), who are the suckers who pay, and who are the ones that get it free? I say child a lot because it's really only kids who cannot see how a system like that plays out.
Just a heads up, with donation systems, typically ~1% of people convert to a donation.
Whoever can and wants to pay is free to pay. Everyone else is free to not pay. Not sure what the problem with that is - seems like a basic human right/freedom unless your brain is consumed by the marketing/advertising virus.
In many cases running something like an online game requires server s/infra , and also requires an active subscription - not something you can generally get around.
The difference is that I am not preventing anyone else from finding their content. I whitelist ads on sites that have good ad policies, like limiting ad size, labeling ads, and not allowing animated ads.
Advertisers only care about attention, if you don't impose editorial standards they'll contaminate your entire site.
On air reads. Lift a finger for your ads. When I spent more time producing podcasts I categorically rejected (and discouraged my clients from doing) injected ads by 3rd parties. They scream “idgaf” and actual on air reads convert better anyway by huge margins in comparison.
Ublock origin et al can’t block those so there’s your solution. Don’t lazily monetize your content.
> So same thing ad-block users have been doing for 20 years now?
Ad-block users didn't mine Pokémon Central for content, then remove them from search listings. Changing the specific criticism made to the generic "denying creators revenue" is a distortion, because they screwed over all people who wanted visitors, not just the people who wanted visitors to milk them for cash.
If I made a forum about trains because I wanted people to come to the forum to talk about trains, Google milked the forum for all of the accumulated information about trains, then made it impossible to attract new users to talk about trains.
well I didn't downvote but there is an obvious difference in thousands of uncoordinated people doing something whenever it benefits vs. a large organization with automated resources doing things at the kinds of speeds and volumes that automation allows.
Maybe take a moment to consider why people are choosing to use adblockers in the first place. And whether having content being monetized through and relying on ads is even a good thing overall (it's not). Advertising and marketing is fundamentally a negative for society in most cases.
>Maybe take a moment to consider why people are choosing to use adblockers in the first place.
So they can get content without compensating for it.
I've been on this train since the beginning. I was there when ad-block-plus read the writing on the wall 15 years ago and decided to make a truce with advertisers. It was clearly unsustainable for 50% of web users to be effectively parasites, so maybe we can negotiate on acceptable ad practices. But to the users, a truce with advertisers!?!? Ublock Origin was born days later.
Creators don't get compensation when people ad-block.
Creators don't get compensation when LLMs scrape.
It's totally, and completely, unambiguous. The internet just has collective brain damage from the grassroots morals of it being formed 30 years ago by teenagers. How surprising that a bunch of kids decided that the way to save the internet was to make it better for themselves, and worse for the people who make the internet the thing they love.
Some of us have grown up now, and realize the correct answer to save the internet was to not engage with ad supported content period.
There are ways to get paid without ads and you can do on-air reads like I said. adblockers don’t impact them. You also don’t have to play Google and YouTube’s games. I’m sorry folks are caught in that arm’s race between users and Google but Google has made browsing so miserable it’s just reality.
Adblocking is basic security now. I am not compromising on it. I say this as a “content creator”
You don’t need to get sarcastic with me over this.
Content creation comes in many forms. You can also promote things in your copy. People do it all the time. Adblockers aren’t going to somehow remove your words. People disclose their sponsorships at the top/bottom of their written content all the time and frequently use affiliate links.
The same thing happened with my blog a few weeks ago. It was well referenced for years and suddenly almost all of my entries are not indexed anymore. The Search Console indicates that the URLs were crawled but are currently not indexed, and contrary to technical problems, there nothing I can do to fix it, I just have to accept that most of my articles cannot be found via Google anymore.
EDIT: I don't actually think it is related, but now that I think of it, the timing corresponds with when I started setting up TDMRep to forbid using my content to train LLMs.
Same. I've been running a personal blog for over 20 years. Last year, I couldn't find any links to my blog on Google. Went to Google Search Console to find all my links are "Crawled by not indexed", with no reason given.
If Google already slurped up any training data from your site, then not indexing it probably gives them something of a moat over anyone using Google search for site discovery.
Google neither hates nor loves any of us, the only thing it cares about as an institution is cramming as many advertisements in front of as many people as it can get away with to generate increasingly ridiculous piles of money.
This is not meant to be a defense of Google, which is (like virtually every large corporation) completely sociopathic.
Public corporations have historically been multi-cellular biological organisms made up of individual cells working toward the collective goal of continuing the organism's existence. They're probably most analogous to bee hives.
Each cell receives nourishment from the corporation in the form of monetary compensation (and other benefits). Some cells have a more direct role in the "reasoning" process of the organism than others, depending on their logical position within the corporation.
The corporations aren't sentient in the collective, though it can be argued many of their constituent cells are. The corporations are able to influence their environment using individual constituent cells to communicate with similar cells in other organisms.
Ultimately, the corporation itself has the goal of producing value for its owners, since its owners provide the working capital necessary for the corporation to function.
The methods corporations use to achieve their goal of returning value can be opaque to the owners and potentially inscrutable to the individual constituent cells. Their "reasoning" is a manifest property coming from the interaction of the cells with the environment, the cells interacting with each other (both within and outside the corporation), and other organisms.
(There's the neat rub that individual cells can be constituents of multiple organisms simultaneously, too!)
If the owners stop receiving value and withdraw their working capital the corporation becomes unable to nourish its cells and it dies.
Recently these organisms have become biological / technological hybrids, incorporating unconscious computational models in their reasoning process. This change increases the inscrutability and opacity of the reasoning process. It's likely the unconscious computational models will eventually be tasked with communicating with similar models in other organisms, at which point the inscrutability will probably increase by an even greater amount.
It's going to be interesting when the corporations, talking with other corporations, manifestly decide that they don't need human components anymore. All of that can happen without the pesky need for consciousness, too.
I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
> If the owners stop receiving value the organism becomes unable to nourish its cells and it dies.
Owners are people. They're vulnerable to sentiment. There's plenty of failing businesses with their doors open for this reason.
You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
> You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
I'm mainly riffing for fun. I don't have any thesis, beyond just expressing a general unease for how much power corporations have to influence social discourse, laws, and public policy.
I'm using this as an excuse to play w/ the mental picture I've had for decades of corporations as Godzilla-like monsters roaming the social landscape predating, excreting, and generally smashing-up anything that displeases them while individual people look on in horror, mostly powerless.
Now that humans are bolting large computational models onto corporate governance and strategy we're entering an exciting new mecha-Godzilla realm where, likely, individual human accountability to corporate actions will be even less (though it's hard for me to believe that's possible).
re: rationality
Corporations are irrational because all the actors in the corporation, and those outside who are making the rules, are irrational.
Their irrationality, unpredictability, and adaptability to regulation, particularly when they're hulking daikaiju-like monstrosities shambling thru society wantonly smashing their tails into other institutions and social infrastructure (or mating with other entities to create super-monstrosities), is what's troubling to me.
> I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
The law is a component of the environment. The law binds the corporation together, but it also constrains and shapes how it can act. I don't see the human legal system, as it relates to corporations, a whole lot differently than the laws of physics controlling the chemistry that make biological cells work.
A big difference, though, is that corporations can allocate resources to get the law changed. They can alter their environment to suit their manifest desires. Many times they simply adapt to the law (changing business processes to achieve legal compliance). Sometimes they just act counter to the law, likely because some individual cells working in a reasoning capacity will have significant individual gain and very little individual risk (Dieselgate, or maybe the subprime crisis of 2008).
I'm particularly troubled by the Citizens United decision, in the US, because it gave corporations themselves the power of speech. I think they'd always been able to alter their environment through influencing their owners and constituent cells, but this ruling gave them very direct ability. To belabor the Godzilla analogy, we used to be able to call upon the government to battle these monsters when their destruction was too severe. Now the monsters have exciting mind-control powers that they can unleash upon the government (by way of spending on political issues).
> Owners are people. They're vulnerable to sentiment.
Some owners are people, and some of those people are vulnerable to sentiment. I don't put too much faith in individual owners to give much of a crap about what their pet corporations are doing (beyond returning value). A very small fraction of people are invested in individual corporations (and those who are invested individually in a significant manner are highly motivated to help their pet corporations adapt to or change the environment to maximize returns).
Individual people are participating in a different kind of inscrutable manifest organism (pension funds, ETFs, etc) and I don't think they think much about their ownership. Those people are, by and large, just looking at returns, if they're even doing that. I'd argue that kind of ownership by-proxy dilutes individual sentimentality to the point of making it very, very ineffectual.
Increasingly I see marketing as akin to LLM training. We are all being trained (by activating and reinforcing neural pathways in our meaty heads) to respond to certain stimuli in a certain way (e.g.: at the store, select _this_ brand of soap).
I'm not sure most of those calling the shots at Google realizes they even have paying customers other than advertisers. Notably though, website publishers and consumers of their massive products like Search, Gmail, and Android are not really customers.
A bit outside of HN's typical domain, but I wonder the treatment that top ad buyers get from Google.
Like if you are the VP of advertising for Procter & Gamble, and your nephews gmail account gets banned, are these guys treated so specially that they can get a white glove unban just like that? I wouldn't be surprised for Googles golden geese if they can
I wonder about that. Part of me thinks maybe they might not bother, because what are they gonna do, just not advertise? The entire online ad market is basically Google and Meta now. You'd get fired if you pulled all your ads because Google didn't do you a favor.
But on the other hand, I think they might definitely do that kind of thing as more of an old-boys-network type thing.
Wikis are just high-risk for SEO. Getting my own personal wiki to be indexed was such a challenge that I'd just about given up when a friend who is more acquainted with the whole thing helped me make sure I had all the bits and bobs in the right place. If you're not careful, people can easily put spam all over your site and then it'll really ruin your presence on a search engine.
Google is really big, though. Really really big. They're so big that not even all the people inside Google are trustworthy to them on a subject like this.
But they don't universally hate wikis and so on. It's just you have to do a lot of work and make sure you don't have spam on your wiki, and then fill in all of the information in your meta tags, and have a sitemap.xml, and all that. Here's my wiki for example: https://wiki.roshangeorge.dev/w/images/8/89/Screenshot_-_Goo...
Tangential: this would be an excellent time for the Kagi folks to do a Mullvad style campaign around "Remember Google before the apocalypse? Meet Kagi."
After yesterday's keynote and the changes to Search, it became clear in the near future, Google will cease to direct any traffic to websites and the search results will just become a footnote in Gemini's response.
I still have 42k page indexed, but previously I had 20k impressions per day, past week impressions started to change. And now I have 399 impressions per 24 hours :/
Content creators need to accept that traditional SEO is a thing of the past, because traditional search engines are a thing of the past. None of my normie friends use search engines any longer. They just ask the AI - anything and everything - the AI has all the answers they need. The best that any content creators can hope for in terms of engagement is that they were the quoted source and the user cares enough to check sources. Content creators just need to find new ways of driving engagement now, and we're never going back so there's no use crying about it.
I suspect this is a cloudflare thing since the other search engines are doing fine. I'd look closer into your cloudflare settings and see what you can relax.
Google's always adjusting its search rankings, but it's rare for a legitimate site to suffer such a sudden massive hit without reason.
My first thought would be that they accidentally blocked Google's crawler (maybe through some kind of anti-AI setting?) or that Google believes that the site is serving malware or spam. Either scenario can have that kind of effect. I can see that their forum at least appears to have strong Cloudflare anti-bot rules in place, so that might be the case.
They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
But aside from that usual stuff, there's one more possible reason that's specific to this site. In November of last year, the Pokemon Company rebranded their "Pokemon Trainer Club" to "Pokemon Trainer Central", which is the first result that comes up when you search for "Pokemon Central".
That change was made a few months before the sudden drop in traffic, but could still be a viable explanation here. Google does routine re-ranking on a daily basis along with occasional major re-ranking, which happens maybe a few times a year, so the delayed hit that they saw could have come from Google finally recognizing that most people who search for "Pokemon Central" are no longer looking for the wiki like was once true in the past.
A blocked Googlebot would’ve caused a report of 403s in the Search Console, which isn’t the case. And the subdomain has worked perfectly fine for the last 15 years.
You may have a point with the Trainer Central rebranding, but please consider this in the context of Italian language results. It’s not about reaching the home of the wiki (which is pretty much the only page that’s still indexed), it’s all the other search queries (Pokémon names, moves, games, etc - without adding “pokemon central” even) that usually returned our wiki’s dedicated page as first result (or top 5 at least) and now those specific pages are not even indexed anymore.
> They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
Any sources for this? AFAIK, Google treats websites on a subdomain as a separate entity.
> Things we believe you shouldn't focus on: As SEO has evolved, so have the ideas and practices (and at times, misconceptions) related to it. What was considered best practice or top priority in the past may no longer be relevant or effective due to the way search engines (and the internet) have developed over time.
> Subdomains versus subdirectories: From a business point of view, do whatever makes sense for your business. For example, it might be easier to manage the site if it's segmented by subdirectories, but other times it might make sense to partition topics into subdomains, depending on your site's topic or industry.
Doesn't quite explicitly say it treats them the same, but it kinda implies it.
Google have also blatantly lied in the past about SEO. My familiarity with the subject is about a decade old, but I don't expect Google to have improved in that regard, quite the contrary. I don't think you can take anything in their guidelines as 'truth that will help for SEO'.
It appears for me when I search for it. Even Gemini is cool with looking for it.
Here is a part of the Gemini result I got which was directly above the regular result link.
"Pokémon Central is a major community network and independent Italian encyclopedia for everything Pokémon-related"
Honestly, the title is super clickbait and it doesn't even reflect reality.
Its so easy picking some giant entity far away and create some drama about it. Dont get me wrong, I am not a google fan, but I also dislike clickbaits and whiney dramatic claims, moreover if unverified.
My guess is that the combination of Wiki and Pokemon is highly suspect for Google.
The Pokemon Industrial Complex has advanced astroturfing especially on YouTube/Twitch, where streamers mention the damn things in any second episode, they "accidentally" meet people going to Pokemon conventions in live streams and so on.
One of the sad things about this story is that everyone has to read tea leaves to guess what reasoning might be going on at Google's end. Tech companies have normalized the practice of cutting people off with no explanation, or saying 'we investigated and found a violation' without articulating what the violation is. Naturally, they want to secure themselves against abuse and people trying to game their system, but refusing to provide any information does not achieve that.
It does infuriate legitimate users, enables other kind of abuse and scamming (eg immunize yourself against delisting with this one weird trick!', link farming etc), and act as a fig leaf for abusive behavior by platform operators. Effectively, we've allowed large teach companies to act as digital dictatorships with no accountability to their customers. Yes I consider users to be 'customers' even if they're uploading content or doing searches 'for free'. If you're monetizing their activity on your platform, they are your customers whether or not you call them that to avoid legal liability.
They're a private company, they can ban whoever they want.
Or at least that's what I heard a few years ago when it was politically incorrect people complaining about being banned with no accountability. They're a private company, it's their servers. You may not even be paying anything. So they can do anything they want to you and you have no cause for complaint.
I think it's become increasingly clear that search engines, in their most basic form, aren't cut out for the internet of today. The open internet is basically dead and I see the world returning to old yahoo-like web page indexes that are manually verified and sorted
Kagi has been mentioned already, just to provide anecdotal reference: searching for "pokemon wiki" with the country set to Italy shows OP's website as first result.
Pokemon Central runs ads (Google AdSense at that!), which is probably how they pay for everything.
Google is likely their biggest inbound source of traffic, so they're probably experiencing a marked revenue drop as well.
It's unfortunate that so many livelihoods are subject to the capricious whims of a single company. A company that is increasingly seeking to keep users on their engine without sending eyeballs or revenue to any third parties at all.
We're watching Google's "embrace-extend-extinguish" arc for the web. It's not over by a long shot, but they absolutely intend to finish the job.
Why is it a lot of work? Could you specify some
off the more difficult effort? Wouldn’t LLMs help speed this up? This is the one area where I’d think Llms could really take Google down by empowering in house ad platforms.
Depends on how in-house you want to go. If you go full in-house, you'll need sales staff to make deals with advertisers. You'll then need a way of hosting the media provided. You'll need a way to deal with media that does not match what you've requested. You'll need a system to allocate ad space accordingly to contracts with ad clients. It's like a whole new department in your company.
I don’t have a sales staff. I just call a company, tell them why I’m calling and what’s the opportunity, crack some jokes, get serious and make them an offer. It helps that I’m in a niche, thoroughly know the sector and that they most likely already know my websites. As long as I can get to the owner of the company, then I’m golden.
We would like the wiki to be free of ads, but hosting costs at our scale are real. Since we don’t like ads either, we compromise like this: users can register for free and never see an ad (they are only served to anonymous visitors); they can also use an ad blocker and we won’t bug them about it.
No, I think "us" is apt, considering this will eventually affect all sites that rely on traffic from Google search, which is basically every text heavy site.
All we can hope for is that people will stop using search (after eventually having enough of the AI wave) for these sort of niche sites and will bookmark and access them directly in future. I don't have much hope.
I spend a lot of time wondering about the true role of search in people's lives in 2026. When I watch people use the Internet, it seems like most of them perform searches simply as fuzzy-matching navigation to the websites they use. Like the way many people use Spotlight to launch desktop apps.
I think this is because
(A) bookmarks lists are inconvenient - scrolling to find a bookmark is slower than typing "youtube" or (cringe) "bank of america" in the URL bar
(B) typing URLs directly requires precision of memory with TLDs being numerous and even things that were once predictable are now mere suggestions (e.g. is your city or town at cityofwhatever.com? city.org? city.gov? Could be anything!)
(C) related to (B) if you screw up a full URL you may well end up at a phishing site that looks like the site you wanted.
I really believe that 90% of Google and Bing searches today are probably for the names (or misspelled or partial names) of the top 100 websites.
If the dominant browsers weren't Google Chrome and Mobile Safari (who gets paid by Google for every search) browsers would build bookmarks for you of your frequently-used sites, and ordered by frequency of visits, present those for direct navigation when you type a word in the search bar, and not send any query to a search engine if you chose one of those. But all incentives point very strongly against doing that and toward sending you to a SERP with 13 ads and an "AI Overview" above the organic results.
Perhaps they will investigate why 541,000 pages aren’t being indexed. In my experience, Google provides adequate tools for identifying and resolving indexing issues.
Google won’t serve pages it hasn’t indexed. Seems they left a lot of relevant details out of that tweet.
Edit: and the most likely answer would be that their current robots.txt disallows virtually all indexing. I’m no SEO expert but entries like this seem like footguns:
User-agent: Google-Extended
Disallow: /
Edit 2: there’s more info in the full thread but that was only viewable via the xcancel link someone else shared (despite having the X app installed - deeplinks don’t work today). A helpful example of why X is not the best platform for sharing multi-post threads. Seems robots.txt was considered but ruled out.
In the first image you can see how indexed pages go from 40k+ to 11 in the matter of days. Further down the thread I show how 114k+ pages are marked as “crawled but not index” and we can’t understand why. The rest is stuff that is (correctly) blocked by robots.txt.
Google does not hate us... it is worse than that - it is indifferent to us. Hate requires some sort of recognition. I mean this single incident may not mean anything but overall google is heading to an _interesting_ place. In short, it was state of the art but in 20 years it became just another conglomerate sacrificing quality for shareholder gain, I think?
As a search engine, it does not work for me. I see promoted links above the thing I actually search for. Moved to Kagi and didn't look back.
As an AI it does not work for me. I am seeing an arbitrary usage limit, refreshing in 5 hours and a weekly quota given in a percentage. That is as opaque as it gets. Again, to give Kagi as an example I look at my usage details and I see how much is remaining in a clear way. Not working for Kagi by the way, I am just a happy customer.
As a cloud storage, it does not work for me. Probably some shared folder I am working with others has a spam user and/or a hacked account and they periodically spam x-rated notifications. And that's not only me (https://www.reddit.com/r/techsupport/comments/1azf25v/myster...). Moved to apple iCloud and done with it.
Mail is fine. After 22 years of usage, I kind of delegated it to a non-important stage in my life. The important bits have relocated to European providers anyway.
I feel like a crazy person, but I've been using Yandex as the last resort and having positive results in finding stuff that I know is out there but Google has decided to stop letting me see. (I tried DDG but for my use it's been worse than Google).
Someone round here said Yandex shows you what you searched for, while Google shows you what it thinks you should have searched for.
Nah you're not crazy. I also felt crazy when i discovered that some obscure censored russian search engine gives me overall better search results in 2026 than google.com
Not crazy, I always resort to yandex when I know google is not showing me the results I am looking for
DDG doesn’t click for me sadly, and I cannot point my finger to where or why
A bet the stupid name doesn’t help.
I know what you mean though, I use it but it’s never quite right. Hard to say exactly why.
The promoted links have gotten insane, the first 5-6 links often appear to be ads
Worse, they often aren't even relevant: we searched "passport renewal" and you had to go the the second page to even get the government site that renews passports, and not ad scams masquerading as the real thing. Optimized for engagement, presumably.
Edit: come to think of it, I don't know why I still use Google. I don't care if they track me. But when they have been actively try to prevent me from finding the information I'm looking for, and instead try to scam me?
> Edit: come to think of it, I don't know why I still use Google.
A guess: because you type queries in the URL bar, and they're the default search engine in your web browser?
(I'm convinced that these days, this is 90% of Google's advantage)
Image search is so hyper-optimised for shopping it's useless.
Even after that, for whatever reason, the next tranche of links is a mixture of AI slop and shopping links. If I'm looking for information about something and not a product to buy, I often have to, gasp, go to the 2nd page of results.
The 'mail is fine' is an impending apocalypse that most people don't think too much about. Google can dump you at any time for any reason or no reason. Your chances are small, but if it happens its incredibly disruptive. I don't know the durable answer, but I definitely need to complete that step of degoogling, the job just seems huge.
Get a custom domain. Strat using that. Route to Gmail to start but easily decouple.
It took me about a year of updates but now I rarely get anything to a @gmail
They're a wiki. Wiki spammers are relentless now.
Source: a small wiki I help manage, for an obscure game with <10k players, recently had to disable new signups, because the spam was so bad (and it was stuck on an old version of MediaWiki, which didn't have CAPTCHA-support).
On a popular wiki, and it sounds like this one was fairly popular, I imagine even CAPTCHA's won't be enough to stop wiki spammers. If those spammers were posting more than just "buy my penis pill" garbage (e.g. they were putting links to malware sites), Google probably, and somewhat legitimately, saw them as a source of such malware.
I imagine the fix for the OP is a thorough audit/cleansing of all malicious content on the wiki, followed by some sort of appeal to Google (which will no doubt take months, if they even respond at all, because ... Google).
Really OP's only hope is that the Google team responsible for this has an Italian Pokemon fan; otherwise they are probably screwed.
We have very good anti-bot system set up with a good number of Cloudflare fine-tuned rules, limited permissions for newly created accounts, and a very dedicated team of volunteers that patrol the recent edits constantly. I cannot exclude that somewhere on a rarely visited page (out of 37k+) there is a spam link, but I doubt it’s the reason for the deindexing. I think this would also appear on the Google Search Console.
I'd still recommend doing searches for common spam topics to see if you have "bad" stuff. On our wiki everything looked fine until you searched for (say) "finance" (which most users never would) ... and then you'd find a mess of spam finance stuff.
As for whether it's responsible or not, obviously I don't know. What I do know is that, without all the info, "Google saw malicious content on your wiki" is a far more logical theory than "Google just decided to hate us out of the blue".
“I was having a hard time until I found a great investment advisor.”
“How can I contact your advisor?”
“Their name is <three part unique name>; just search for them and reach out.”
“Great. I found them and their results look impressive. I reached out and hope they get back to me soon.”
I saw a comment on here a few days ago and the user mentioned that they use a Captcha AI bot in their day to day life because a solve costs $.003. So even if you had the captcha-enabled new version it might not have helped!
If your project is popular enough to the point where tailored automation make sense there no way to fight spam really.
If its small enough you can usually avoid all the spam bots by adding any none-standard flow in registration procedure. E.g static picture or audio of something only your audience know with like drop down option to click on picture saying "I'm not a bot". Or add one more email verification for first post or edits. Or make users watch large YouTube video at certain timespamt with correct answer, etc. Anything non-standard works.
Breaks 99.9% of automation and SERP spammers wont bother create unique one for your wiki / forum / etc.
If your site is very popular you're fckd obviously and it's just arm race. This is where you can use Hashcash or something that will burn lots of CPU / GPU / RAM / etc single time so spammers will just blacklist you.
Captcha does nothing against the spammers. I have found that blocking email domains from signups works pretty well. My list is at https://www.rejectionwiki.com/index.php?title=MediaWiki:Emai... (this is a built-in feature of Media Wiki and should work ok with most versions)
Weird Gloop (wiki host, started with runescape but now has dozens) has blogged about this https://weirdgloop.org/blog/clankers
Do you have any basis for saying that this wiki is overrun with spam, or are you just hand-waving? They were explicit in their Twitter thread about not being full of AI slop, and that they checked their list of pages that were marked as 'crawled but not indexed' and found no abuse.
I understand that you were taken aback by spam attacks on the wiki you help manage, but it's not reasonable to generalize from yours to theirs.
AI slop wouldn’t be on my top ten list of annoying wiki spam, having been the one dealing with such things in the past. You can be free of slop and still overflowing with spam.
How old a version? I've been running a much more obscure game (<150 players, down from ~1k in 2010) for some time, and it was using QuestyCaptcha back in...2008 or so, I think? Certainly at least 15 years ago. It's almost always been sufficient: just put in a couple of questions based on knowledge of the game itself.
Also running a wiki. Similar. Had a sign-up based on in-game knowledge. LLMs now crack it and I had to turn off signups about a year ago. Now people email me directly if they want an account.
Social sites should have all have a tree-based invite system. This would allow wiping out spammers and their enablers in a single hit. It would allow vetting of good actors too.
I feel like the dream solution is more like tree-based content: you see content that is vouched for by people you vouch for; if someone's account is compromised then their vouches get updated to not matter anymore, cutting their whole tree off at the root to make it invisible. Spammers should end up in largely disconnected components of the trees.
Pretty much what xwitter / bsky is on the following page. The algorithm layer atop twitter was pretty good connecting me with people/content before daddy elon came along. And this algorithm layer is actually needed (in my view) to make the social network thing work, otherwise there is no critical mass
Thankfully bsky is not that good, so I don't get hooked by it at all. But i miss it
How does new content or content from new accounts get seen by anyone?
People search for a thing, find said thing, then share it themselves under their own name. You know, like how conversations work in real life
Same tree that gave them credentials gives them weight that is used to spread their content.
New User problem has been around for a minute though - wikipedia and stack overflow both faced it, as does every social media platform nowadays.
Reality is, though, new visitors are getting the same blast radius as early adopters got when they started, just that the early adopters now have blast radiuses that are much bigger
iykyk rules
It doesn't solve as much as it sounds.
- You can't vouch for downstream invites, so the tree aspect isn't useful.
- It's not your fault if someone's account gets taken over by a spammer.
- Just because you vouched for someone once doesn't mean you vouch for them in the future.
- What should the punishment be if you accidentally invite a bad actor?
- Your community has to be large and desirable enough for people to bother. The vast majority of sites will die before anyone cares about jumping through hoops.
Addressing issues like these ends up kinda defeating the ideals of the proposal and regresses it into a mechanic that simply makes it harder to register. Which might be useful wrt anti-spam, but it has its own issues, like people having to constantly grovel for invites, shutting out earnest contributors, etc.
In the "real" world we keep our relationship bubbles to a small number for just this reason - we have our close relationships, the people we trust, and then we have gradually less and less close relationships - people we know to greet, or know have done x or y or z
We know that people who are on our outer orbits should not be shared with in certain ways.
Our communities are in fact lots of overlapping bubbles, x knows y, and y knows z, but x sees them as a stranger.
The internet changes that dynamic, and we don't yet know how to manage it - we cannot all live in the town square, and we know to be very careful there, pickpockets, thieves, and robbers abound - and we have no idea who is who
Again our historical approach has been places like universities, where we have "trusted" advisors (teachers) who guide us on subjects being discussed openly - but who also ensure that we avoid pitfalls, like heated debates where people abandon logic and instead use rhetoric or violence, and who ensure that commercial interests are managed - that is, some advertisements are allowed, but unauthorised advertising is forbidden
That approach (moderation) has its own set of problems
You still need criteria to handle reputation: does an account invited years ago and now spamming affects the reputation of the inviter, how much? What about the hacked accounts?
For small platforms it makes a lot of sense, for larger the potential for abuse is still there in different forms.
Now you just created a market for farmed "legit" accounts.
Yes, but the site operator can significantly increase the market price for such an account. This makes spamming more expensive.
Blue tick accounts :)
That’s literally how Facebook started
I remember begging my older step brother for an invite since he had the college email to get in
Then it’s just hacked account whack-a-mole and deciding who legitimately got their account hacked and who is lying.
It raises the bar at least somewhat though!
Interesting to compare this site and lobste.rs for that
Both from safety and volume perspectives, I’d imagine. Openness has value.
I find myself much more interested in the conversations here, but I enjoy the more tangential discussions.
Yes Lobste.rs is great but much more limited in conversational scope. I don’t think the content on each site is directly comparable. The sites are not equal in audience and intention. For example, Lobste.rs doesn’t allow rampant evangelism or want to attract start-ups and thus doesn’t attract a more “spammy” crowd.
Lobste.rs has an invite based system however.
An organization I'm involved with has had to add Anubis (https://github.com/TecharoHQ/anubis) because of the recent wiki attacks from LLM scrapers. It's finally fixed our outages.
To be honest it's probably just jank on Google's end.
There's a lot of delayed cause and effect in search, and it's much easier to make a minor mistake that excludes 0.1% of websites from crawling or indexing than it is to detect that it's happened except from affected websites telling you about it.
Like in marginalia I've had a bug that affected websites in the condition that if the root path didn't support HEAD, but did support GET with a `Range` header, and it correctly responded with a HTTP 206, then the website wouldn't be indexed because some code that was testing the root document for issues as an initial probe handled that as an error state. Most websites that support range requests also support HEAD (as this usually means the document isn't generated). Except a handful of Caddy-based configurations, about 0.3% of servers.
Or just some AI flagging it as some sort of content they don't want to show. There's no way they can be perfect at that.
from 511k indexed paged to just 11? that is some serious jank
From the perspective of a web search engine indexing tens to hundreds of billions of documents 500k docs is not very noticeable.
Why would Google need to direct traffic to the website when they've already scraped and trained their models on the data? Content creators and legitimate websites were wham-bammed and thank-you-ma’amed.
Personifying Google in this way is not realistic. The search team alone at Google is made of thousands of people who are all working on different things with an over-arching mission of making the web MORE accessible, not less. Any release from any of those people could have created a side effect of this kind. Is there a chance it was an intentional policy implementation? Sure. But the odds are heavily against it.
This seems akin to saying that humans shouldn't be personified because their brains are made of millions of neurons that are all doing different things. But the actions or motives of individual processing units are hardly relevant, especially at the scale of The Google. We don't need to speculate how non-malevolent individuals cause harmful side effects. It doesn't even matter what The Google "thinks". The system is what it does, and what it does is consistently operate in ways that are not for the mere benefit of users of the Web. The conceptual model that The Google hates (or is callously indifferent to) us makes far better predictions than a model presuming thousands of people make mistakes while trying to make the Web more accessible. It doesn't matter if the former model isn't a technically perfect reflection of reality. We are less likely to be victim to The Google when we act as if it is a hostile force. Diffusing the results of its actions across thousands of nameless humans increases the risk that one finds themself posting on HN or X about how The Google spontaneously locked them out of their entire life.
Someone at Google are ultimately responsible for the overall direction. Saying that a company is made up by thousands of people and they should be judge, perhaps not individually, but at least not as one gigant whole, is asking the employees to absorb moral responsibility, while the corporate is excused of any wrong doing.
Is this irony? Cause there’s no way anyone believes these “we want to make the world a better place” cliches anymore lol
> over-arching mission of making the web MORE accessible, not less
Right, that's why they pushed AMP and upranked AMP pages in their results. That's also why they decided to severely neuter/remove as blocking extensions for Chrome. That's also probably why google search results are getting worse by the month with more and more ads and spam being upranked to the top.
It's because google has a mission of making the web more accessible. Okay bud.
It's laughable to assume good intentions at this point, this predatory monopolist makes every decision against a free and open internet and in favor of monetization, authoritarianism, and enshittification.
The over arching mission is to make profit.
And accessibility was meant for Google so they can collect all the data to make even more profit.
lol.
I was listening to David Bowie's Suffragette City as I read your comment (Apparently Bowie was a popularizer of 'wham bam, tym' usage)
>wham-bammed and thank-you-ma’amed.
So same thing ad-block users have been doing for 20 years now?
Edit: You can downvote, but you can't tell me the difference, can you?
Edit 2: Funny how when you call out ad block users for denying creators revenue, they go on about how the internet was fine in '96, how no one should expect anything for putting content online, or how it's their computer so they can chose what loads on it. Where did those arguments go?
Users take part and improve wikis, it's the whole model. If they don't take the adverts, they still can contribute. Googlebot isn't making edits, not even giving signal to the site about what is useful allowing the owners to hone the site.
Two ways in which issues who have adblock are better than bots.
Users will promote organically, which can win more credence than even a higher listing in SERPs. Depends if your wiki is part of building a community.
Does "users" refer to 100% of users or 0.05%?
Because while your argument sounds nice, if you break out the numbers, it becomes largely meaningless. In fact you find that the average internet user, especially in the tech/gaming space, usually contributes nothing, while watching/loading no ads and self congratulates themselves for doing so while encouraging others to do the same.
Does "the numbers" refer to observable data or gross generalization?
What bizarre and absurd line of reasoning. Users who care about their privacy and opt out of downloading ads and malware are 'denying creators revenue'?
Are you denying creators revenue by not reading reading/observing every ad that comes your way and making purchases based on them? Maybe you should read/comment on HN less and focus on consuming more ads instead?
What at an incredibly stupid thing to say.
When you don't want the ads and privacy invasion, you don't visit the website. There are still honestly free things on the internet one can enjoy.
Like if a video game is too expensive for your liking, you simply don't buy it. Going and pirating it is not a valid response. You get the game and creator gets nothing. You can just stick to playing honestly free games, there are plenty out there.
This idea that digital data is worthless is stupid child logic born from when kids ruled the internet. Obviously it has value, as evidence by the very top level post I responded to.
(Also, as an aside, it's only heavy ad-block/privacy tool users who get malware and scam ads, because they have no profile and only bottom feeders bid on their views. Regular users get Tide and Chevy ads.)
> When you don't want the ads and privacy invasion, you don't visit the website.
First of all, I can and will visit any website I want, and I will use an ad blocker while doing so. Second - how do you know what ads and privacy invasion a website might have before you visit it? Makes no sense.
> Like if a video game is too expensive for your liking, you simply don't buy it. Going and pirating it is not a valid response
In either case the creator gets zero $. It could be argued that pirating might actually benefit the creator more - since it would increase overall usage/adoption/prevalence of the product/game. So your argument is kinda backwards.
> This idea that digital data is worthless is stupid child logic born from when kids ruled the internet.
You keep mentioning 'kids' and 'teenagers' across your comments seemingly as a way to imply that you have some kind of greybeard wisdom and special knowledge. You don't and your arguments don't make sense - your own realization of that is probably what triggers you to call everyone who disagrees with your kids and teenagers LMAO.
And for the record - intellectual property is a made up scam, the only purpose of which is to stifle competition.
>First of all, I can and will visit any website I want, and I will use an ad blocker while doing so.
And so can LLMs, so I don't see why anyone should be upset about "stealing content"
>In either case the creator gets zero $. It could be argued that pirating might actually benefit the creator more - since it would increase overall usage/adoption/prevalence of the product/game. So your argument is kinda backwards.
So how do you decide (I'm asking you), who are the suckers who pay, and who are the ones that get it free? I say child a lot because it's really only kids who cannot see how a system like that plays out.
Just a heads up, with donation systems, typically ~1% of people convert to a donation.
Whoever can and wants to pay is free to pay. Everyone else is free to not pay. Not sure what the problem with that is - seems like a basic human right/freedom unless your brain is consumed by the marketing/advertising virus.
In many cases running something like an online game requires server s/infra , and also requires an active subscription - not something you can generally get around.
Users, ad-block users, and scrapers all consume the publicly-available content whether you like it or not.
I expect the difference is that the scrapers are the most likely to regurgitate the content one way or the other.
The difference is that I am not preventing anyone else from finding their content. I whitelist ads on sites that have good ad policies, like limiting ad size, labeling ads, and not allowing animated ads.
Advertisers only care about attention, if you don't impose editorial standards they'll contaminate your entire site.
In the tech space, using youtube as an example, tech youtubers, who are widely lauded, still have about 40-50% of users ad-blocking and <1% donating.
So thank you, but you are one of about 14 people on the internet who actually use a whitelist.
On air reads. Lift a finger for your ads. When I spent more time producing podcasts I categorically rejected (and discouraged my clients from doing) injected ads by 3rd parties. They scream “idgaf” and actual on air reads convert better anyway by huge margins in comparison.
Ublock origin et al can’t block those so there’s your solution. Don’t lazily monetize your content.
The actual answer is to move everything to an app and kill your API, so you control everything in a locked down environment.
This is a much bigger issue than just podcasts. It's every form of binary encoded data.
If there is any model on the internet that has proven you don't need to monetize through ads for a working business model, it's Wikipedia.
Except that it isn't a business.
> So same thing ad-block users have been doing for 20 years now?
Ad-block users didn't mine Pokémon Central for content, then remove them from search listings. Changing the specific criticism made to the generic "denying creators revenue" is a distortion, because they screwed over all people who wanted visitors, not just the people who wanted visitors to milk them for cash.
If I made a forum about trains because I wanted people to come to the forum to talk about trains, Google milked the forum for all of the accumulated information about trains, then made it impossible to attract new users to talk about trains.
well I didn't downvote but there is an obvious difference in thousands of uncoordinated people doing something whenever it benefits vs. a large organization with automated resources doing things at the kinds of speeds and volumes that automation allows.
You can run unblockable ads on your site.
You just have to not use third party integrations that run untrusted code on your visitors computers.
The edits are likely why you’re getting downvoted so much tbh.
Trust me, the downvotes were instant.
People really hate it when you hold up a mirror to illustrate a problem. They tend to reflexively punch the mirror
Maybe take a moment to consider why people are choosing to use adblockers in the first place. And whether having content being monetized through and relying on ads is even a good thing overall (it's not). Advertising and marketing is fundamentally a negative for society in most cases.
>Maybe take a moment to consider why people are choosing to use adblockers in the first place.
So they can get content without compensating for it.
I've been on this train since the beginning. I was there when ad-block-plus read the writing on the wall 15 years ago and decided to make a truce with advertisers. It was clearly unsustainable for 50% of web users to be effectively parasites, so maybe we can negotiate on acceptable ad practices. But to the users, a truce with advertisers!?!? Ublock Origin was born days later.
Users do not compensate websites for serving ads. Your argument just doesn't make any sense.
Also - negotiating 'a truce with advertisers'? What does that even mean? Granting the ads industry even more power and control over the internet?
Can you come up with an idea that isn't a dystopian hellhole on its face?
Man, I wish folks calibrated their E(I am actually wrong|downvotes). Have you considered what that value could be in this case?
Creators don't get compensation when people ad-block.
Creators don't get compensation when LLMs scrape.
It's totally, and completely, unambiguous. The internet just has collective brain damage from the grassroots morals of it being formed 30 years ago by teenagers. How surprising that a bunch of kids decided that the way to save the internet was to make it better for themselves, and worse for the people who make the internet the thing they love.
Some of us have grown up now, and realize the correct answer to save the internet was to not engage with ad supported content period.
There are ways to get paid without ads and you can do on-air reads like I said. adblockers don’t impact them. You also don’t have to play Google and YouTube’s games. I’m sorry folks are caught in that arm’s race between users and Google but Google has made browsing so miserable it’s just reality.
Adblocking is basic security now. I am not compromising on it. I say this as a “content creator”
Please ping me when you figure out how to do on-air reads on a website.
You don’t need to get sarcastic with me over this.
Content creation comes in many forms. You can also promote things in your copy. People do it all the time. Adblockers aren’t going to somehow remove your words. People disclose their sponsorships at the top/bottom of their written content all the time and frequently use affiliate links.
The same thing happened with my blog a few weeks ago. It was well referenced for years and suddenly almost all of my entries are not indexed anymore. The Search Console indicates that the URLs were crawled but are currently not indexed, and contrary to technical problems, there nothing I can do to fix it, I just have to accept that most of my articles cannot be found via Google anymore.
EDIT: I don't actually think it is related, but now that I think of it, the timing corresponds with when I started setting up TDMRep to forbid using my content to train LLMs.
Same. I've been running a personal blog for over 20 years. Last year, I couldn't find any links to my blog on Google. Went to Google Search Console to find all my links are "Crawled by not indexed", with no reason given.
If Google already slurped up any training data from your site, then not indexing it probably gives them something of a moat over anyone using Google search for site discovery.
I can even tell you that Google hates us all
Google neither hates nor loves any of us, the only thing it cares about as an institution is cramming as many advertisements in front of as many people as it can get away with to generate increasingly ridiculous piles of money.
This is not meant to be a defense of Google, which is (like virtually every large corporation) completely sociopathic.
Public corporations have historically been multi-cellular biological organisms made up of individual cells working toward the collective goal of continuing the organism's existence. They're probably most analogous to bee hives.
Each cell receives nourishment from the corporation in the form of monetary compensation (and other benefits). Some cells have a more direct role in the "reasoning" process of the organism than others, depending on their logical position within the corporation.
The corporations aren't sentient in the collective, though it can be argued many of their constituent cells are. The corporations are able to influence their environment using individual constituent cells to communicate with similar cells in other organisms.
Ultimately, the corporation itself has the goal of producing value for its owners, since its owners provide the working capital necessary for the corporation to function.
The methods corporations use to achieve their goal of returning value can be opaque to the owners and potentially inscrutable to the individual constituent cells. Their "reasoning" is a manifest property coming from the interaction of the cells with the environment, the cells interacting with each other (both within and outside the corporation), and other organisms.
(There's the neat rub that individual cells can be constituents of multiple organisms simultaneously, too!)
If the owners stop receiving value and withdraw their working capital the corporation becomes unable to nourish its cells and it dies.
Recently these organisms have become biological / technological hybrids, incorporating unconscious computational models in their reasoning process. This change increases the inscrutability and opacity of the reasoning process. It's likely the unconscious computational models will eventually be tasked with communicating with similar models in other organisms, at which point the inscrutability will probably increase by an even greater amount.
It's going to be interesting when the corporations, talking with other corporations, manifestly decide that they don't need human components anymore. All of that can happen without the pesky need for consciousness, too.
> made up of individual cells
I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
> If the owners stop receiving value the organism becomes unable to nourish its cells and it dies.
Owners are people. They're vulnerable to sentiment. There's plenty of failing businesses with their doors open for this reason.
You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
Bottom-line-up-front:
> You're attempting to rationalize something in biological terms that's somewhat irrational in logical terms.
I'm mainly riffing for fun. I don't have any thesis, beyond just expressing a general unease for how much power corporations have to influence social discourse, laws, and public policy.
I'm using this as an excuse to play w/ the mental picture I've had for decades of corporations as Godzilla-like monsters roaming the social landscape predating, excreting, and generally smashing-up anything that displeases them while individual people look on in horror, mostly powerless.
Now that humans are bolting large computational models onto corporate governance and strategy we're entering an exciting new mecha-Godzilla realm where, likely, individual human accountability to corporate actions will be even less (though it's hard for me to believe that's possible).
re: rationality
Corporations are irrational because all the actors in the corporation, and those outside who are making the rules, are irrational.
Their irrationality, unpredictability, and adaptability to regulation, particularly when they're hulking daikaiju-like monstrosities shambling thru society wantonly smashing their tails into other institutions and social infrastructure (or mating with other entities to create super-monstrosities), is what's troubling to me.
> I think this analogy is flawed. Corporations cannot exist without laws pertaining to them. They're made up of _laws_. The individual components all have actions dictated to them by these laws.
The law is a component of the environment. The law binds the corporation together, but it also constrains and shapes how it can act. I don't see the human legal system, as it relates to corporations, a whole lot differently than the laws of physics controlling the chemistry that make biological cells work.
A big difference, though, is that corporations can allocate resources to get the law changed. They can alter their environment to suit their manifest desires. Many times they simply adapt to the law (changing business processes to achieve legal compliance). Sometimes they just act counter to the law, likely because some individual cells working in a reasoning capacity will have significant individual gain and very little individual risk (Dieselgate, or maybe the subprime crisis of 2008).
I'm particularly troubled by the Citizens United decision, in the US, because it gave corporations themselves the power of speech. I think they'd always been able to alter their environment through influencing their owners and constituent cells, but this ruling gave them very direct ability. To belabor the Godzilla analogy, we used to be able to call upon the government to battle these monsters when their destruction was too severe. Now the monsters have exciting mind-control powers that they can unleash upon the government (by way of spending on political issues).
> Owners are people. They're vulnerable to sentiment.
Some owners are people, and some of those people are vulnerable to sentiment. I don't put too much faith in individual owners to give much of a crap about what their pet corporations are doing (beyond returning value). A very small fraction of people are invested in individual corporations (and those who are invested individually in a significant manner are highly motivated to help their pet corporations adapt to or change the environment to maximize returns).
Individual people are participating in a different kind of inscrutable manifest organism (pension funds, ETFs, etc) and I don't think they think much about their ownership. Those people are, by and large, just looking at returns, if they're even doing that. I'd argue that kind of ownership by-proxy dilutes individual sentimentality to the point of making it very, very ineffectual.
Increasingly I see marketing as akin to LLM training. We are all being trained (by activating and reinforcing neural pathways in our meaty heads) to respond to certain stimuli in a certain way (e.g.: at the store, select _this_ brand of soap).
Don't make the mistake of anthropomorphizing Google, as a wise man once said.
All large companies are sociopaths, but few tech companies treat their paying customers with the level of contempt that Google does.
I'm not sure most of those calling the shots at Google realizes they even have paying customers other than advertisers. Notably though, website publishers and consumers of their massive products like Search, Gmail, and Android are not really customers.
A bit outside of HN's typical domain, but I wonder the treatment that top ad buyers get from Google.
Like if you are the VP of advertising for Procter & Gamble, and your nephews gmail account gets banned, are these guys treated so specially that they can get a white glove unban just like that? I wouldn't be surprised for Googles golden geese if they can
I wonder about that. Part of me thinks maybe they might not bother, because what are they gonna do, just not advertise? The entire online ad market is basically Google and Meta now. You'd get fired if you pulled all your ads because Google didn't do you a favor.
But on the other hand, I think they might definitely do that kind of thing as more of an old-boys-network type thing.
This. They will call up a high rank and tell them about it and then it would trickle down. It would not happen through regular support channels.
I don't think that's necessarily true - some large companies are coöps, they don't seem to be sociopathic?
Coops usually wouldn't be, but they almost all went under some time back where I live.
You guys made the classic SEO mistake of building a real community site instead of a Reddit thread, a coupon subfolder, or an AI summary.
Scherzi a parte, spero che possiate recuperare presto…
Grazie! Speriamo anche noi.
Wikis are just high-risk for SEO. Getting my own personal wiki to be indexed was such a challenge that I'd just about given up when a friend who is more acquainted with the whole thing helped me make sure I had all the bits and bobs in the right place. If you're not careful, people can easily put spam all over your site and then it'll really ruin your presence on a search engine.
Google is really big, though. Really really big. They're so big that not even all the people inside Google are trustworthy to them on a subject like this.
But they don't universally hate wikis and so on. It's just you have to do a lot of work and make sure you don't have spam on your wiki, and then fill in all of the information in your meta tags, and have a sitemap.xml, and all that. Here's my wiki for example: https://wiki.roshangeorge.dev/w/images/8/89/Screenshot_-_Goo...
Tangential: this would be an excellent time for the Kagi folks to do a Mullvad style campaign around "Remember Google before the apocalypse? Meet Kagi."
After yesterday's keynote and the changes to Search, it became clear in the near future, Google will cease to direct any traffic to websites and the search results will just become a footnote in Gemini's response.
Interesting. My small game wiki was also affected ~3 weeks ago. It doesn't even show up on Google anymore even if you directly search for the URL.
We don't get any spam since there's no public signups for editing access.
interesting indeed. very short sighted by google if so. before, the incentive for publishing on google was discovery... if that goes away, then what?
Google does not hate you, they simply do not care about you at all. There is a very minor difference
I'm not sure about that, they could do far worse than delist you if they really hated you.
I still have 42k page indexed, but previously I had 20k impressions per day, past week impressions started to change. And now I have 399 impressions per 24 hours :/
Content creators need to accept that traditional SEO is a thing of the past, because traditional search engines are a thing of the past. None of my normie friends use search engines any longer. They just ask the AI - anything and everything - the AI has all the answers they need. The best that any content creators can hope for in terms of engagement is that they were the quoted source and the user cares enough to check sources. Content creators just need to find new ways of driving engagement now, and we're never going back so there's no use crying about it.
Well LLMs are trained on text influenced by SEO, and they use search tools also weighted by SEO. So SEO is not irrelevant I guess ?
Public pressure campaigns work on companies, and this situation is thanks to Google.
I suspect this is a cloudflare thing since the other search engines are doing fine. I'd look closer into your cloudflare settings and see what you can relax.
oof that sucks; i really wish there was more info on why google decides to crawl or not crawl a page
They sucked out all the content and then pulled the plug.
Why not use startpage.com which is doing anonymous searches in google?
default vivaldi engine. no complaints.
Google's always adjusting its search rankings, but it's rare for a legitimate site to suffer such a sudden massive hit without reason.
My first thought would be that they accidentally blocked Google's crawler (maybe through some kind of anti-AI setting?) or that Google believes that the site is serving malware or spam. Either scenario can have that kind of effect. I can see that their forum at least appears to have strong Cloudflare anti-bot rules in place, so that might be the case.
They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
But aside from that usual stuff, there's one more possible reason that's specific to this site. In November of last year, the Pokemon Company rebranded their "Pokemon Trainer Club" to "Pokemon Trainer Central", which is the first result that comes up when you search for "Pokemon Central".
That change was made a few months before the sudden drop in traffic, but could still be a viable explanation here. Google does routine re-ranking on a daily basis along with occasional major re-ranking, which happens maybe a few times a year, so the delayed hit that they saw could have come from Google finally recognizing that most people who search for "Pokemon Central" are no longer looking for the wiki like was once true in the past.
https://gonintendo.com/contents/54863-pokemon-trainer-club-r...
A blocked Googlebot would’ve caused a report of 403s in the Search Console, which isn’t the case. And the subdomain has worked perfectly fine for the last 15 years.
You may have a point with the Trainer Central rebranding, but please consider this in the context of Italian language results. It’s not about reaching the home of the wiki (which is pretty much the only page that’s still indexed), it’s all the other search queries (Pokémon names, moves, games, etc - without adding “pokemon central” even) that usually returned our wiki’s dedicated page as first result (or top 5 at least) and now those specific pages are not even indexed anymore.
> They're also using a subdomain for both their wiki and forum, which Google has been observed to punish. They might consider moving each of those to their own separate .com domain.
Any sources for this? AFAIK, Google treats websites on a subdomain as a separate entity.
https://developers.google.com/search/docs/fundamentals/seo-s...
> Things we believe you shouldn't focus on: As SEO has evolved, so have the ideas and practices (and at times, misconceptions) related to it. What was considered best practice or top priority in the past may no longer be relevant or effective due to the way search engines (and the internet) have developed over time.
> Subdomains versus subdirectories: From a business point of view, do whatever makes sense for your business. For example, it might be easier to manage the site if it's segmented by subdirectories, but other times it might make sense to partition topics into subdomains, depending on your site's topic or industry.
Doesn't quite explicitly say it treats them the same, but it kinda implies it.
Google have also blatantly lied in the past about SEO. My familiarity with the subject is about a decade old, but I don't expect Google to have improved in that regard, quite the contrary. I don't think you can take anything in their guidelines as 'truth that will help for SEO'.
It appears for me when I search for it. Even Gemini is cool with looking for it.
Here is a part of the Gemini result I got which was directly above the regular result link.
"Pokémon Central is a major community network and independent Italian encyclopedia for everything Pokémon-related"
Honestly, the title is super clickbait and it doesn't even reflect reality. Its so easy picking some giant entity far away and create some drama about it. Dont get me wrong, I am not a google fan, but I also dislike clickbaits and whiney dramatic claims, moreover if unverified.
Maybe the claims in the Twitter thread weren’t clear enough; I’ve added some clarification especially for you: https://x.com/pokemoncentral/status/2057157781611024764
Thanks for the clarification, the title is still a whiney clickbait.
Do you suspect they have faked the search console screenshots as well? Make your accusation explicit.
My guess is that the combination of Wiki and Pokemon is highly suspect for Google.
The Pokemon Industrial Complex has advanced astroturfing especially on YouTube/Twitch, where streamers mention the damn things in any second episode, they "accidentally" meet people going to Pokemon conventions in live streams and so on.
Try to audit the Wiki if anyone abused it.
One of the sad things about this story is that everyone has to read tea leaves to guess what reasoning might be going on at Google's end. Tech companies have normalized the practice of cutting people off with no explanation, or saying 'we investigated and found a violation' without articulating what the violation is. Naturally, they want to secure themselves against abuse and people trying to game their system, but refusing to provide any information does not achieve that.
It does infuriate legitimate users, enables other kind of abuse and scamming (eg immunize yourself against delisting with this one weird trick!', link farming etc), and act as a fig leaf for abusive behavior by platform operators. Effectively, we've allowed large teach companies to act as digital dictatorships with no accountability to their customers. Yes I consider users to be 'customers' even if they're uploading content or doing searches 'for free'. If you're monetizing their activity on your platform, they are your customers whether or not you call them that to avoid legal liability.
They're a private company, they can ban whoever they want.
Or at least that's what I heard a few years ago when it was politically incorrect people complaining about being banned with no accountability. They're a private company, it's their servers. You may not even be paying anything. So they can do anything they want to you and you have no cause for complaint.
https://xcancel.com/pokemoncentral/status/205712380740463825...
Thank you
This might be useful: https://addons.mozilla.org/fr/firefox/addon/toxcancel (Redirects to xcancel.com (a mirror of x) when the browser is about to load an x.com page).
Easily a legal request.
Can someone start a new Google, please? Just search, nothing more. I'm willing to pay 10 USD a month for that. API access included.
https://kagi.com/pricing
I think it's become increasingly clear that search engines, in their most basic form, aren't cut out for the internet of today. The open internet is basically dead and I see the world returning to old yahoo-like web page indexes that are manually verified and sorted
Kagi has been mentioned already, just to provide anecdotal reference: searching for "pokemon wiki" with the country set to Italy shows OP's website as first result.
Yeah Kagi already exists luckily, it’s extremely good and worth the money.
Now? Google has hated us since at least the DoubleClick aquistion in 2008. That's when people became the product
Perhaps they're decommissioning search in favor of LLM:s.
That's only supposed to happen later this week.
This aligns with their Google Zero doctrine, keep all info internal and make the goal for the user to hit 0 external websites.
Pokemon Central runs ads (Google AdSense at that!), which is probably how they pay for everything.
Google is likely their biggest inbound source of traffic, so they're probably experiencing a marked revenue drop as well.
It's unfortunate that so many livelihoods are subject to the capricious whims of a single company. A company that is increasingly seeking to keep users on their engine without sending eyeballs or revenue to any third parties at all.
We're watching Google's "embrace-extend-extinguish" arc for the web. It's not over by a long shot, but they absolutely intend to finish the job.
Hi EU. How about one of those lovely anti-trust cases?
I really hope eu can extinguish Google before they extinguish all websites. Will be an exponential death of website very soon once they lose traffic.
I'm worried about the Trump/Tech-Oligarch axis interfering with legal processes in the EU and preventing needed regulation.
We should've gotten out of US dependence decades ago.
It’s why I moved to in-house advertising. It’s a lot of work, but I hope it is the right decision.
Why is it a lot of work? Could you specify some off the more difficult effort? Wouldn’t LLMs help speed this up? This is the one area where I’d think Llms could really take Google down by empowering in house ad platforms.
Depends on how in-house you want to go. If you go full in-house, you'll need sales staff to make deals with advertisers. You'll then need a way of hosting the media provided. You'll need a way to deal with media that does not match what you've requested. You'll need a system to allocate ad space accordingly to contracts with ad clients. It's like a whole new department in your company.
I don’t have a sales staff. I just call a company, tell them why I’m calling and what’s the opportunity, crack some jokes, get serious and make them an offer. It helps that I’m in a niche, thoroughly know the sector and that they most likely already know my websites. As long as I can get to the owner of the company, then I’m golden.
Spiders Georg runs his company from a cave and buys 10,000 spiders every day; he is an outlier and should not be counted.
Its better off if ads go away. Just use ad blockers.
We would like the wiki to be free of ads, but hosting costs at our scale are real. Since we don’t like ads either, we compromise like this: users can register for free and never see an ad (they are only served to anonymous visitors); they can also use an ad blocker and we won’t bug them about it.
there are many laudable places such as yours, most auspiscious, deserving of reparations, alas such favourable anachronisms are unduly burdened .
i like to mail a fiver or two to a P.O.B.
Title could be: Apparently Google hates Pokémon Central Wiki now
(to be clearer what the source of the post is)
No, I think "us" is apt, considering this will eventually affect all sites that rely on traffic from Google search, which is basically every text heavy site.
All we can hope for is that people will stop using search (after eventually having enough of the AI wave) for these sort of niche sites and will bookmark and access them directly in future. I don't have much hope.
I spend a lot of time wondering about the true role of search in people's lives in 2026. When I watch people use the Internet, it seems like most of them perform searches simply as fuzzy-matching navigation to the websites they use. Like the way many people use Spotlight to launch desktop apps.
I think this is because
(A) bookmarks lists are inconvenient - scrolling to find a bookmark is slower than typing "youtube" or (cringe) "bank of america" in the URL bar
(B) typing URLs directly requires precision of memory with TLDs being numerous and even things that were once predictable are now mere suggestions (e.g. is your city or town at cityofwhatever.com? city.org? city.gov? Could be anything!)
(C) related to (B) if you screw up a full URL you may well end up at a phishing site that looks like the site you wanted.
I really believe that 90% of Google and Bing searches today are probably for the names (or misspelled or partial names) of the top 100 websites.
If the dominant browsers weren't Google Chrome and Mobile Safari (who gets paid by Google for every search) browsers would build bookmarks for you of your frequently-used sites, and ordered by frequency of visits, present those for direct navigation when you type a word in the search bar, and not send any query to a search engine if you chose one of those. But all incentives point very strongly against doing that and toward sending you to a SERP with 13 ads and an "AI Overview" above the organic results.
A wiki with only 11 pages?
Perhaps they will investigate why 541,000 pages aren’t being indexed. In my experience, Google provides adequate tools for identifying and resolving indexing issues.
Google won’t serve pages it hasn’t indexed. Seems they left a lot of relevant details out of that tweet.
Edit: and the most likely answer would be that their current robots.txt disallows virtually all indexing. I’m no SEO expert but entries like this seem like footguns:
Edit 2: there’s more info in the full thread but that was only viewable via the xcancel link someone else shared (despite having the X app installed - deeplinks don’t work today). A helpful example of why X is not the best platform for sharing multi-post threads. Seems robots.txt was considered but ruled out.In the first image you can see how indexed pages go from 40k+ to 11 in the matter of days. Further down the thread I show how 114k+ pages are marked as “crawled but not index” and we can’t understand why. The rest is stuff that is (correctly) blocked by robots.txt.