Starting on November 16, 2026, conversations in Discoverable Servers will be easier to find for people searching the web. Admins can choose which channels are surfaced on the web. Private servers, DMs, and GDMs are not included.
A lot of article websites whitelist Google’s crawler, right? Can’t that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though
The bots will pretend to be search engine crawlers. And not all have their own IP ranges.
You can’t impersonate search engine crawlers, because the server can verify that it’s really a search engine crawler
See this from google: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
Here, duckduckgo: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot
A lot of article websites whitelist Google’s crawler, right? Can’t that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though