• AppleMango@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    3 days ago

    A lot of article websites whitelist Google’s crawler, right? Can’t that be done on a domain-basis? Or maybe have some kind of authentication-based system 🤔, I guess that would require the crawlers to be refactored though