The rise and fall of robots.txt
www.theverge.com
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
    • T156@lemmy.worldEnglish
      14·
      9 months ago

      Except that it’d also catch out people who use accessibility devices might see the link anyways, or use the keyboard to navigate a site instead of a mouse.

      • HACKthePRISONS@kolektiva.social
        54·
        9 months ago

        i don’t know, maybe there’s a canvas trick. i’m not a webdev so i am a bit out of my depth and mostly guessing and remembering 20-year-old technology

    • oatscoop@midwest.socialEnglish
      71·
      9 months ago

      If it weren’t so difficult and require so much effort, I’d rather poison their data.

      Clicking the link causes the server to switch to page content designed to fuck with a LLM.

      • HelloHotel@lemm.eeEnglish
        3·
        9 months ago

        Visiting /enter_spoopmode.html will choose a theme and mangle the text for any page you next go to accordingly (think search&replace with swear words or santa clause)

        It will also show a banner letting the user know they are in spoop mode, with a javascript button to exit the mode, where the AJAX request URL is ofuscated (think base64) The banner is at the bottom of the html document (not nesisarly the screen itself) and/or inside unusual/normally ignored tags. <script type="spoop/text" style='display:block">you are in spoop mode</script>

        Or have a secret second page that is only followed if you ignore robots.txt /spoop_post/yvlhcigcigc is a clone of /post/yvlhcigcigc in ‘spoop mode’

      • T156@lemmy.worldEnglish
        2·
        9 months ago

        Would that be effective? A lot of poisoning seems targeted to a specific version of an LLM, rather than being general.

        Like how the image poisoning programs only work for some image generators and not others.