Appoxo@lemmy.dbzer0.com to Technology@lemmy.worldEnglish · 13 hours agoCloudflare plans marketplace to sell permission to scrape websitestechcrunch.comexternal-linkmessage-square10fedilinkarrow-up1132arrow-down13cross-posted to: technology@lemmy.ml
arrow-up1129arrow-down1external-linkCloudflare plans marketplace to sell permission to scrape websitestechcrunch.comAppoxo@lemmy.dbzer0.com to Technology@lemmy.worldEnglish · 13 hours agomessage-square10fedilinkcross-posted to: technology@lemmy.ml
minus-squaregaylord_fartmaster@lemmy.worldlinkfedilinkEnglisharrow-up40·12 hours agoThey’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent.
They’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent.