How crawlers impact the operations of the Wikimedia projects https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/ #AI, #Crawlers, #Infrastructure, #KnowledgeAsAService, #KnowledgeContent, #Operations, #Scraping, #ScrapingBots, #TechNews, #Traffic, #WikimediaFoundation, #WikimediaFoundation, #WikimediaProjects

Last week, Wikimedia reported that AI bots saturated their available bandwidth. Here's why the bad bots are getting so much worse...

https://www.plagiarismtoday.com/2025/04/10/the-battle-against-the-bots/

Introducing yet another "groundbreaking" tool to scrape the wisdom of armchair tech experts on Hacker News! Because obviously, the internet desperately needs more regurgitated opinions delivered by #AI.
https://github.com/simonw/llm-hacker-news #HackerNews #Scraping #TechOpinions #Groundbreaking #Tools #RegurgitatedContent #HackerNews #ngated

KI-Verordnung: verbotene Praktiken und die DSGVO

Die Umsetzung der KI-Verordnung wird von der Europäischen Kommission vorangetrieben. Ein Schwerpunkt liegt dabei auf den Praktiken, die nach der KI-Verordnung verboten sind. Zum Teil bietet auch die DSGVO bereits Schutz vor solchen Praktiken. Der Beitrag geht hierauf anhand eines Beispiels genauer e(...)
https://www.dr-datenschutz.de/ki-verordnung-verbotene-praktiken-und-die-dsgvo/

Managing AI Bots+ w/ Apache MPM, FPM, & Fail2Ban: https://tech.haacksnetworking.org/2025/04/06/managing-ai-bots-w-apache-mpm-fpm-fail2ban/ There's been a lot of continued discussion on this topic, so I decided to investigate some of the common reports, compare those to my own hardware, theoretical ceilings and caps, and then adjusted my LAMP stack and fail2ban as per this blog entry. Let me know what yall think or if you find any errors or questionable claims. -oemb1905 #ai #scraping #apache #opensource #freesoftware #floss #ddos #php

The Markup: A Guide on How to Legally Web Scrape EU Data. “At The Markup, some of our data journalists recently had questions about the legal risks involved in scraping websites hosted in the European Union. We conducted our own research to answer this question, and offer a summary of what we learned below. Our goal is to help other journalists, researchers, and advocates come up with a […]

https://rbfirehose.com/2025/04/06/the-markup-a-guide-on-how-to-legally-web-scrape-eu-data/

Unpopular opinion: copyright was also violated when somebody put a didigtized Studio Ghibli image on the internet. To begin with.
#scraping #AI #ghibli

Nice nice take https://blog.sysopscafe.com/posts/ai-crawlers-hammering-git-repos/ ty to @p4p4j0hn for sharing! #ai #scraping #floss #freesoftware #opensource

Web Scraping With Cheerio in 2025, by @apify.bsky.social:

https://blog.apify.com/web-scraping-with-cheerio/

With all the talk about AI-scraping, I decided to run the numbers on my little wiki and tech blog. Here's what we got. https://haacksnetworking.org/bot-scrape-04-04-25.txt #AI #bot #scraping

**#Scraping of #LLM ’s explained:**
* take a company that lives off of answering people’s questions, e.g. WikiHow
* take all WikiHow’s guides and turn them into an answerbot.
* make money by providing the answers to WikiHow users with your chatbot
* claim that you are not a competitor to WikiHow, and your use of their entire content library is #FairUse
* repeat with the entire internet

I've set up my new #inkscape website AI bot trap. It works by giving everyone a chance to not fall into it.

An anchor link that says "I am a bot" and links to /P3W-451/{datetime}/ it's got a fixed position at top -100px so should never be seen

The robots.txt says "Disallow: /P3W-451/" so if you were reading the robots, you'd know.

Then #nginx logs the requests to a log of their ip-addresses and browser strings and sends them a 301 redirect to google.com

#ai #Scraping

1/2

Joshua Yuvaraj, co-director of the New Zealand Centre for Intellectual Property, was interviewed on RNZ yesterday, about the degree to which copyright law might be used to prevent scraping of the open web by #MOLE Trainers;

https://www.rnz.co.nz/national/programmes/nights/audio/2018981590/what-can-writers-do-about-their-work-being-used-to-train-ai-models

As Cory Doctorow noted back in 2023;

"In privacy and labor fights, copyright is a clumsy tool at best."

https://pluralistic.net/2023/09/17/how-to-think-about-scraping/