+- Scivillage.com Casual Discussion Science Forum (https://www.scivillage.com)
+-- Forum: Site Related (https://www.scivillage.com/forum-119.html)
+--- Forum: Bots-Spiders-Scrapers (https://www.scivillage.com/forum-138.html)
+--- Thread: 365+ Bots-Spiders-Scrapers (/thread-19617.html)
Bots-Spiders-Scrapers - stryder - Jan 16, 2026
If you are a the owner/developer of a bot/agent or AI,
Please consider that if you abuse sites with over-scrapeing it doesn't just jeopardise your access to this one site.
Poor ettiquette can lead to poorly trained models since they likely find themselves with partial or no data at all.
I'd advise using or developing to be shared a read-only mirror server to scrape from, as this would lower site traffic from continual scrapeing.