Woof.group's been getting slower and slower lately, and I think it's cuz we're getting hammered by (maybe LLM?) scrapers. Hard to say, really, but don't buy that there are *that* many Windows/Chrome users clicking around through every single tag page.
Gonna add a bunch of the LLM bots to robots.txt--I know many of the big players just ignore robots.txt and fudge their UAs, but maybe it'll make a little dent. Fully 2% of our requests are ByteDance, and 5% are ahrefs--both of those should be blockable.
No idea what to do about what I suspect is residential proxy traffic, which makes up the vast majority of our load. I assume throwing Anubis in front of a Mastodon instance is going to break a ton of legitimate use cases.
@aphyr we built out a trap for scanners: LLM bots will crawl the CT logs, so anyone making HTTP requests to the certs requested by our mail servers are fuckin' around and can go directly into the firewall.
@aphyr our version is for freebsd blocklistd, but the idea ought to be a weekends worth of implementation with other firewall rule engines https://fossil.se30.xyz/ratrap