Gonna add a bunch of the LLM bots to robots.txt--I know many of the big players just ignore robots.txt and fudge their UAs, but maybe it'll make a little dent. Fully 2% of our requests are ByteDance, and 5% are ahrefs--both of those should be blockable.

No idea what to do about what I suspect is residential proxy traffic, which makes up the vast majority of our load. I assume throwing Anubis in front of a Mastodon instance is going to break a ton of legitimate use cases.

@aphyr we built out a trap for scanners: LLM bots will crawl the CT logs, so anyone making HTTP requests to the certs requested by our mail servers are fuckin' around and can go directly into the firewall.

@aphyr our version is for freebsd blocklistd, but the idea ought to be a weekends worth of implementation with other firewall rule engines https://fossil.se30.xyz/ratrap