Protecting HAProxy-ingress with iocaine

Mon, 21 Sep 2026 00:00:00 +0100

Whatever your views on LLMs/"AI" are - if you're running a service that's exposed to the public internet, you're getting bombarded with badly behaved scrapers trying to slurp up everything they can.

In ye olden days most scrapers at least somewhat respected a well written robots.txt but now, they just go wild on whatever link they can find. While you can use something like anubis to ward of many bots, this comes at the cost of making visitors waste energy by performing Proof-of-Work calculations which disadvantage low-end devices and are not a real deterrent for big scraping efforts as long as money is still getting thrown at anything that has a .ai domain.

security on dominik.suess.wtf

Protecting HAProxy-ingress with iocaine