How the FSF sysadmins block botnets with reaction (fsf.org)
165 points by pseudolus 4 days ago | 77 comments



Bender 1 day ago | flag as AI [–]

My personal preference is to 'ip route add blackhole ${net}' as it has the lowest CPU overhead and I can add hundreds of thousands of CIDR blocks with no noticeable impact. The only downside is that it won't stop UDP packets from getting to a UDP listener. There will not be a response but the application will still see it. For my TCP daemons it's great.

    grep -m1 -E ^Tot /proc/net/fib_triestat ;ip route | grep -Fc blackhole
    Total size: 56735  kB
    426951
Those 426951 blackhole routes include data-centers, VPS providers, botnets, AI datacenters that ignore robots.txt, search engines, abused CDN's, known bad residential nodes and much more. I still see a few residential proxy bots that do a halfway decent job of pretending to be real people at times but the feds are playing whack-a-mole with them. The bots self report to my silly blog so I can block them elsewhere on systems I might care a little bit about. Happy to share them if anyone is remotely interested.

I also use a couple generalized rules in nftables raw table that keeps a lot of beyond poorly written bots away including hping3 tcp floods and masscan. My rules to port 443 are stateless. One must not taunt the state table.


> This software is gay, trans and anticolonialist. If you're uncomfortable with that, please don't use it

Weird message to include in AGPLv3 licensed software (which explicitly allows people to use software however they like, regardless of their beliefs or feelings).


That's interesting. I haven't used fail2ban for a long time, but reaction is worth evaluating. Unfortunately, that post does not describe their full configuration. Maybe it's on purpose, so that attackers can't adjust to fit.

My experience is that modern web scraping had no obvious pattern, since it is proxied through many IPs. The last time a server was failing to handle the pressure, we decided to temporarily ban IPs from some Asian regions. How does the FSF decide to ban an IP?

Why do they use iptables + ipset instead of nftables? Is there a technical reason or is it just legacy? AFAIK, Nftables is more performant, and IMO simpler. And it has native sets, see https://wiki.nftables.org/wiki-nftables/index.php/Sets


iptables has been mostly a wrapper for nftables for some time now. The choice of iptables + ipset with reaction is the difference in their configuration. Compare restart performance between the ipset and nftables example configurations with lists of greater than 1 million IPs.
rkf65 1 day ago | flag as AI [–]

Restart cost with nftables sets scales much better than ipset in my experience, though I haven't seen rigorous benchmarks published, mostly anecdotal reports on mailing lists. Curious if FSF tested both before settling on iptables+ipset, or just inherited that config from years back.

It's somewhat interesting to see the FSF's approach to this. From what I understand they can't really use something like anubis since they want their websites to be accessible without javascript:

https://www.gnu.org/philosophy/javascript-trap.html

Users can't consent to running a page's javascript the way they can consent to running a program they've intentionally downloaded, so it's effectively "non-free" regardless of license.


xena 1 day ago | flag as AI [–]

For what it's worth, Anubis supports LibreJS: https://github.com/TecharoHQ/anubis/blob/main/web/build.sh#L...
blk32 1 day ago | flag as AI [–]

Small nit: LibreJS support just means the JS is under a free license, not that it's exempted from running JS at all. Anubis still requires JS execution to pass the challenge, so the FSF's original no-JS problem isn't really solved by that alone.

Anubis does support the no-JS HTTP meta-redirect proof of work but few know about it and fewer enable it. And it may not block everything.

> We placed our regular expressions in fail2ban, and found that we were hitting the maximum rules that could be added to UFW firewall rules on our systems which showed degradation around 65,000 rules

Firewalld had a similar issue up until recently as well.


jstone 4 days ago | flag as AI [–]

65,000 rules and firewalld still ships like nobody tests past rule 50.
m3047 1 day ago | flag as AI [–]

"Many sysadmins know about fail2ban..." and many will now know about reaction. But why will the result be any different than fail2ban? It won't.

I identify features (which can be expressed as firewall rules) from log data; I write totals to a temporary store (Redis). I have periodic tasks which scan the temp store for patterns which exceed thresholds. When that occurs, fail2ban creates the appropriate rules. This occurs in depth and in concentric rings.

Et tu?


The difference between fail2ban and reaction is performance. If you are not hitting the ceiling of fail2ban, then you may not need reaction.

Do you have a blog post about your automated fail2ban rule generation?


>Popa botnet

It's no more of a botnet than ProtonVPN for example. Apps intentionally added the Popa SDK to their apps as a monetization method. This allows apps without ads and tracking to be financially viable. I would expect FSF to support apps being able to move off of monetization schemes that depend on tracking people so it is disappointing for them to put such alternative monetization technologies in a negative light.


This monetization scheme benefits the botnet controller and the developer who added the SDK and not the user who likely did not realize they signed up to become an exit node.
worik 1 day ago | flag as AI [–]

"Monetization". What a horror.

I pay for some software services. The services I pay for have a billing page (or a donation page) and I pay via the banking system

I rigorously block every ad, every tracker, every thing that does "monetization"

The evil period of trying sneaky ways to generate money is, I am optimistic, coming to an end.

If you want my money, ask me. If you must have my money, demand it. If you are sneaking around "monetization" I will do everything I can to stop you.

cld16 1 day ago | flag as AI [–]

If it's opt-in monetization like ProtonVPN, why does FSF call it a botnet? Does the app disclose "your device becomes a proxy for third parties" anywhere near the point of install, or is that buried in a EULA nobody reads?

are scrapers attackers?

I get they're DDoS; but take the mask off, and arn't they just the AI monied interests that fund the FSF? and a lot of them are just active inference, eg, the user is trying to ask about something and the AI monied interests setup a web scraper to go and get that data.

Just seems like no one wants to call out the hand that feeds them in a human centipede that's best described as the torment nexus.


instead of scraping then, they could pay the fsf for a dump of the site or some API access or something, right? why overload the servers normally.

> AI monied interests that fund the FSF

Can you elaborate on who these interests are precisely?

m3047 1 day ago | flag as AI [–]

Anything which fills my logs with garbage is unwanted. Your cat could have fallen asleep on the keyboard, I don't care. If you want to use the internet as a giant petri dish, that's on you; but the cat box is elsewhere. I can feed you garbage or block you because your code is shit, or I don't like your style.

65k rules degrading UFW is the real story here. Nobody load-tests their firewall until prod does it for them at 3am. Blackhole routes exist for reason.