Just a Rant: Moral Hazard for Good (TCP Tuning as an Example)

MORAL HAZARD FOR GOOD: I say this and people think I'm summoning flying monkeys with horns.

The simplified formula for the successful delivery rate of a SYN/ACK depending on the loss rate (assuming the loss is the same for send and receive) is:

n_syns * n_synacks * (loss^2)

With a loss rate of 0.5 (and delivery rate of 0.5) 2 SYNs are sent and 1 SYN is received and 2 SYN/ACKs are attempted and 1 gets through:

2 * 2 *0.25 == 1

The default on this Linux box is 5 SYNACK attempts for every SYN received:

# cat /proc/sys/net/ipv4/tcp_synack_retries

5

That's 5x amplification during a SYN reflection attack!

Like I've said elsewhere, I've set it as low as 2 in prod, and I would consider 1. (People looked at me like the flying monkeys were gonna arrive at any moment, but with a little whistling we got past the graveyard.) The attacker has a geometric advantage. Emotionally thinking that you're gonna increase your SYNACKs for good is a fool's errand that only hurts you and some anonymous victim: you have to /geometrically/ increase it to offset the attacker. Thinking this way is the usual outcome of moral hazard.

Moral hazard for good is embracing hostility and reducing SYNACK retries to something like 2... if not 1... and assuming that somebody who really wants to connect will selfishly up their initial SYN attempts correspondingly. So the sender is prepared to send up to 4 SYNs losing 2 to attrition (and if they send them all at once without backing off or waiting for responses then they're jackasses and should be dealt with accordingly), and the SYNACK sender sends 2 losing 1 to attrition and the result is the same. So sad for that client having to send 4 SYNs instead of 2, my heart is breaking at the cruelty of teh netz!

It's that simple: moral hazard can be a force for good and you owe teh netz nothing. There's no mathematical reason to spread the risk around; all it does in practice in the face of an adversary is to cause more damage and spread that around.

If there is a question lurking here, it's whether tcp_synack_retries is for each SYN successfully received or whether it's for the connection. If I find some free time maybe I'll set up a VM and use Scapy to fire some SYNs at it.

  • People looked at me like the flying monkeys were gonna arrive at any moment

    It might help to make a diagram for them using load testing and spoofing tools. Then attack a stand-alone version of your company code and show the differences before and after.

    For what it's worth, I have always set synack retries to 0 on my web, smtp and dns servers. I leave it up to the client applications to retry. This is just my hobby crud but I agree 5 is a risky default that can amplify attacks. I have never run into problems setting it to 0 but I do not have the most all-inclusive set of applications to test. Most companies put their applications behind a load balancer that has some SYN protection enabled by default and that is probably why it's less of an issue.

    As a side note I have also found that some of the syn-spoofing-flooding tools lack some TCP options that are very easy to filter within iptables modules such as missing or odd values for MSS in syn packets, odd tcp syn lengths, etc... One example from my raw table:

        -A PREROUTING -i eth0 -p tcp -m tcp --syn -m tcpmss ! --mss 1220:1460 -j DROP
    
    Most home connections will be 1460. Some Cisco VPN's are 1454 or 1424. Cell phones using ipv6ipv4 gateways are sometimes 1280 or 1380. There are other less common variations such as people using jumbo frames ranging anywhere from 9036 to obscenely high numbers. The most common scanning tool "masscan" does not even set MSS at all. Same for hping3 and a few other tools.

    Another obscure one in the raw table I use when I do not need or expect cell phones to hit a port and this happens because most cell phones enable TCP timestamps which usually bumps their TCP SYN length to 60. Why cell phones? Modern bot farms seem to be using many old cell phones to launch their attacks and I have no idea why.

        -A PREROUTING -i eth0 -p tcp -m tcp --dport 1:442 --syn -m length --length 58: -j DROP
    
    As a more obscure note, there was a time I increased syn/synack retries beyond the default and that is when we used NFS Diskless in our production datacenters. But that's a much longer story and a much older kernel.

  • I'm not super familiar with Linux's syncookies, but if being used as for a reflection attack, you're probably into syncookie territory, and that gets you down to one syn/ack per syn anyway.

    I guess if the attacker is sending a low number of SYNs to get the reflection, you could still hit the the 5 retries (is that actually 5 SYN/ACKs or the original + 5 retries?). But then again, if it's a low number, does it make a big difference? Maybe, but there's a lot bigger reflection vectors out there.