Detecting and preventing SYN Flood attacks on web servers running Linux

The other day I helped a client deal with a SYN flood denial of service attack. This article describes the symptoms, diagnosis and solution from a Linux server point of view.

Definition

A SYN flood attack exploits one of the properties of the TCP/IP protocol: by sending SYN requests, and then never following up with an ACK, this leaves the server using one network "slot" and waiting for the other side for some time. Doing this many times ties up network resources and the server becomes unresponsive.

Symptom: web site visitor

The symptom to an end user, a web site visitor, is that a site takes a long time to load, or loads some elements of a page but not others. The general impression is that the site is slow or down.

Symptom: on the server

From a server administrator perspective, you will not see high CPU utilization, nor memory utilization. The server will be lightly loaded when this happens.

In order to check whether your server is under a SYN flood attack, you use the wget command times how much it takes for your site to load, from the same box (to eliminate the network as a cause).

For a normal site, you get something like this instantly:

$ time wget -O /dev/null www.example.com
Resolving www.example.com... 1.1.1.1
Connecting to www.example.com|1.1.1.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
...
real 0m0.018s
The request takes just 18 milliseconds, which is quite normal for a well configured server.

Compare that to when a SYN flood attack was active:

$ time wget -O /dev/null www.example.com
Resolving www.example.com... 1.1.1.1
Connecting to www.example.com|1.1.1.1|:80... connected.
And this is where things hang for a long time ... The server takes a very long time to open a network socket.

Then after a long time, you would get this:

HTTP request sent, awaiting response... 200 OK
...
real 0m45.002s
Ouch! 45 millisconds for the home page to load locally!

This can vary from a few seconds, to several minutes, and running it several times will give different results, but all or most of them will be over 1 or 2 seconds.

Diagnosis

Then when you look at netstat's output, you find that there are lots of connections in the SYN_RECV state:

netstat -tuna | grep :80 | grep SYN_RECV
The output will look like this:

tcp 0 0 1.1.1.1:80 70.56.83.204:1609 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1723 SYN_RECV
tcp 0 0 1.1.1.1:80 209.112.192.126:4988 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1724 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1727 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1733 SYN_RECV
tcp 0 0 1.1.1.1:80 24.158.121.0:3337 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1753 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1811 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1821 SYN_RECV
tcp 0 0 1.1.1.1:80 2.2.2.2:1831 SYN_RECV
tcp 0 0 1.1.1.1:80 24.7.27.61:52142 SYN_RECV
tcp 0 0 1.1.1.1:80 207.118.0.58:50819 SYN_RECV
tcp 0 0 1.1.1.1:80 115.64.40.38:52865 SYN_RECV
You will see a lot of SYN requests from the same addresses. Do not bother with tracing what this address is, because it is easily faked, and the attacker is probably using fake addresses.

Solution

So, how do you solve this?

The solution varies, but the best one is to enable SYN cookies on your load balancer or the server itself.

To enable that on a current Linux kernel, you enter the following command:

sysctl -w net.ipv4.tcp_syncookies=1
And then add the following line to the /etc/sysctl.conf file to make make it persist across reboots:

net.ipv4.tcp_syncookies = 1
You may optionally want to increase the size of the SYN backlog queue as well, from a default of 1024, to 2048, using the following command:

sysctl -w net.ipv4.tcp_max_syn_backlog=2048
And you add this to /etc/sysctl.conf:

net.ipv4.tcp_max_syn_backlog = 2048
Results

After you do the above, SYN Flood attacks will continue, but it will not affect the server negatively. You will see a message like this in your logs:

[1116377.589736] possible SYN flooding on port 80. Sending cookies.
[1116439.567828] possible SYN flooding on port 80. Sending cookies.
[1116500.631623] possible SYN flooding on port 80. Sending cookies.
But there will be no ill effect from it ...

And there you have it, attackers will still send SYN requests, but the server will remain responsive ...

Resources

For more reading on this issue, check the following links:

RFC 4987: TCP SYN Flooding Attacks and Common Mitigations
http://www.securityfocus.com/infocus/1729
http://www.iss.net/security_center/advice/Exploits/TCP/SYN_flood/default.htm
http://www.usenix.org/events/sec01/invitedtalks/oliver.pdf, dated, and hence doesn't cover SYN cookies, but describes the problem well.
http://www.networkcomputing.com/unixworld/security/004/004.txt.html, dated as well.