TCP timers and keepalives

Netstat -o includes tcp timers which are useful for Apache keepalive analysis.

netstat -ntpo | grep ESTAB | egrep ":80|:443"

Output looks like this

tcp  0  0  98.xxx.xxx.106:80  83.138.139.7:41089   ESTAB  25812/httpd  keepalive  (7196.71/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.7:41088   ESTAB  19385/httpd  keepalive  (7196.71/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.45:41090  ESTAB  26995/httpd  keepalive  (7196.71/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.37:41087  ESTAB  25813/httpd  keepalive  (7196.70/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.7:41088   ESTAB  19385/httpd  keepalive  (7196.71/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.27:41090  ESTAB  26995/httpd  on         (7196.71/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.17:41088  ESTABLISHED  19385/httpd  off        (7196.71/0/0)
tcp  0  0  98.xxx.xxx.106:80  83.138.139.17:41090  ESTABLISHED  26995/httpd  off        (7196.71/0/0)

The last column denotes what the connection is doing.

'on' - Actively transfering data.

'off' - Currently disconnecting

'keepalive' - Connections are using TCP keepalives. The first number denotes the time in seconds from when the last data was transferred until when the next TCP keepalive probe will be sent. By default this starts at 7200s, and resets again every time more data is sent. If the value is low , for eg. 4000 seconds , it means some of the keep alive connections are hanging or doing nothing for a long period. Note, connections to internal proxy or other internal processes might hang longer but this should not happen to web based connection.

The defaults are dictated by the sysctl values :-

''net.ipv4.tcp_keepalive_probes'' - How many keepalive probes TCP sends out, until it decides that the connection is broken. Default value: 9.

''net.ipv4.tcp_keepalive_time'' - How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours (7200 seconds)

''net.ipv4.tcp_keepalive_intvl'' - How frequently the probes are send out. Multiplied by tcp_keepalive_probes it is time to kill not responding connection, after probes started. Default value: 75sec i.e. connection will be aborted after ~11 minutes of retries.

More information on the sysctly values can be found in the kernel documentation ''/usr/share/doc/kernel-doc-/Documentation/networking/ip-sysctl.txt''