Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout

[Reindl Harald]

on httpd this is just a single config line

https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500

while we have ratelimiting and max-connection per ip/subnet to solve
that problems on the firewall instead in the attacked application it
makes me tired in case of external security audits that i have to
explain every single time that this is because of rate-control
whitelists for the scanner IP

Severity
Medium

Type
Configuration
Reported by module Slow_HTTP_DOS

Description
Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.

Slowloris and Slow HTTP POST DoS attacks rely on the fact that the HTTP
protocol, by design, requires requests to be completely received by the
server before they are processed. If an HTTP request is not complete, or
if the transfer rate is very low, the server keeps its resources busy
waiting for the rest of the data. If the server keeps too many resources
busy, this creates a denial of service.

Impact
A single machine can take down another machine's web server with minimal
bandwidth and side effects on unrelated
services and ports.

[Reindl Harald]

the current timeout configs are terrible

16:04:00 - request start
16:05:00 - still no repsone while expected
16:09:00 - Proxy: Inactivity Timeout

WTF - that's likely the "Timeout 30" but i strongly doubt httpd waits 5
minutes to close the backend connection and so for whatever reason
"proxy.config.http.transaction_no_activity_timeout_out" get triggerd

20170821.16h09m03s CONNECT: could not connect to *.*.*.* for
'http://example.com/timeout.php' (setting last failure time)
20170821.16h09m03s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://example.com/timeout.php'

and after that you pretend "could not connect [INACTIVE_TIMEOUT]" to
follow up requests which would hahve been served promptly (at least only
for that domain and not the other 200 on the same origin IP)

<?php
$start = time();
header('Content-Type: text/plain');
sleep(60);
echo "OK: ", (time() - $start), "\n";
?>

CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 5
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 1
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 5
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 300
CONFIG proxy.config.http.transaction_active_timeout_in INT 900
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.0

Post by Reindl Harald
on httpd this is just a single config line
https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
while we have ratelimiting and max-connection per ip/subnet to solve
that problems on the firewall instead in the attacked application it
makes me tired in case of external security audits that i have to
explain every single time that this is because of rate-control
whitelists for the scanner IP
Severity
Medium
Type
Configuration
Reported by module Slow_HTTP_DOS
Description
Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.
Slowloris and Slow HTTP POST DoS attacks rely on the fact that the HTTP
protocol, by design, requires requests to be completely received by the
server before they are processed. If an HTTP request is not complete, or
if the transfer rate is very low, the server keeps its resources busy
waiting for the rest of the data. If the server keeps too many resources
busy, this creates a denial of service.
Impact
A single machine can take down another machine's web server with minimal
bandwidth and side effects on unrelated
services and ports.

[Reindl Harald]

[***@arrakis:~]$ tail -f php_error.log
[21-Aug-2017 16:31:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60


WTF - log on the Backend-Server
[21-Aug-2017 16:31:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:32:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:33:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:34:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:35:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60

and now what - trafficserver is trying every 60 seconds for
"proxy.config.http.transaction_no_activity_timeout_out" and don't say
anything to the client?

WTF - log on the trafficserver
20170821.16h30m39s CONNECT:[0] could not connect [INACTIVE_TIMEOUT] to
*.*.*.* for 'http://www.example.com/timeout.php?x=1'
20170821.16h35m41s CONNECT: could not connect to *.*.*.* for
'http://www.example.com/timeout.php?x=1' (setting last failure time)
20170821.16h35m41s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://www.example.com/timeout.php?x=1'

yeah it can take some time on a output-buffering backend application to
respond and hence "CONFIG
proxy.config.http.transaction_no_activity_timeout_out INT 300" so i have
no idea why the script below leads to 5 log-entries and trafficserver
let the client sit and wait all the time without any response

<?php
ob_start();
$start = time();
sleep(60);
echo "OK: ", (time() - $start), "\n";
error_log('TEST PROXY/ORIGIN TIMOUT: 60');
?>

Post by Reindl Harald
the current timeout configs are terrible
16:04:00 - request start
16:05:00 - still no repsone while expected
16:09:00 - Proxy: Inactivity Timeout
WTF - that's likely the "Timeout 30" but i strongly doubt httpd waits 5
minutes to close the backend connection and so for whatever reason
"proxy.config.http.transaction_no_activity_timeout_out" get triggerd
20170821.16h09m03s CONNECT: could not connect to *.*.*.* for
'http://example.com/timeout.php' (setting last failure time)
20170821.16h09m03s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://example.com/timeout.php'
and after that you pretend "could not connect [INACTIVE_TIMEOUT]" to
follow up requests which would hahve been served promptly (at least only
for that domain and not the other 200 on the same origin IP)
<?php
$start = time();
header('Content-Type: text/plain');
sleep(60);
echo "OK: ", (time() - $start), "\n";
?>
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 5
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 1
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 5
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 300
CONFIG proxy.config.http.transaction_active_timeout_in INT 900
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.0

Post by Reindl Harald
on httpd this is just a single config line
https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
while we have ratelimiting and max-connection per ip/subnet to solve
that problems on the firewall instead in the attacked application it
makes me tired in case of external security audits that i have to
explain every single time that this is because of rate-control
whitelists for the scanner IP
Severity
Medium
Type
Configuration
Reported by module Slow_HTTP_DOS
Description
Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.
Slowloris and Slow HTTP POST DoS attacks rely on the fact that the
HTTP protocol, by design, requires requests to be completely received
by the server before they are processed. If an HTTP request is not
complete, or if the transfer rate is very low, the server keeps its
resources busy waiting for the rest of the data. If the server keeps
too many resources busy, this creates a denial of service.
Impact
A single machine can take down another machine's web server with
minimal bandwidth and side effects on unrelated
services and ports.