ATS 7.1 graceful shutdown - client connections

[ezko]

Hi,
what is the cleanest way to shutdown traffic server in terms of client
connections.
The use case is rolling updates insides k8s. During update all new traffic
is routed to a new instance , but how do we make sure the terminating
instance has had a change to gracefully complete it's client connections.
Is using 'traffic_ctl server stop' good enough ?
what about keep alive ?
Thanks.



--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Masaori Koshiba]

"Drain" option is available from v8.0.0. You can use it by "traffic_ctl
server stop --drain". Details in below.
This works for both of HTTP/1.1 and HTTP/2.

https://docs.trafficserver.apache.org/en/8.0.x/appendices/command-line/traffic_ctl.en.html#cmdoption-traffic-ctl-server-restart-drain

For older version, you can inject "Connection: close" header by
header_rewrite plugin. But please note that this only works for HTTP/1.1.
HTTP/2 graceful shutdown is supported by v8.0.0.

Thanks,
Masaori

Post by ezko
Hi,
what is the cleanest way to shutdown traffic server in terms of client
connections.
The use case is rolling updates insides k8s. During update all new traffic
is routed to a new instance , but how do we make sure the terminating
instance has had a change to gracefully complete it's client connections.
Is using 'traffic_ctl server stop' good enough ?
what about keep alive ?
Thanks.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[ezko]

Thanks.
a few follow up questions.
for v8.0.0 documentation for drain states:
This option modifies the behavior of traffic_ctl server restart ...
So will drain also work for traffic_ctl server stop ?

for older version:
do you recommend to always inject "Connection: close" , or can we trigger it
before shutdown ?
Thanks.



--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Masaori Koshiba]

Post by ezko
This option modifies the behavior of traffic_ctl server restart ...
So will drain also work for traffic_ctl server stop ?

Oh, docs looks wrong. According to the source, server stop also has --drain
option.

https://github.com/apache/trafficserver/blob/82bbd37f80dec30315c86bbeb8803d73a487387e/src/traffic_ctl/server.cc#L122

Post by ezko
do you recommend to always inject "Connection: close" , or can we trigger it
before shutdown ?

We can trigger it before shutdown. Example of this case is here.

# Close Connections for draining
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/plugins/header_rewrite.en.html#close-connections-for-draining

- Masaori

Post by ezko
Thanks.
a few follow up questions.
This option modifies the behavior of traffic_ctl server restart ...
So will drain also work for traffic_ctl server stop ?
do you recommend to always inject "Connection: close" , or can we trigger it
before shutdown ?
Thanks.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[ezko]

thanks Masaori Koshiba-2
we will try it.




--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Jeremy Payne]

Maybe I'm missing something here..

If the main purpose here is to issue a 'Connection:close'
why not just disable client facing http keep-alive ?

traffic_ctl config set proxy.config.http.keep_alive_enabled_in 0

After the param has been activated, requests on existing connections
will receive a Connection:close.

Of course you still need to take your server out of l4/7 lb rotation
or dns load balancing rotation
so subsequent connections don't end up on the 'draining' server.

Post by ezko
thanks Masaori Koshiba-2
we will try it.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[ezko]

Thanks Jermy,
this sounds like a much simpler solution.





--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Phillip Moore]

We have a tool that we use to take nodes in and out of service. Basically
it either creates or removes a server_enabled file. This causes our L4 load
balancer to remove the ATS node from its pool. In addition to facilitate
draining in ATS header_rewrite plugin config we have this:

# close keepalive sessions if server is disabled
cond %{SEND_RESPONSE_HDR_HOOK}
cond %{ACCESS:/path/to/server_enabled} [NOT,OR]
set-header Connection "close"

Caveat is this doesn't work with HTTP/2 I'm told, but we don't do that so
this works nicely for us. I can't take credit for the header_rewrite trick,
I believe it came from Yahoo.

--
Phillip

Post by ezko
Thanks Jermy,
this sounds like a much simpler solution.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[ezko]

Hi Phillip,
we used Jermy's suggestion for draining keep-alives with k8s as follows:

lifecycle:
preStop:
exec:
command:
- /bin/bash
- -c
- rm /etc/environment;
traffic_ctl config set
proxy.config.http.keep_alive_enabled_in 0; sleep 10;
traffic_ctl server stop; sleep 5;
readinessProbe:
exec:
command:
- cat
- /etc/environment

so prior to terminating the POD , k8s will run our preStop hook where we do
the following:
1) remove the file so readiness probe starts to fail , and thus K8s stops
sending us new connections
2) set keep_alive_enabled to 0 , so any current keep alive connections are
drained
3) gracefully stop traffic server

we did some limited scope load testing with Jmeter (our use case is live
HLS) and rolling updates passed without errors.
But interestingly enough they also passed without steps 2) and 3) above. i'm
not sure why (maybe due to some Jmeter keep-alive configuration)

Additionally we tried using nginx ingress controller. this one had some
errors with Jmeter due to:
https://github.com/kubernetes/ingress-nginx/issues/489





--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/