ATS and TLS close-notify

[Jeremy Payne]

Context:

Openssl 102k
ATS 714

I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.

"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."


However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
Which will never come, since per spec:

"Any data received after a closure alert is ignored."


Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?

I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.

Thanks!

[Susan Hinrichs]

Yes, ATS should respond with close notify or at least FIN the connection.
What version of ATS are you seeing this with?

If there was already an application data packet in flight, it may arrive
after the client sends the close notify. But in general ATS should shut
down the connection.

Post by Jeremy Payne
Openssl 102k
ATS 714
I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.
"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."
However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
"Any data received after a closure alert is ignored."
Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?
I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.
Thanks!

[Chou, Peter]

Susan,

The problem below is reported against 7.1.4. As background, we believe that the “lingering connections” situation described below is either causing or contributing to the number of connections on the server to increase. We see that nearby 6.2.1 nodes under similar load would see a smaller increase in the number of connections compared to 7.1.4. Is it possible that changes between 6.2.1 and 7.1.4 may have an effect such as the tuning done towards calling SSL_shutdown() in commit 0ab449? Appreciate any insights.

Thanks,
Peter


From: Susan Hinrichs [mailto:***@oath.com]
Sent: Saturday, September 01, 2018 3:57 PM
To: ***@trafficserver.apache.org
Subject: Re: ATS and TLS close-notify

Yes, ATS should respond with close notify or at least FIN the connection. What version of ATS are you seeing this with?

If there was already an application data packet in flight, it may arrive after the client sends the close notify. But in general ATS should shut down the connection.

On Fri, Aug 31, 2018, 11:31 PM Jeremy Payne <***@gmail.com<mailto:***@gmail.com>> wrote:
Context:

Openssl 102k
ATS 714

I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.

"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."


However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
Which will never come, since per spec:

"Any data received after a closure alert is ignored."


Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?

I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.

Thanks!

[Susan Hinrichs]

I'd have to dig in some more, but a quick look at the commit you referenced
I don't think would cause what Jeremy is describing. It is an addition to
do_io_close. At that point ATS will at least send a FIN and will possibly
send a close-notify. It will not continue to send data.

Post by Chou, Peter
Susan,
The problem below is reported against 7.1.4. As background, we believe
that the “lingering connections” situation described below is either
causing or contributing to the number of connections on the server to
increase. We see that nearby 6.2.1 nodes under similar load would see a
smaller increase in the number of connections compared to 7.1.4. Is it
possible that changes between 6.2.1 and 7.1.4 may have an effect such as
the tuning done towards calling SSL_shutdown() in commit 0ab449? Appreciate
any insights.
Thanks,
Peter
*Sent:* Saturday, September 01, 2018 3:57 PM
*Subject:* Re: ATS and TLS close-notify
Yes, ATS should respond with close notify or at least FIN the connection.
What version of ATS are you seeing this with?
If there was already an application data packet in flight, it may arrive
after the client sends the close notify. But in general ATS should shut
down the connection.
Openssl 102k
ATS 714
I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.
"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."
However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
"Any data received after a closure alert is ignored."
Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?
I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.
Thanks!

[Susan Hinrichs]

Thinking on this some more, this sounds like bad interactions with the TCP
half closed logic in the state machine. If you are doing HTTP 1 over
non-TLS, it is legal for a client to send a FIN but then read more data
that the server sends. There is some logic to turn off this half close
logic in traffic server in inappropriate cases but it is not perfect and
has varied over time.

Earlier this year there was a PR to add a knob to turn off this behavior,
but I don't know where it landed. I will check that out when I get back to
the office.

Susan

Post by Susan Hinrichs
Yes, ATS should respond with close notify or at least FIN the connection.
What version of ATS are you seeing this with?
If there was already an application data packet in flight, it may arrive
after the client sends the close notify. But in general ATS should shut
down the connection.

Post by Jeremy Payne
Openssl 102k
ATS 714
I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.
"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."
However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
"Any data received after a closure alert is ignored."
Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?
I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.
Thanks!

[Leif Hedstrom]

That seems plausible , but isn’t the indication that things got a lot worse from v6.x to 7.x?

The half close logic is old, isn’t it? Did we change something into it in 7.x?

— Leif

Thinking on this some more, this sounds like bad interactions with the TCP half closed logic in the state machine. If you are doing HTTP 1 over non-TLS, it is legal for a client to send a FIN but then read more data that the server sends. There is some logic to turn off this half close logic in traffic server in inappropriate cases but it is not perfect and has varied over time.
Earlier this year there was a PR to add a knob to turn off this behavior, but I don't know where it landed. I will check that out when I get back to the office.
Susan

Post by Chou, Peter
Yes, ATS should respond with close notify or at least FIN the connection. What version of ATS are you seeing this with?
If there was already an application data packet in flight, it may arrive after the client sends the close notify. But in general ATS should shut down the connection.

Post by Jeremy Payne
Openssl 102k
ATS 714
I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.
"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."
However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
"Any data received after a closure alert is ignored."
Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?
I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.
Thanks!

[Susan Hinrichs]

True the change in behavior between 6.x and 7.x is concerning. The basic
half-open logic hasn't changed as far as I know. There is one H2 change on
master, but that is not on the 7.1.x branch (just checked). There may be
some tweaky little change. Or it may be something else entirely.

Post by Leif Hedstrom
That seems plausible , but isn’t the indication that things got a lot
worse from v6.x to 7.x?
The half close logic is old, isn’t it? Did we change something into it in 7.x?
— Leif
Thinking on this some more, this sounds like bad interactions with the TCP
half closed logic in the state machine. If you are doing HTTP 1 over
non-TLS, it is legal for a client to send a FIN but then read more data
that the server sends. There is some logic to turn off this half close
logic in traffic server in inappropriate cases but it is not perfect and
has varied over time.
Earlier this year there was a PR to add a knob to turn off this behavior,
but I don't know where it landed. I will check that out when I get back to
the office.
Susan

Post by Susan Hinrichs
Yes, ATS should respond with close notify or at least FIN the connection.
What version of ATS are you seeing this with?
If there was already an application data packet in flight, it may arrive
after the client sends the close notify. But in general ATS should shut
down the connection.

Post by Jeremy Payne
Openssl 102k
ATS 714
I notice that at times a client will send a TLS 1.2 close-notify,
immediately followed by a FIN-ACK. Which seems to be following spec.
"It is not required for the initiator of the close to wait for the
responding close_notify alert before
closing the read side of the connection."
However, in response, ATS continuous to send 'application data'
instead of issuing its own TLS 1.2 close-notify. Which then results in
connections lingering waiting for an ACK back from the client.
"Any data received after a closure alert is ignored."
Is ATS still within TLS 1.2 spec by continuing to send application
data, even though the client sent a close notify ?
I tested some other https servers compiled against openssl 102k, and I
see a close notify sent by the client, with the https server
responding with it's own close notify.
Thanks!