Discussion:
very long cache_open_read_end - timeout
Mateusz Zajakala
2018-02-22 11:59:17 UTC
Permalink
Hi,

I have a problem - from time to time my ATS 7.2.1 has problems with serving
static files from origin, client transactions last over 10s and I see the
following diagnostic (I have enabled logging "slow requests")

cache_open_read_begin: 0.000 cache_open_read_end: 10.640

Any requests to the same URL are blocked for over 10s. Restarting ATS
solves the problem.

What does it mean? Does it have to do with open_read feature? but the
timeouts don't add...

Any suggestions appreciated!

Here's my records.config with configs related to avoiding thundering herd...

CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200

CONFIG proxy.config.http.cache.open_write_fail_action INT 2
Veiko Kukk
2018-02-23 11:14:16 UTC
Permalink
Hi, Mateusz

I wonder if you might experience same issue as me.
https://github.com/apache/trafficserver/issues/3057

For me, solution, or I would rather call it workaround was to create
volumes.

Unfortunately, documentation regarding to storage performance issues/tuning
is missing.

Best regards,
Veiko
Post by Mateusz Zajakala
Hi,
I have a problem - from time to time my ATS 7.2.1 has problems with
serving static files from origin, client transactions last over 10s and I
see the following diagnostic (I have enabled logging "slow requests")
cache_open_read_begin: 0.000 cache_open_read_end: 10.640
Any requests to the same URL are blocked for over 10s. Restarting ATS
solves the problem.
What does it mean? Does it have to do with open_read feature? but the
timeouts don't add...
Any suggestions appreciated!
Here's my records.config with configs related to avoiding thundering herd...
CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200
CONFIG proxy.config.http.cache.open_write_fail_action INT 2
Bryan Call
2018-02-23 19:21:28 UTC
Permalink
Veiko,

How many volumes did you create?

-Bryan
Post by Veiko Kukk
Hi, Mateusz
I wonder if you might experience same issue as me.
https://github.com/apache/trafficserver/issues/3057 <https://github.com/apache/trafficserver/issues/3057>
For me, solution, or I would rather call it workaround was to create volumes.
Unfortunately, documentation regarding to storage performance issues/tuning is missing.
Best regards,
Veiko
Hi,
I have a problem - from time to time my ATS 7.2.1 has problems with serving static files from origin, client transactions last over 10s and I see the following diagnostic (I have enabled logging "slow requests")
cache_open_read_begin: 0.000 cache_open_read_end: 10.640
Any requests to the same URL are blocked for over 10s. Restarting ATS solves the problem.
What does it mean? Does it have to do with open_read feature? but the timeouts don't add...
Any suggestions appreciated!
Here's my records.config with configs related to avoiding thundering herd...
CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200
CONFIG proxy.config.http.cache.open_write_fail_action INT 2
Veiko Kukk
2018-02-26 08:15:22 UTC
Permalink
Hi Bryan,

We decided to create 4 volumes. No strong reasoning behind it, just 2
seemed too little to avoid those blocking timeouts and 4 was next
comfortable to set using 25%.

Veiko
Post by Bryan Call
Veiko,
How many volumes did you create?
-Bryan
Hi, Mateusz
I wonder if you might experience same issue as me.
https://github.com/apache/trafficserver/issues/3057
For me, solution, or I would rather call it workaround was to create volumes.
Unfortunately, documentation regarding to storage performance
issues/tuning is missing.
Best regards,
Veiko
Post by Mateusz Zajakala
Hi,
I have a problem - from time to time my ATS 7.2.1 has problems with
serving static files from origin, client transactions last over 10s and I
see the following diagnostic (I have enabled logging "slow requests")
cache_open_read_begin: 0.000 cache_open_read_end: 10.640
Any requests to the same URL are blocked for over 10s. Restarting ATS solves the problem.
What does it mean? Does it have to do with open_read feature? but the
timeouts don't add...
Any suggestions appreciated!
Here's my records.config with configs related to avoiding thundering herd...
CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200
CONFIG proxy.config.http.cache.open_write_fail_action INT 2
Bryan Call
2018-02-26 18:01:43 UTC
Permalink
That is what I set ATS to do in production for us. Funny that we both came up with 4. It was mainly do to lock contention issues and people benchmarking or running ATS at high request rates.

$ tail -4 volume.config
volume=1 scheme=http size=25%
volume=2 scheme=http size=25%
volume=3 scheme=http size=25%
volume=4 scheme=http size=25%

-Bryan
Post by Veiko Kukk
Hi Bryan,
We decided to create 4 volumes. No strong reasoning behind it, just 2 seemed too little to avoid those blocking timeouts and 4 was next comfortable to set using 25%.
Veiko
Veiko,
How many volumes did you create?
-Bryan
Post by Veiko Kukk
Hi, Mateusz
I wonder if you might experience same issue as me.
https://github.com/apache/trafficserver/issues/3057 <https://github.com/apache/trafficserver/issues/3057>
For me, solution, or I would rather call it workaround was to create volumes.
Unfortunately, documentation regarding to storage performance issues/tuning is missing.
Best regards,
Veiko
Hi,
I have a problem - from time to time my ATS 7.2.1 has problems with serving static files from origin, client transactions last over 10s and I see the following diagnostic (I have enabled logging "slow requests")
cache_open_read_begin: 0.000 cache_open_read_end: 10.640
Any requests to the same URL are blocked for over 10s. Restarting ATS solves the problem.
What does it mean? Does it have to do with open_read feature? but the timeouts don't add...
Any suggestions appreciated!
Here's my records.config with configs related to avoiding thundering herd...
CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200
CONFIG proxy.config.http.cache.open_write_fail_action INT 2
Loading...