Understanding RAM cache size and limits

Discussion:

Veiko Kukk

2018-02-16 11:56:36 UTC

Hi,

We have strange situation with ATS 7.1.2 where RAM cache has grown beyond
set limits.

# /opt/trafficserver/bin/traffic_ctl metric match ram
proxy.process.cache.ram_cache.total_bytes 3112697856
proxy.process.cache.ram_cache.bytes_used 4493088640
proxy.process.cache.ram_cache.hits 1548
proxy.process.cache.ram_cache.misses 1635443
proxy.process.cache.volume_1.ram_cache.total_bytes 778174464
proxy.process.cache.volume_1.ram_cache.bytes_used 1067949184
proxy.process.cache.volume_1.ram_cache.hits 374
proxy.process.cache.volume_1.ram_cache.misses 403206
proxy.process.cache.volume_2.ram_cache.total_bytes 778174464
proxy.process.cache.volume_2.ram_cache.bytes_used 1129483520
proxy.process.cache.volume_2.ram_cache.hits 368
proxy.process.cache.volume_2.ram_cache.misses 410164
proxy.process.cache.volume_3.ram_cache.total_bytes 778174464
proxy.process.cache.volume_3.ram_cache.bytes_used 1084551424
proxy.process.cache.volume_3.ram_cache.hits 357
proxy.process.cache.volume_3.ram_cache.misses 408656
proxy.process.cache.volume_4.ram_cache.total_bytes 778174464
proxy.process.cache.volume_4.ram_cache.bytes_used 1211104512
proxy.process.cache.volume_4.ram_cache.hits 449
proxy.process.cache.volume_4.ram_cache.misses 413417

Relevant config parameters:
# /opt/trafficserver/bin/traffic_ctl config match ram
proxy.config.cache.ram_cache_cutoff: 16777216
proxy.config.cache.ram_cache.size: -1
proxy.config.cache.ram_cache.algorithm: 1
proxy.config.cache.ram_cache.use_seen_filter: 1
proxy.config.cache.ram_cache.compress: 0
proxy.config.cache.ram_cache.compress_percent: 90
proxy.config.ssl.server.dhparams_file: NULL
proxy.config.http2.max_frame_size: 16384

I understand that, if set to -1, ATS will determine RAM cache size
automatically and that in our case would be
proxy.process.cache.ram_cache.total_bytes 3112697856 .

Why and how can it use more than that?

--
Veiko

Bryan Call

2018-02-16 21:34:09 UTC

Permalink

Are you see this on more than one host? Have you tried restarting it and see if it goes over the maximum again?

Here are are the stats I am seeing from a 7.1.2 server in production:

[***@e10 ~]$ traffic_ctl metric match ram
proxy.process.cache.ram_cache.total_bytes 31999999997
proxy.process.cache.ram_cache.bytes_used 31993472640
proxy.process.cache.volume_1.ram_cache.total_bytes 8000000000
proxy.process.cache.volume_1.ram_cache.bytes_used 7997441792
proxy.process.cache.volume_1.ram_cache.hits 32096586
proxy.process.cache.volume_1.ram_cache.misses 10833490
proxy.process.cache.volume_2.ram_cache.total_bytes 8000000000
proxy.process.cache.volume_2.ram_cache.bytes_used 7998451328
proxy.process.cache.volume_2.ram_cache.hits 34837324
proxy.process.cache.volume_2.ram_cache.misses 10840846
proxy.process.cache.volume_3.ram_cache.total_bytes 8000000000
proxy.process.cache.volume_3.ram_cache.bytes_used 7999866624
proxy.process.cache.volume_3.ram_cache.hits 32447086
proxy.process.cache.volume_3.ram_cache.misses 10697251
proxy.process.cache.volume_4.ram_cache.total_bytes 7999999997
proxy.process.cache.volume_4.ram_cache.bytes_used 7997786880
proxy.process.cache.volume_4.ram_cache.hits 30927240
proxy.process.cache.volume_4.ram_cache.misses 10826102

Hi,
We have strange situation with ATS 7.1.2 where RAM cache has grown beyond set limits.
# /opt/trafficserver/bin/traffic_ctl metric match ram
proxy.process.cache.ram_cache.total_bytes 3112697856
proxy.process.cache.ram_cache.bytes_used 4493088640
proxy.process.cache.ram_cache.hits 1548
proxy.process.cache.ram_cache.misses 1635443
proxy.process.cache.volume_1.ram_cache.total_bytes 778174464
proxy.process.cache.volume_1.ram_cache.bytes_used 1067949184
proxy.process.cache.volume_1.ram_cache.hits 374
proxy.process.cache.volume_1.ram_cache.misses 403206
proxy.process.cache.volume_2.ram_cache.total_bytes 778174464
proxy.process.cache.volume_2.ram_cache.bytes_used 1129483520
proxy.process.cache.volume_2.ram_cache.hits 368
proxy.process.cache.volume_2.ram_cache.misses 410164
proxy.process.cache.volume_3.ram_cache.total_bytes 778174464
proxy.process.cache.volume_3.ram_cache.bytes_used 1084551424
proxy.process.cache.volume_3.ram_cache.hits 357
proxy.process.cache.volume_3.ram_cache.misses 408656
proxy.process.cache.volume_4.ram_cache.total_bytes 778174464
proxy.process.cache.volume_4.ram_cache.bytes_used 1211104512
proxy.process.cache.volume_4.ram_cache.hits 449
proxy.process.cache.volume_4.ram_cache.misses 413417
# /opt/trafficserver/bin/traffic_ctl config match ram
proxy.config.cache.ram_cache_cutoff: 16777216
proxy.config.cache.ram_cache.size: -1
proxy.config.cache.ram_cache.algorithm: 1
proxy.config.cache.ram_cache.use_seen_filter: 1
proxy.config.cache.ram_cache.compress: 0
proxy.config.cache.ram_cache.compress_percent: 90
proxy.config.ssl.server.dhparams_file: NULL
proxy.config.http2.max_frame_size: 16384
I understand that, if set to -1, ATS will determine RAM cache size automatically and that in our case would be proxy.process.cache.ram_cache.total_bytes 3112697856 .
Why and how can it use more than that?
--
Veiko

Veiko Kukk

2018-02-19 07:52:42 UTC

Permalink

Yes, other server have same issue too.

It takes some time to fill RAM cache and it's efficiency for our use
case/setup is not good as seen from my original post.
We are actually planning to disable RAM cache entirely. But I don't see why
it would not go over limit again after restarting. It's not like
computers/programs behave differently at every startup.

Veiko

Post by Bryan Call
Are you see this on more than one host? Have you tried restarting it and
see if it goes over the maximum again?
proxy.process.cache.ram_cache.total_bytes 31999999997
proxy.process.cache.ram_cache.bytes_used 31993472640
proxy.process.cache.volume_1.ram_cache.total_bytes 8000000000
proxy.process.cache.volume_1.ram_cache.bytes_used 7997441792
proxy.process.cache.volume_1.ram_cache.hits 32096586
proxy.process.cache.volume_1.ram_cache.misses 10833490
proxy.process.cache.volume_2.ram_cache.total_bytes 8000000000
proxy.process.cache.volume_2.ram_cache.bytes_used 7998451328
proxy.process.cache.volume_2.ram_cache.hits 34837324
proxy.process.cache.volume_2.ram_cache.misses 10840846
proxy.process.cache.volume_3.ram_cache.total_bytes 8000000000
proxy.process.cache.volume_3.ram_cache.bytes_used 7999866624
proxy.process.cache.volume_3.ram_cache.hits 32447086
proxy.process.cache.volume_3.ram_cache.misses 10697251
proxy.process.cache.volume_4.ram_cache.total_bytes 7999999997
proxy.process.cache.volume_4.ram_cache.bytes_used 7997786880
proxy.process.cache.volume_4.ram_cache.hits 30927240
proxy.process.cache.volume_4.ram_cache.misses 10826102

Post by Veiko Kukk
Hi,
We have strange situation with ATS 7.1.2 where RAM cache has grown

beyond set limits.

Post by Veiko Kukk
# /opt/trafficserver/bin/traffic_ctl metric match ram
proxy.process.cache.ram_cache.total_bytes 3112697856
proxy.process.cache.ram_cache.bytes_used 4493088640
proxy.process.cache.ram_cache.hits 1548
proxy.process.cache.ram_cache.misses 1635443
proxy.process.cache.volume_1.ram_cache.total_bytes 778174464
proxy.process.cache.volume_1.ram_cache.bytes_used 1067949184
proxy.process.cache.volume_1.ram_cache.hits 374
proxy.process.cache.volume_1.ram_cache.misses 403206
proxy.process.cache.volume_2.ram_cache.total_bytes 778174464
proxy.process.cache.volume_2.ram_cache.bytes_used 1129483520
proxy.process.cache.volume_2.ram_cache.hits 368
proxy.process.cache.volume_2.ram_cache.misses 410164
proxy.process.cache.volume_3.ram_cache.total_bytes 778174464
proxy.process.cache.volume_3.ram_cache.bytes_used 1084551424
proxy.process.cache.volume_3.ram_cache.hits 357
proxy.process.cache.volume_3.ram_cache.misses 408656
proxy.process.cache.volume_4.ram_cache.total_bytes 778174464
proxy.process.cache.volume_4.ram_cache.bytes_used 1211104512
proxy.process.cache.volume_4.ram_cache.hits 449
proxy.process.cache.volume_4.ram_cache.misses 413417
# /opt/trafficserver/bin/traffic_ctl config match ram
proxy.config.cache.ram_cache_cutoff: 16777216
proxy.config.cache.ram_cache.size: -1
proxy.config.cache.ram_cache.algorithm: 1
proxy.config.cache.ram_cache.use_seen_filter: 1
proxy.config.cache.ram_cache.compress: 0
proxy.config.cache.ram_cache.compress_percent: 90
proxy.config.ssl.server.dhparams_file: NULL
proxy.config.http2.max_frame_size: 16384
I understand that, if set to -1, ATS will determine RAM cache size

automatically and that in our case would be proxy.process.cache.ram_cache.total_bytes
3112697856 .

Post by Veiko Kukk
Why and how can it use more than that?
--
Veiko