child always re-validates object fetched from parent

[Patrick O'Brien]

Hello,

I am testing out hierarchical caching in ATS 7.1.1. Everything seems
to work, although it looks like the child node is always needing to
revalidate objects fetched from the parent node. Here are the headers
on an example GET:
https://gist.github.com/poblahblahblah/928aaa88c24ef3c05366433a514f1a28

The parent has the object stored in RAM and doesn't require
validation, but the child thinks the response is stale. Here is the
cache.config file on both the parent and the child:
https://gist.github.com/poblahblahblah/36b54ac8b7d75071c1342da6694d3c51

The problem with this is that the connection between the child and
parent is pretty flaky and when the parent is down we are met with a
502: https://gist.github.com/poblahblahblah/d50657354c815040faae50539fa45836

Here is the parent.config on the child:
https://gist.github.com/poblahblahblah/9f1dcb4cf674c21c80a27173bd0ef81b

Relevant configs from records.config for the parent:
https://gist.github.com/poblahblahblah/e6ac573e9f9cd63088c2e06e6544d801
Relevant configs from records.config for the child:
https://gist.github.com/poblahblahblah/5a6dc02a88b24cafca0e8c70d353b92e

Any guidance on this would be much appreciated.

-patrick

[Jeremy Payne]

Can you post the complete records.config from your child cache ?
If I had to guess, it looks like heurstics or fuzzytimer may be set too low ???
But if you provide a company sanitized version of your records.config
that should give us a better understanding of your setup.






On Wed, Jan 3, 2018 at 6:10 PM, Patrick O'Brien

Post by Patrick O'Brien
Hello,
I am testing out hierarchical caching in ATS 7.1.1. Everything seems
to work, although it looks like the child node is always needing to
revalidate objects fetched from the parent node. Here are the headers
https://gist.github.com/poblahblahblah/928aaa88c24ef3c05366433a514f1a28
The parent has the object stored in RAM and doesn't require
validation, but the child thinks the response is stale. Here is the
https://gist.github.com/poblahblahblah/36b54ac8b7d75071c1342da6694d3c51
The problem with this is that the connection between the child and
parent is pretty flaky and when the parent is down we are met with a
502: https://gist.github.com/poblahblahblah/d50657354c815040faae50539fa45836
https://gist.github.com/poblahblahblah/9f1dcb4cf674c21c80a27173bd0ef81b
https://gist.github.com/poblahblahblah/e6ac573e9f9cd63088c2e06e6544d801
https://gist.github.com/poblahblahblah/5a6dc02a88b24cafca0e8c70d353b92e
Any guidance on this would be much appreciated.
-patrick

[Patrick O'Brien]

Hey Jeremy,

Here's the full config from the child node:
https://gist.github.com/poblahblahblah/a1abdad3d64ebdfba003c75e50c9fe00

Thanks for looking!

Post by Jeremy Payne
Can you post the complete records.config from your child cache ?
If I had to guess, it looks like heurstics or fuzzytimer may be set too low ???
But if you provide a company sanitized version of your records.config
that should give us a better understanding of your setup.
On Wed, Jan 3, 2018 at 6:10 PM, Patrick O'Brien

Post by Patrick O'Brien
Hello,
I am testing out hierarchical caching in ATS 7.1.1. Everything seems
to work, although it looks like the child node is always needing to
revalidate objects fetched from the parent node. Here are the headers
https://gist.github.com/poblahblahblah/928aaa88c24ef3c05366433a514f1a28
The parent has the object stored in RAM and doesn't require
validation, but the child thinks the response is stale. Here is the
https://gist.github.com/poblahblahblah/36b54ac8b7d75071c1342da6694d3c51
The problem with this is that the connection between the child and
parent is pretty flaky and when the parent is down we are met with a
502: https://gist.github.com/poblahblahblah/d50657354c815040faae50539fa45836
https://gist.github.com/poblahblahblah/9f1dcb4cf674c21c80a27173bd0ef81b
https://gist.github.com/poblahblahblah/e6ac573e9f9cd63088c2e06e6544d801
https://gist.github.com/poblahblahblah/5a6dc02a88b24cafca0e8c70d353b92e
Any guidance on this would be much appreciated.
-patrick

[Jeremy Payne]

Can you modify the below to reflect non-zero values ?

##############################################################################
# Heuristic cache expiration. Docs:
# https://docs.trafficserver.apache.org/records.config#heuristic-expiration
##############################################################################
# Setting all of these to zero in order for everything to be considered stale
# by ATS. If we want to change this in the future, this is where to do it.
CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 0
CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 0
CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000000
##############################################################################

On Mon, Jan 8, 2018 at 10:24 AM, Patrick O'Brien

Post by Patrick O'Brien
Hey Jeremy,
https://gist.github.com/poblahblahblah/a1abdad3d64ebdfba003c75e50c9fe00
Thanks for looking!

Post by Jeremy Payne
Can you post the complete records.config from your child cache ?
If I had to guess, it looks like heurstics or fuzzytimer may be set too low ???
But if you provide a company sanitized version of your records.config
that should give us a better understanding of your setup.
On Wed, Jan 3, 2018 at 6:10 PM, Patrick O'Brien

Post by Patrick O'Brien
Hello,
I am testing out hierarchical caching in ATS 7.1.1. Everything seems
to work, although it looks like the child node is always needing to
revalidate objects fetched from the parent node. Here are the headers
https://gist.github.com/poblahblahblah/928aaa88c24ef3c05366433a514f1a28
The parent has the object stored in RAM and doesn't require
validation, but the child thinks the response is stale. Here is the
https://gist.github.com/poblahblahblah/36b54ac8b7d75071c1342da6694d3c51
The problem with this is that the connection between the child and
parent is pretty flaky and when the parent is down we are met with a
502: https://gist.github.com/poblahblahblah/d50657354c815040faae50539fa45836
https://gist.github.com/poblahblahblah/9f1dcb4cf674c21c80a27173bd0ef81b
https://gist.github.com/poblahblahblah/e6ac573e9f9cd63088c2e06e6544d801
https://gist.github.com/poblahblahblah/5a6dc02a88b24cafca0e8c70d353b92e
Any guidance on this would be much appreciated.
-patrick

[Patrick O'Brien]

Good catch!

This didn't fix the issue by itself, but it did remind me that
proxy.config.http.cache.when_to_revalidate was set to 2 (Always
Stale).

I guess on our non-chained proxies everything worked because of our
rules in cache.config?

I appreciate the advice, Jeremy. Thank you!

Post by Jeremy Payne
Can you modify the below to reflect non-zero values ?
##############################################################################
# https://docs.trafficserver.apache.org/records.config#heuristic-expiration
##############################################################################
# Setting all of these to zero in order for everything to be considered stale
# by ATS. If we want to change this in the future, this is where to do it.
CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 0
CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 0
CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.000000
##############################################################################
On Mon, Jan 8, 2018 at 10:24 AM, Patrick O'Brien

Post by Patrick O'Brien
Hey Jeremy,
https://gist.github.com/poblahblahblah/a1abdad3d64ebdfba003c75e50c9fe00
Thanks for looking!

Post by Jeremy Payne
Can you post the complete records.config from your child cache ?
If I had to guess, it looks like heurstics or fuzzytimer may be set too low ???
But if you provide a company sanitized version of your records.config
that should give us a better understanding of your setup.
On Wed, Jan 3, 2018 at 6:10 PM, Patrick O'Brien

Post by Patrick O'Brien
Hello,
I am testing out hierarchical caching in ATS 7.1.1. Everything seems
to work, although it looks like the child node is always needing to
revalidate objects fetched from the parent node. Here are the headers
https://gist.github.com/poblahblahblah/928aaa88c24ef3c05366433a514f1a28
The parent has the object stored in RAM and doesn't require
validation, but the child thinks the response is stale. Here is the
https://gist.github.com/poblahblahblah/36b54ac8b7d75071c1342da6694d3c51
The problem with this is that the connection between the child and
parent is pretty flaky and when the parent is down we are met with a
502: https://gist.github.com/poblahblahblah/d50657354c815040faae50539fa45836
https://gist.github.com/poblahblahblah/9f1dcb4cf674c21c80a27173bd0ef81b
https://gist.github.com/poblahblahblah/e6ac573e9f9cd63088c2e06e6544d801
https://gist.github.com/poblahblahblah/5a6dc02a88b24cafca0e8c70d353b92e
Any guidance on this would be much appreciated.
-patrick