ATS act as S3 endpoint?

[Jin]

I am thinking to build something like this:

clients -> ats server -> s3 (ceph, aws...)

By doing this, users do not need to change existing code to use ats as
proxy. Instead, ats becomes an S3 endpoint.

I have a working ats setup that clients may utilize ats as proxy to access
s3 store.

curl -vx ats-server.corp.com http://s3-store.corp.com/bucket/object

Trying to work out several different remap rules but couldn't get it. Is it
possible that users do 'curl ats-server.corp.com/bucket/object' directly
with some remap rules?

Thank you.


Jin.





--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Miles Libbey]

Post by Jin
clients -> ats server -> s3 (ceph, aws...)
By doing this, users do not need to change existing code to use ats as
proxy. Instead, ats becomes an S3 endpoint.
I have a working ats setup that clients may utilize ats as proxy to access
s3 store.
curl -vx ats-server.corp.com http://s3-store.corp.com/bucket/object

A few thoughts:
- I think curl assumes port 1080 if you don't specify one with -x. So,
if ats-server.corp.com is running on port 80, you'd need
curl -vx ats-server.corp.com:80 http://s3-store.corp.com/bucket/object
- Is your Amazon bucket public? If not, you'll want the s3 plugin --
and if you have multiple buckets with different keys, you'll need
different remaps for each to configure those secrets.
- s3 will want the Host: header to be s3.amazonaws... so, need to
check/override the
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/records.config.en.html#proxy.config.url_remap.pristine_host_hdr
if it's not the default.

Seems like your remap rules would look like
map http://s3-store.corp.com/private/bucket
http://s3.amazonaws.com/private/bucket \
@plugin=s3_auth.so @pparam=--config @pparam= ...

map http://s3-store.corp.com/public/bucket
http://s3.amazonaws.com/public/bucket
...

Post by Jin
Trying to work out several different remap rules but couldn't get it. Is it
possible that users do 'curl ats-server.corp.com/bucket/object' directly
with some remap rules?
Thank you.
Jin.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Jin]

s3-store.corp.com is an internal Ceph storage. I also have ats and Ceph
gateway on port 8080.

So, this works:
curl -vx ats-server.corp.com:8080
http://s3-store.corp.com:8080/bucket/object

I hope to have:
curl http://atx-server.corp.com:8080/bucket/object

So, I added a remap rule:
regex_map /(.*) http://s3-store.corp.com:8080/$1

I am trying to figure out why it does not with this regex_map. The log says
could not connect INACTIVE_TIMEOUT to x.x.x.x for
http://ats-server.corp.com:8080/bucket/object (x.x.x.x is the ip of
ats-server.corp.com).

Is it some kind of a lopping situation?

Re s3_auth, this is a proof-of-concept setup. So, I will not use s3 auth
plugin yet as the bucket is public.

Thank you.

Jin.






--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/

[Miles Libbey]

I don't think I understand what's happening and what the problem is.
if
curl -vx ats-server.corp.com:8080 http://s3-store.corp.com:8080/bucket/object
works, and your regex_remap below is on ats-server.corp.com, that
seems to suggest that everything is working.

As written, your regex_remap should just be a simple map. That is,
this is exactly equivalent:
map / http://s3-store.corp.com:8080/
(eg, everything after the slash will get appended onto
http://s3-store.corp.com:8080/ when making up the url to go to the
origin.)

Post by Jin
s3-store.corp.com is an internal Ceph storage. I also have ats and Ceph
gateway on port 8080.
curl -vx ats-server.corp.com:8080
http://s3-store.corp.com:8080/bucket/object
curl http://atx-server.corp.com:8080/bucket/object
regex_map /(.*) http://s3-store.corp.com:8080/$1
I am trying to figure out why it does not with this regex_map. The log says
could not connect INACTIVE_TIMEOUT to x.x.x.x for
http://ats-server.corp.com:8080/bucket/object (x.x.x.x is the ip of
ats-server.corp.com).
Is it some kind of a lopping situation?
Re s3_auth, this is a proof-of-concept setup. So, I will not use s3 auth
plugin yet as the bucket is public.
Thank you.
Jin.
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/