HTTP Headers in a forward proxy

[Geoffrey Kneale]

Hello All,

Is it possible to manipulate the HTTP Headers in Trafficserver configured
to be a forward proxy? My goal is to support the YouTube Education filter,
which utilizes either A or B below.

A) Add new HTTP header rule

Modify your hardware filter or proxy server settings so that all outgoing
traffic to youtube.com contains the following custom HTTP header. The ID to
use in the HTTP header configuration, written below, is an example of a
unique ID for your school’s network only. If your school is blocked at the
district level, this HTTP header is then unique to the district network.

Example:
X-YouTube-Edu-Filter:ABCD1234567890abcdef

Field Name: X-YouTube-Edu-Filter
Description: When YouTube sees this header and an accompanying valid school
ID in the incoming traffic, YouTube will serve a limited EDU-only site to
all computers behind the school hardware filter.
Field Value Format: alphanumeric [a-z][A-Z][0-9]
Field Value Length: up to 44 characters

B) Create URL parameter rewrite rule

If your hardware filter does not support HTTP header modification, please
rewrite all outgoing URLs to youtube.com by appending the parameter
“edufilter” at the end. Exclude the &edufilter parameter from these file
types: .css, .gif, .png, .js, .xml
Example:
http://youtube.com/?edufilter=ABCD1234567890abcdef

Thanks in advance,

Geoffrey

[Alan M. Carroll]

I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.

Is it possible to manipulate the HTTP Headers in Trafficserver configured to be a forward proxy? My goal is to support the YouTube Education filter, which utilizes either A or B below.
A) Add new HTTP header rule
Modify your hardware filter or proxy server settings so that all outgoing traffic to youtube.com contains the following custom HTTP header.
X-YouTube-Edu-Filter:ABCD1234567890abcdef
B) Create URL parameter rewrite rule
If your hardware filter does not support HTTP header modification, please rewrite all outgoing URLs to youtube.com by appending the parameter “edufilter” at the end. Exclude the &edufilter parameter from these file types: .css, .gif, .png, .js, .xml
http://youtube.com/?edufilter=ABCD1234567890abcdef
http://youtu.be/gM95HHI4gLk

[James Peach]

Post by Alan M. Carroll
I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.

Could the header_filter plugin do this?

<http://trafficserver.apache.org/docs/trunk/admin/plugins/header_filter/index.en.html>

Post by Alan M. Carroll

Is it possible to manipulate the HTTP Headers in Trafficserver configured to be a forward proxy? My goal is to support the YouTube Education filter, which utilizes either A or B below.
A) Add new HTTP header rule
Modify your hardware filter or proxy server settings so that all outgoing traffic to youtube.com contains the following custom HTTP header.
X-YouTube-Edu-Filter:ABCD1234567890abcdef
B) Create URL parameter rewrite rule
If your hardware filter does not support HTTP header modification, please rewrite all outgoing URLs to youtube.com by appending the parameter “edufilter” at the end. Exclude the &edufilter parameter from these file types: .css, .gif, .png, .js, .xml
http://youtube.com/?edufilter=ABCD1234567890abcdef
http://youtu.be/gM95HHI4gLk

[Alan M. Carroll]

I think it would be a good start, but the problem would be that he wanted this in a forward proxy. He would have to set up remap rules to trigger the header filtering only on the appropriate targets. I'm not sure if it would be simpler to tweak the plugin to check the URL or not.

Post by James Peach

Post by Alan M. Carroll
I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.

Could the header_filter plugin do this?
<http://trafficserver.apache.org/docs/trunk/admin/plugins/header_filter/index.en.html>

[Leif Hedstrom]

Post by Alan M. Carroll
I think it would be a good start, but the problem would be that he wanted this in a forward proxy. He would have to set up remap rules to trigger the header filtering only on the appropriate targets. I'm not sure if it would be simpler to tweak the plugin to check the URL or not.

Yeah. He could (as you said) create one (or a few) remap.config rules for
youtube (he has to make sure remap.config is not required, but enabled via
the reverse proxy config). E.g.

map http://static.youtube.com http://static.youtube.com
@plugin=header_filter.so @pparam=youtube.config


or some such. Also remember he could use regular expressions here (on the
host part). e.g.

regex_map http://(.+)\.youtube\.(.+) http://$1.youtube.$2
@plugin=header_filter.so


(or some such, I don't know what the various youtube URLs looks like).

Cheers,

-- Leif

[Geoffrey Kneale]

Hello Alan and Leif,

Thank you for your insight.

I have downloaded and installed the header_filter plugin. However, I am
having some difficulty with making it work. I can visit YouTube and other
pages, but it seems to be ignoring the header manipulation. If you see any
glaring mistakes, I would appreciate and advise you may have.

So far, I have the following configuration items:

--- records.config ---
CONFIG proxy.config.url_remap.remap_required INT 0

--- remap.config ---
regex_map http://www.youtube.com
http://***@plugin=/libexec/trafficserver/header_filter.so
@pparam=/etc/trafficserver/youtube.config
regex_map http://(.*) http://$1

--- youtube.config ---
[SEND_REQUEST_HDR]
X-YouTube-Edu-Filter +*youtube_code_here*+

Post by Leif Hedstrom

Post by Alan M. Carroll
I think it would be a good start, but the problem would be that he wanted
this in a forward proxy. He would have to set up remap rules to trigger the
header filtering only on the appropriate targets. I'm not sure if it would
be simpler to tweak the plugin to check the URL or not.

Yeah. He could (as you said) create one (or a few) remap.config rules for
youtube (he has to make sure remap.config is not required, but enabled via
the reverse proxy config). E.g.
or some such. Also remember he could use regular expressions here (on the
host part). e.g.
regex_map http://(.+)\.youtube\.(.+) http://$1.youtube.$2
@plugin=header_filter.so
(or some such, I don't know what the various youtube URLs looks like).
Cheers,
-- Leif

[Alan M. Carroll]

Post by Geoffrey Kneale
--- remap.config ---
regex_map http://(.*) http://$1

Are there really no spaces in that? You need one in "comhttp" ("com http") and one before the '@'. Any reason for using regex_map instead of map? You should use map for straight text, and regex_map for actual regular expressions (note the difference in Leif's original email). If you're not familiar with regular expressions, we can point you documentation for that.

My experience is that all the You Tube URLS end in youtube.com. He would want to check his logs, though, to be sure.

I have to be careful, I've never actually used the remap stuff, but something more like

map http://youtube.com http://youtube.com @plugin=headerfilter.so @pparam=/etc/trafficserver/youtube.config
regex_map http://(.+)[.]youtube[.]com/(.*) http://$1.youtube.com/$2 @plugin=header_filter.so @pparam=/etc/trafficserver/youtube.config

Put the header_filter.so in the libexec directory of ATS, that is where it will look w/o a path. Otherwise you need to be sure that ATS is installed in the root directory for what you had. If you didn't configure that, it's more likely to be in /usr/local/.

Also, did you actually compile the header_filter.so so that it can be placed in the libexec directory?

[Leif Hedstrom]

Post by Alan M. Carroll

Post by Geoffrey Kneale
--- remap.config ---
regex_map http://(.*) http://$1

I think your MUA messed it up, it had spaces in my client :).

In any case, I'm guessing that the URL(s) don't match that first rule (which
is a strange rule to have as a regex, since it has no regexes in it). I also
think the last rule is unnecessary, either use "map /" or no map rule at all
(since you disabled remap required).

Post by Alan M. Carroll
My experience is that all the You Tube URLS end in youtube.com. He would want to check his logs, though, to be sure.
I have to be careful, I've never actually used the remap stuff, but something more like

Yeah, that makes more sense.

-- Leif

[Leif Hedstrom]

Post by Geoffrey Kneale
--- remap.config ---
regex_map
@pparam=/etc/trafficserver/youtube.config
regex_map http://(.*) http://$1

Two more things while I remember it:

1) You shouldn't have to type the full path to the plugin, as long as
you installed it in the normal install directory. I.e.
@plugin=header_filter.so should work.

2) This particular plugin is a bit "strange", in that it is both a remap
plugin, and normal (hooks) plugin. This means for it to function
properly, you also have to add a line to plugins.config, to make sure
all the hooks are activated. I don't know if the documentation specifies
this or not?

Cheers,

-- Leif

[Geoffrey Kneale]

Thanks Leif!

I have a working config. To note, I also needed to configure the
records.config to NOT force pristine headers.

In summary:
1. Configure records.config to not force remap and not force pristine
headers.
2. Download and install header filter plugin.
3. Configure plugins to use header filter (global w/o configuration file).
4. Configure the configuration file for the header filter.
5. Configure remap to use the header filter with Youtube requests.

Invaluable tool:
1. Wireshark with display filter: http.response !=0 || http.request.method
!= "TRACE" and check the Hypertext Transfer Protocol section for the header
modifications.

Post by Geoffrey Kneale
--- remap.config ---

Post by Leif Hedstrom
youtube.config
regex_map http://(.*) http://$1

1) You shouldn't have to type the full path to the plugin, as long as you
should work.
2) This particular plugin is a bit "strange", in that it is both a remap
plugin, and normal (hooks) plugin. This means for it to function properly,
you also have to add a line to plugins.config, to make sure all the hooks
are activated. I don't know if the documentation specifies this or not?
Cheers,
-- Leif

[Conan Wang]

btw, I'v written a simple remap plugin to do this stuff ( add one header to
origin server ). I modify code in "ats/example/add-header" from "global"
plugin to "remap" plugin, so you don't need to configure plugin.config.

https://github.com/wkl/add_header
(the header "key:value" should not contain space)

Post by Geoffrey Kneale
Thanks Leif!
I have a working config. To note, I also needed to configure the
records.config to NOT force pristine headers.
1. Configure records.config to not force remap and not force pristine
headers.
2. Download and install header filter plugin.
3. Configure plugins to use header filter (global w/o configuration file).
4. Configure the configuration file for the header filter.
5. Configure remap to use the header filter with Youtube requests.
1. Wireshark with display filter: http.response !=0 || http.request.method
!= "TRACE" and check the Hypertext Transfer Protocol section for the header
modifications.

Post by Geoffrey Kneale
--- remap.config ---

Post by Leif Hedstrom
youtube.config
regex_map http://(.*) http://$1

1) You shouldn't have to type the full path to the plugin, as long as you
should work.
2) This particular plugin is a bit "strange", in that it is both a remap
plugin, and normal (hooks) plugin. This means for it to function properly,
you also have to add a line to plugins.config, to make sure all the hooks
are activated. I don't know if the documentation specifies this or not?
Cheers,
-- Leif

[Leif Hedstrom]

Post by Conan Wang
btw, I'v written a simple remap plugin to do this stuff ( add one header
to origin server ). I modify code in "ats/example/add-header" from
"global" plugin to "remap" plugin, so you don't need to configure
plugin.config.
https://github.com/wkl/add_header
(the header "key:value" should not contain space)

Did the header_filter plugin not work? If so, I'm curious why so I can fix
it :).

-- Leif

[Conan Wang]

I didn't try the header_filter plugin. I wrote my plugin because I prefer
"remap" only mode and I can write config in remap.config (for easy
management and reloading support).

And I guess "global" plugin is not faster than "remap" plugin if have many
map rules. Because every request will go through the global part of the
"global" plugin even if we don't want plugin to process that request. (
just my general opinion about global and remap plugin, not for
header_filter plugin)

Anyway, I'm sure the global plugin is fast enough according to my test
before. :)

Post by Leif Hedstrom

Post by Conan Wang
btw, I'v written a simple remap plugin to do this stuff ( add one header
to origin server ). I modify code in "ats/example/add-header" from
"global" plugin to "remap" plugin, so you don't need to configure
plugin.config.
https://github.com/wkl/add_**header <https://github.com/wkl/add_header>
(the header "key:value" should not contain space)

Did the header_filter plugin not work? If so, I'm curious why so I can fix
it :).
-- Leif

[Leif Hedstrom]

Post by James Peach

Post by Alan M. Carroll
I think you would need to write a plugin, but both of those should be easy to do. The plugin would not be very complex (50-100 lines of code). A Type A plugin would be simpler to implement.

Could the header_filter plugin do this?
<http://trafficserver.apache.org/docs/trunk/admin/plugins/header_filter/index.en.html>

It would, unless the header value is "dynamic" in nature (e.g. generated on
the fly for each request).

-- Leif