Alexey Kovyrin's Blog

Yet Another Admin's Blog

Nginx-Fu: X-Accel-Redirect From Remote Servers

Posted by Alexey Kovyrin under Admin-tips, Development, Networks

We use nginx and its features a lot in Scribd. Many times in the last year we needed some pretty interesting, but not supported feature – we wanted nginx X-Accel-Redirect functionality to work with remote URLs. Out of the box nginx supports this functionality for local URIs only. In this short post I want to explain how did we make nginx serve remote content via X-Accel-Redirect.

First of all, here is why you may need this feature. Let’s imagine you have a file storage on Amazon S3 where you store tons of content. And you have an application where you have some content downloading functionality that you want to be available for logged-in/paying/premium users and/or you want to keep track of downloads your users perform on your site. If your content was on your web server, you could have used simple controlled downloads functionality built-in to nginx out of the box. But the problem is that your content is remote.

Here is what we do to solve this problem.

First, we create a special location on our nginx server. This location will be used as a proxy for all our accelerated file downloads:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Proxy download
location ~* ^/internal_redirect/(.*?)/(.*) {
  # Do not allow people to mess with this location directly
  # Only internal redirects are allowed
  internal;

  # Location-specific logging
  access_log logs/internal_redirect.access.log main;
  error_log logs/internal_redirect.error.log warn;

  # Extract download url from the request
  set $download_uri $2;
  set $download_host $1;

  # Compose download url
  set $download_url http://$download_host/$download_uri;

  # Set download request headers
  proxy_set_header Host $download_host;
  proxy_set_header Authorization '';

  # The next two lines could be used if your storage
  # backend does not support Content-Disposition
  # headers used to specify file name browsers use
  # when save content to the disk
  proxy_hide_header Content-Disposition;
  add_header Content-Disposition 'attachment; filename="$args"';

  # Do not touch local disks when proxying
  # content to clients
  proxy_max_temp_file_size 0;

  # Download the file and send it to client
  proxy_pass $download_url;
}

After adding this location to our nginx config we could start sending responses with headers like the following:

1
2
3
4
5
6
7
# This header will ask nginx to download a file
# from http://some.site.com/secret/url.ext and send it to user
X-Accel-Redirect: /internal_redirect/some.site.com/secret/url.ext

# This header will ask nginx to download a file
# from http://blah.com/secret/url and send it to user as cool.pdf
X-Accel-Redirect: /internal_redirect/blah.com/secret/url?cool.pdf

Here is an example code you could use in a Rails application to use our internal redirect location:

1
2
3
4
5
6
7
8
9
10
def x_accel_url(url, file_name = nil)
  uri = "/internal_redirect/#{url.gsub('http://', '')}"
  uri << "?#{file_name}" if file_name
  return uri
end

def download
  headers['X-Accel-Redirect'] = x_accel_url(some_secret_url, pretty_name)
  render :nothing => true
end

As you can see, nginx is really powerful tool and when you turn your creativity on you can make it even more powerful. Stay tuned for more Nginx-Fu posts.

Related posts:

  1. Using X-Accel-Redirect in Nginx to Implement Controlled Downloads
  2. Connecting Two Remote Local Networks With Transparent Bridging Technique
  3. Monitoring nginx Server Statistics With rrdtool
  4. Typical Configurations Overview For Nginx HTTP(S) Reverse Proxy/Web Server
  5. Useful Cacti Templates to Monitor Your Servers

3 Responses to this entry

Evert says:

Why are you not using signed urls? That way you can easily control amazon s3 downloads.

James Miller says:

These are signed URLs. Trying to proxy the file from the (signed) Amazon URL to the user in a way that it rewrites the URL to ours for masking and caching reasons. Some of our files are local, some are on S3 and there’s no reason the user should know about that. Plus, with Amazon’s signed URLs the arguments change every time so it messes up caching.

Leave a Reply