Nginx proxy pass generating 404 errors

We had an interesting problem from one of our clients today. They were trying to add a proxy pass rule to an existing Nginx based webserver, but the URLs kept generating 404 errors on the server doing the proxying with nothing registering on the server being proxied to. Frustrated after trying various suggestions returned by Internet searches they asked us to take a look.

So they had a simple proxy pass snippet like this:

 
proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;

location /proxyit/ {
  proxy_set_header Host example.com;
  proxy_pass http://example.com:80/;
}

What’s wrong with that, why might it not work? Well nothing is actually wrong with that and it would work in a simple Nginx configration. Much of the advice you’ll see online will tell you to drop or add trailing slashes and they had spent some time trying and trying this again.

The problem in this case actually came from the URLs they wanted to proxy and other items in their Nginx configuration. The URLs were for images and PHP scripts on the example.com server, but their Nginx configuration had other blocks that also dealt with PHP and Images, in this case a PHP block to pass to a FastCGI processor:

 
location ~ \.php$ {
	# ...
}

And another rather innocent block to add some caching headers to images:

location ~* \.(?:js|css|png|jpg|jpeg|gif|ico)$ {
  expires 30d;
  log_not_found off;
}

Now the problem may be more obvious to those familiar with the precedence of location blocks in Nginx, but is frustrating otherwise because there is nothing syntactically wrong with the original proxy pass. All the requests for images and PHP scripts were however being processed by these two blocks in preference to the Proxy pass despite that block apparently matching. This is because regular expression blocks have higher precedence than bare location matches.

The solution? Simple two characters ^~. Changing the proxy location block to:

proxy_redirect off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;

location ^~ /proxyit/ {
  proxy_set_header Host example.com;
  proxy_pass http://example.com:80/;
}

Gave it a higher precedence and now all the Images and PHP URLS starting with /proxyit/ were correctly sent back to the example.com server.