Setting up nginx as reverse proxy on Linux Centos 5.2 (and Plesk 9)

Following my blog article on optimizing your web server by using nginx and memcached, I'll now detail the first step: setting up nginx as reverse proxy on your server. This is going to be a bit tricky, and you'll be getting your hands dirty, so be warned.

What does this consist in?
Well basically, your website will be served by two daemons: nginx for the static content (images, js, css, html...) and Apache for the dynamic content. Nginx will be listening on port 80, will serve static content to visitors, and redirect any dynamic data query to Apache, running on another port -- in our case we'll be using port 8080.

What is nginx?
Nginx is a lightweight open-source http daemon (http server). It is said to be extremely fast, a lot more than Apache, and I have to admit by personal experience this seems to be very true. Using nginx for serving static content dramatically improved the speed of my high traffic website. Actually, some major websites such as Wordpress.com, *cough* Youporn.com, use nginx exclusively for serving web content.

Major issues
This configuration is a bit tricky and can be difficult to achieve particularly if you have numerous domains & subdomains. There are several issues with this configuration:
1) I happen to be using Plesk 9 (admin control panel) for easy domain & subdomain management. Unfortunately it doesn't seem to be compatible with nginx at the moment, it only works with Apache. So we'll run into a few problems very soon.
2) We'll have to work out the configuration files manually (including vhosts - virtual hosts configurations)  so be careful about what you're doing or you might run into annoying problems.
3) Plesk rebuilds the virtual hosts configuration files every time you make the slightest change in the web configuration.

Step 1: download and install nginx
nginx requires the PCRE library. If it's not installed on your system, run the following command:

# yum install pcre

# yum install pcre-devel

Try these libraries aswell just in case they aren't on your system already:

# yum install zlib

# yum install openssl

# yum install openssl-devel

# yum install gcc

Visit the official nginx website (nginx.net) and find the latest version. From your shell, run this command:

# wget http://sysoev.ru/nginx/nginx-0.7.62.tar.gz

Unzip the files with the following command

# tar zxvf ./nginx-0.7.62.tar.gz

Change directory to the nginx folder

# cd nginx-0.7.62

Run the following commands:

# ./configure

If you get no errors, you're all set, go on with the next couple of commands. If you get an error, try to make sure all the libraries are installed.

# make

# make install

2. Nginx base configuration
Congrats if you've made it this far! Now let's have a look at the base configuration of nginx. By default, the main configuration file should be found here: /usr/local/nginx/conf/nginx.conf or /etc/nginx/nginx.conf
Open it up and we'll have a look at some of the settings:
- worker_processes : the amount of processes that will be ran. In most architectures, 1 process = 1 core; so if you want to fully make use of your multi-core CPU, might aswell use as many processes as your CPU has cores. In my case my CPU's a quad-core, so I'll be using 4 worker processes.
- worker_connections : how many connections a process will accepted simultaneously.
For more configuration keys, I suggest having a look here ! Excellent article.
If you can't be arsed, here's the configuration I'll use:
- user apache apache; # might aswell use the same user and user group as apache! this will allow nginx to have read permissions on the same files as apache
- #tcp_nopush on -- leave this commented.

-  tcp_nodelay on; # to be inserted below #tcp_nopush on;


- keepalive_timeout 30; # should be enough unless your site's really slow

- gzip on; # yeah, why disable it? unless you have a really crappy CPU...

- gzip_proxied any; # proxied requests will also be gzipped

- gzip_comp_level 2; # set gzip compression to 2 (from 1-fast to 9-highly compressed)




There isn't much more to configure here, so we'll start configuring Apache. But before doing so, there's one little additional configuration directive we'll add to nginx.conf, which will make handling virtual hosts a lot easier. In the nginx configuration folder, create a new folder: sites ( /usr/local/nginx/conf/sites/ ).

In the configuration file, below the configuration directives you've put above, insert that new one:

include /usr/local/nginx/conf/sites/*.conf; 

3. Configuring Apache
This is where it'll get dirty. If like me you run Plesk, you probably already have some vhost configuration files all over. You'll have to edit these configuration files one by one, after having modified the main conf file.

- Open your main apache configuration file, probably located somewhere like: /etc/httpd/conf/httpd.conf. Find the "Listen" directive at the beginning of your configuration file. It's probably already set to listen on port 80, so change it to port 8080, and add the line below.

Listen 8080

NameVirtualHost X.X.X.X:8080

Replace X.X.X.X by your actual server IP address. Save and close the file.

- Open your vhosts configuration files one by one, we're going to make some changes. If like me you're using Plesk, the config files for each domain should be located here: /var/www/vhosts/mydomain.com/conf/httpd.include
Replace all references of port 80 by 8080.
Example:
< VirtualHost 49.32.113.160:80 > => < VirtualHost 49.32.113.160:8080 >
ServerName   mydomain.com:80  =>  ServerName   mydomain.com:8080
Do the same for all domains and subdomains that use port 80. We can't allow Apache to use port 80 as it'll be used by nginx! No need to edit the 443 references though, we'll still use Apache for all our https content.

- Once you've edited the configuration files of all your websites, reload your httpd service: service httpd restart.
An error may (or may not) appear upon restarting: [warn] VirtualHost 49.32.113.160:8080 overlaps with VirtualHost 49.32.113.160:8080, the first has precedence, perhaps you need a NameVirtualHost directive.
No need to worry, the fix is simple. Pick one of the vhosts configuration file. Find a vhost directive section such as this: , and add below: NameVirtualHost X.X.X.X:8080 where X.X.X.X is your server's IP address. Save the conf file and reload httpd.

If your httpd reloads without warnings or error, you can proceed to the next section. Otherwise, read carefully the steps I've described above to see if you missed anything.
You can test your changes by accessing your website on port 8080, for example: http://www.mydomain.com:8080/ . Your website should load, even though there might be some display errors due to the port change.

Major issue: when you make any change to the web configuration in Plesk, Plesk rebuilds the vhosts configuration files, which means you'll have to make these changes every time you modify the configuration! There may be some way to prevent this, if you know any, please let me know by posting a comment, I'd be very grateful.

4. Configuring nginx as reverse proxy

So far, we've only installed nginx, and made Apache listen on port 8080 instead of 80. If you stop here, everything's pretty much broken. So read on.
The next step is to configure nginx in order to redirect dynamic content requests to Apache, and return them to the user properly. Start by creating a new file in the nginx configuration folder (same folder as your nginx.conf). Name this file proxy.conf. In this file we'll define the proxy settings. I won't detail each of the settings, this would take ages and you might aswell use the settings below as they should be valid for most sites:

proxy_redirect     off;
proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
client_max_body_size       10m;
client_body_buffer_size    128k;
proxy_connect_timeout      90;
proxy_send_timeout         90;
proxy_read_timeout         90;
proxy_buffer_size          4k;
proxy_buffers               4 32k;
proxy_busy_buffers_size     64k;
proxy_temp_file_write_size 64k;

Credit: papygeek.com
Paste the above lines in the proxy.conf file you've created. We'll be using this file to configure the proxy options in each of our virtual hosts. That's not all though, there is a problem introduced by the proxification of our architecture: how is Apache going to know the real client's IP address? Since nginx will forward the http requests, Apache will be receiving the nginx IP address, in other words, your local IP (your server's IP address). In order to fix this problem, an apache module was created: mod_rpaf.

Begin by installing said module:

# wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz

# tar zxvf mod_rpaf-0.6.tar.gz

# cd mod_rpaf-0.6

# make rpaf-2.0 && make install-2.0
If you run apache2, replace "apxs" by "apxs2" in the command below. If apxs/2 isn't installed on your machine, run this command first: yum install httpd-devel

# apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
# a2enmod rpaf

# service httpd restart

5. Configuring the virtual hosts

Let's now see the final part of this tutorial: configuring the virtual hosts for nginx. First go to your "sites" folder, which you created in the nginx configuration folder (default should be /usr/local/nginx/conf/sites/ ). We'll do this the clean way: for each domain hosted on your machine, create a new .conf file.

Here is the configuration for the "mydomain.com" domain (and thus the content of your mydomain.conf) :

server {
  listen      80;
  server_name www.mydomain.com;
  access_log      off; # Apache already creates access logs, I'd disable them unless you really need them
  error_log       /var/log/httpd/nginx.errors.mydomain.com.log warn; # error log, level "warn":
  # Forward requests to Apache! This is the key to our system
  location / {
    proxy_pass         http://www.mydomain.com:8080/;
    include  /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
  }
  # Select files to be deserved by nginx
  location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root  /var/www/vhosts/mydomain.com/httpdocs; # the httpdocs folder of your domain
    expires           7d; # caching, expire after 7 days
  }
}

# same configuration, without the www.

server {
  listen      80;
  server_name mydomain.com;
  access_log      off;

  error_log       /var/log/httpd/nginx.errors.mydomain.com.log warn;

  location / {
    proxy_pass         http://mydomain.com:8080/;
    include  /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
  }
  location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root  /var/www/vhosts/mydomain.com/httpdocs;

    expires           7d;

  }
}

# subdomains! replace "mysubdomain" by your subdomain name

server {
  listen      80;
  server_name mysubdomain.mydomain.com;

    access_log      off;

    error_log       /var/log/httpd/nginx.errors.mysubdomain.mydomain.com.log warn;
  location / {
    proxy_pass         http://mysubdomain.mydomain.com:8080/;
    include  /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
  }

  location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root  /var/www/vhosts/mydomain.com/subdomains/mysubdomain/httpdocs; # httpdocs folder of your subdomain
    expires           7d;
  }
}

Feel free to copy the subdomain section as many times as you have subdomains.
Try your nginx configuration by running the following command:
# /usr/local/nginx/sbin/nginx -t
You should be receiving this message, provided you've done it correctly:
the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
configuration file /usr/local/nginx/conf/nginx.conf test is successful


6. Setting nginx as service & startup
I've written an init.d script for you to use. Nearly all the credit goes to Slicehost for writing the original Ubuntu one; the one I wrote is for CentOS, although it should work for other systems: download here. Unzip it and place it in your /etc/rc/init.d/ folder. If you have placed the nginx binary in a different folder, you'll have to open up the script and change the $DAEMON path. Give it execute permissions ( chmod +x /etc/rc/init.d/nginx ), after which you'll be able to use the following commands:
Starting the server: # service nginx start
Stopping the server: # service nginx stop
Reloading the configuration: # service nginx reload
Restarting the server: # service nginx restart

If you wish, you can add the "service nginx start" command in the /etc/rc.local file, this will allow you service to be ran on startup.

Well, I guess that's about it!
The next article will deal with memcached, so stay tuned!
Clem