Setting up nginx as reverse proxy on Linux Centos 5.2 (and Plesk 9)

Following my blog article on optimizing your web server by using nginx and memcached, I'll now detail the first step: setting up nginx as reverse proxy on your server. This is going to be a bit tricky, and you'll be getting your hands dirty, so be warned.

What does this consist in?
Well basically, your website will be served by two daemons: nginx for the static content (images, js, css, html...) and Apache for the dynamic content. Nginx will be listening on port 80, will serve static content to visitors, and redirect any dynamic data query to Apache, running on another port -- in our case we'll be using port 8080.

What is nginx?
Nginx is a lightweight open-source http daemon (http server). It is said to be extremely fast, a lot more than Apache, and I have to admit by personal experience this seems to be very true. Using nginx for serving static content dramatically improved the speed of my high traffic website. Actually, some major websites such as Wordpress.com, *cough* Youporn.com, use nginx exclusively for serving web content.

Major issues
This configuration is a bit tricky and can be difficult to achieve particularly if you have numerous domains & subdomains. There are several issues with this configuration:
1) I happen to be using Plesk 9 (admin control panel) for easy domain & subdomain management. Unfortunately it doesn't seem to be compatible with nginx at the moment, it only works with Apache. So we'll run into a few problems very soon.
2) We'll have to work out the configuration files manually (including vhosts - virtual hosts configurations)  so be careful about what you're doing or you might run into annoying problems.
3) Plesk rebuilds the virtual hosts configuration files every time you make the slightest change in the web configuration.

Step 1: download and install nginx
nginx requires the PCRE library. If it's not installed on your system, run the following command:
# yum install pcre
# yum install pcre-devel
Try these libraries aswell just in case they aren't on your system already:
# yum install zlib
# yum install openssl
# yum install openssl-devel
# yum install gcc


Visit the official nginx website (nginx.net) and find the latest version. From your shell, run this command:
# wget http://sysoev.ru/nginx/nginx-0.7.62.tar.gz
Unzip the files with the following command
# tar zxvf ./nginx-0.7.62.tar.gz
Change directory to the nginx folder
# cd nginx-0.7.62
Run the following commands:
# ./configure
If you get no errors, you're all set, go on with the next couple of commands. If you get an error, try to make sure all the libraries are installed.
# make
# make install

2. Nginx base configuration
Congrats if you've made it this far! Now let's have a look at the base configuration of nginx. By default, the main configuration file should be found here: /usr/local/nginx/conf/nginx.conf or /etc/nginx/nginx.conf
Open it up and we'll have a look at some of the settings:
- worker_processes : the amount of processes that will be ran. In most architectures, 1 process = 1 core; so if you want to fully make use of your multi-core CPU, might aswell use as many processes as your CPU has cores. In my case my CPU's a quad-core, so I'll be using 4 worker processes.
- worker_connections : how many connections a process will accepted simultaneously.
For more configuration keys, I suggest having a look here ! Excellent article.
If you can't be arsed, here's the configuration I'll use:
- user apache apache; # might aswell use the same user and user group as apache! this will allow nginx to have read permissions on the same files as apache
- #tcp_nopush on -- leave this commented.
tcp_nodelay on; # to be inserted below #tcp_nopush on;
- keepalive_timeout 30; # should be enough unless your site's really slow
- gzip on; # yeah, why disable it? unless you have a really crappy CPU...
- gzip_proxied any; # proxied requests will also be gzipped
- gzip_comp_level 2; # set gzip compression to 2 (from 1-fast to 9-highly compressed)

There isn't much more to configure here, so we'll start configuring Apache. But before doing so, there's one little additional configuration directive we'll add to nginx.conf, which will make handling virtual hosts a lot easier. In the nginx configuration folder, create a new folder: sites ( /usr/local/nginx/conf/sites/ ).
In the configuration file, below the configuration directives you've put above, insert that new one:
include /usr/local/nginx/conf/sites/*.conf; 


3. Configuring Apache
This is where it'll get dirty. If like me you run Plesk, you probably already have some vhost configuration files all over. You'll have to edit these configuration files one by one, after having modified the main conf file.

- Open your main apache configuration file, probably located somewhere like: /etc/httpd/conf/httpd.conf. Find the "Listen" directive at the beginning of your configuration file. It's probably already set to listen on port 80, so change it to port 8080, and add the line below.
Listen 8080
NameVirtualHost X.X.X.X:8080
Replace X.X.X.X by your actual server IP address. Save and close the file.

- Open your vhosts configuration files one by one, we're going to make some changes. If like me you're using Plesk, the config files for each domain should be located here: /var/www/vhosts/mydomain.com/conf/httpd.include
Replace all references of port 80 by 8080.
Example:
< VirtualHost 49.32.113.160:80 > => < VirtualHost 49.32.113.160:8080 >
ServerName   mydomain.com:80  =>  ServerName   mydomain.com:8080
Do the same for all domains and subdomains that use port 80. We can't allow Apache to use port 80 as it'll be used by nginx! No need to edit the 443 references though, we'll still use Apache for all our https content.

- Once you've edited the configuration files of all your websites, reload your httpd service: service httpd restart.
An error may (or may not) appear upon restarting: [warn] VirtualHost 49.32.113.160:8080 overlaps with VirtualHost 49.32.113.160:8080, the first has precedence, perhaps you need a NameVirtualHost directive.
No need to worry, the fix is simple. Pick one of the vhosts configuration file. Find a vhost directive section such as this: , and add below: NameVirtualHost X.X.X.X:8080 where X.X.X.X is your server's IP address. Save the conf file and reload httpd.

If your httpd reloads without warnings or error, you can proceed to the next section. Otherwise, read carefully the steps I've described above to see if you missed anything.
You can test your changes by accessing your website on port 8080, for example: http://www.mydomain.com:8080/ . Your website should load, even though there might be some display errors due to the port change.

Major issue: when you make any change to the web configuration in Plesk, Plesk rebuilds the vhosts configuration files, which means you'll have to make these changes every time you modify the configuration! There may be some way to prevent this, if you know any, please let me know by posting a comment, I'd be very grateful.

4. Configuring nginx as reverse proxy

So far, we've only installed nginx, and made Apache listen on port 8080 instead of 80. If you stop here, everything's pretty much broken. So read on.
The next step is to configure nginx in order to redirect dynamic content requests to Apache, and return them to the user properly. Start by creating a new file in the nginx configuration folder (same folder as your nginx.conf). Name this file proxy.conf. In this file we'll define the proxy settings. I won't detail each of the settings, this would take ages and you might aswell use the settings below as they should be valid for most sites:
proxy_redirect     off;
proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
client_max_body_size       10m;
client_body_buffer_size    128k;
proxy_connect_timeout      90;
proxy_send_timeout         90;
proxy_read_timeout         90;
proxy_buffer_size          4k;
proxy_buffers               4 32k;
proxy_busy_buffers_size     64k;
proxy_temp_file_write_size 64k;
Credit: papygeek.com
Paste the above lines in the proxy.conf file you've created. We'll be using this file to configure the proxy options in each of our virtual hosts. That's not all though, there is a problem introduced by the proxification of our architecture: how is Apache going to know the real client's IP address? Since nginx will forward the http requests, Apache will be receiving the nginx IP address, in other words, your local IP (your server's IP address). In order to fix this problem, an apache module was created: mod_rpaf.

Begin by installing said module:
# wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
# tar zxvf mod_rpaf-0.6.tar.gz
# cd mod_rpaf-0.6
# make rpaf-2.0 && make install-2.0
If you run apache2, replace "apxs" by "apxs2" in the command below. If apxs/2 isn't installed on your machine, run this command first: yum install httpd-devel
# apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
# a2enmod rpaf

# service httpd restart


5. Configuring the virtual hosts

Let's now see the final part of this tutorial: configuring the virtual hosts for nginx. First go to your "sites" folder, which you created in the nginx configuration folder (default should be /usr/local/nginx/conf/sites/ ). We'll do this the clean way: for each domain hosted on your machine, create a new .conf file.

Here is the configuration for the "mydomain.com" domain (and thus the content of your mydomain.conf) :
server {
  listen      80;
  server_name www.mydomain.com;
  access_log      off; # Apache already creates access logs, I'd disable them unless you really need them
  error_log       /var/log/httpd/nginx.errors.mydomain.com.log warn; # error log, level "warn":
  # Forward requests to Apache! This is the key to our system
  location / {
    proxy_pass         http://www.mydomain.com:8080/;
    include  /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
  }
  # Select files to be deserved by nginx
  location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root  /var/www/vhosts/mydomain.com/httpdocs; # the httpdocs folder of your domain
    expires           7d; # caching, expire after 7 days
  }
}

# same configuration, without the www.
server {
  listen      80;
  server_name mydomain.com;
  access_log      off;
  error_log       /var/log/httpd/nginx.errors.mydomain.com.log warn;
  location / {
    proxy_pass         http://mydomain.com:8080/;
    include  /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
  }
  location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root  /var/www/vhosts/mydomain.com/httpdocs;
    expires           7d;
  }
}


# subdomains! replace "mysubdomain" by your subdomain name

server {
  listen      80;
  server_name mysubdomain.mydomain.com;

    access_log      off;
    error_log       /var/log/httpd/nginx.errors.mysubdomain.mydomain.com.log warn;
  location / {
    proxy_pass         http://mysubdomain.mydomain.com:8080/;
    include  /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
  }

  location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root  /var/www/vhosts/mydomain.com/subdomains/mysubdomain/httpdocs; # httpdocs folder of your subdomain
    expires           7d;
  }
}


Feel free to copy the subdomain section as many times as you have subdomains.
Try your nginx configuration by running the following command:
# /usr/local/nginx/sbin/nginx -t
You should be receiving this message, provided you've done it correctly:
the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
configuration file /usr/local/nginx/conf/nginx.conf test is successful



6. Setting nginx as service & startup
I've written an init.d script for you to use. Nearly all the credit goes to Slicehost for writing the original Ubuntu one; the one I wrote is for CentOS, although it should work for other systems: download here. Unzip it and place it in your /etc/rc/init.d/ folder. If you have placed the nginx binary in a different folder, you'll have to open up the script and change the $DAEMON path. Give it execute permissions ( chmod +x /etc/rc/init.d/nginx ), after which you'll be able to use the following commands:
Starting the server: # service nginx start
Stopping the server: # service nginx stop
Reloading the configuration: # service nginx reload
Restarting the server: # service nginx restart

If you wish, you can add the "service nginx start" command in the /etc/rc.local file, this will allow you service to be ran on startup.

Well, I guess that's about it!
The next article will deal with memcached, so stay tuned!
Clem

Comments

I'd like to add a complementary URL for more configuration examples and (probably better) init scripts: http://wiki.nginx.org/NginxConfiguration
Unknown said…
Hi Clement,

I am writing to you for Packt Publishing, the publishers of computer related books.

We are planning to publish a new book on Nginx Server and currently we are looking out for an author to write it. Your article gives me an impression that you could be a a potential author.

We pay a royalty of 16% and an advance against it. If you find this proposal interesting, please get back to me at kshipras@packtpub.com for a detailed discussion.

Thanks
Kshipra Singh
Author Relationship Manager
Packt Publishing
www.PacktPub.com
Niro said…
Hello Clement,

Thank you very much for the post.

Did you find any solution to the problem that Plesk overwrites the conf files? (I use plesk 8.6).

Thanks,
Niro
alief said…
Thanks a lot, you really saving my day :) I'm searching on google for Plesk setting and finally i found this :)
Eric said…
Although it might not be a huge deal, your RegEx is wrong. You have
^.+.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$

but there should be a backslash before the second dot(.) to make it literal:

^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$
Thanks,
pretty sure I had them in the first place but they might have been filtered somehow. I re-added the backslashes in the regexps, should be fine now.
Anonymous said…
Hello

what about the Webmail ?

in my case webmail stopped working in plesk 9.3+Centos 5

is there some additional i have to do ?

cheers
Well you could try accessing your webmail from http://webmail.website.com:8080 if you are running apache on port 8080... otherwise you need to adjust the webmail configuration for Nginx. I wouldn't know how to do that exactly, well I could but it would probably take me hours/days, which I don't have. good luck! :-)
Anonymous said…
Hi, thanks for your answer

no it is not working under the port 8080 it points to domain it self.

I will try to add it in nginx config as subdomain, but do you know what is the toot path for wbmail/horde ?

and where would be the config file for webmail ?

cheers
Unknown said…
great post... I dont have the nerve to try it just yet. Hope Plesk add this option in the future.

However, I know when I have been messing around with qmail setttings you can prevent plesk from over writting by setting the immuntable flag on a file.

eg chattr +i test_file
Anonymous said…
hello this good document. But im don't start nginx, please help. centos 64 bits and plesk last version.

here before good: 6. Setting nginx as service & startup - but 6. don't start nginx.

and this code problem:


#a2enmod rpaf
-bash: a2enmod: command not found
hi,

it seems that program is not available on your system.
have you done this as written in the article?
"replace "apxs" by "apxs2" in the command below. If apxs/2 isn't installed on your machine, run this command first: yum install httpd-devel"
fabien said…
chattr +i to stop Plesk from resetting your files back to default
thats something very simple that i hadn't considered... just protect your files (make them read only) and plesk wont be able to overwrite them =)) thanks fabien!
Unknown said…
This comment has been removed by the author.
Unknown said…
Hi Clément,

Your post help me to start my nginx serveur into centos! THX!
I've write my own post which include another tips and i think that it can help some people too.
Here it is : http://jv-web.blogspot.fr/2013/06/tuto-nginx-en-revers-proxy-apache.html
Merci Jérôme et bonne continuation pour votre excellent blog!
angelo.:.corbo said…
Excellent stuff Clement.

I’m running nginx on Ubuntu 12.10 with 5 web apps in one box, AND then Apache on OS X (Mac Mini) with a web photo gallery (gallery3) serving a few thousand pictures and videos. I thought about configuring nginx as a reverse proxy for Apache for all the performance benefits it provides. It should be feasible to have Apache handle the gallery app except the actual media (static pictures and videos), which nginx would be ideal to handle, but I’m not sure on the actual details. Do you have any suggestions?

Angelo C
Unknown said…
For other people who do not know what that means reverse proxy I share with

them this post what is reverse proxy
elson cade said…
A week ago I hired ServerBuddies.com to install for me the required and very specific video and audio codecs needed for my website to run properly. They did not only install the required codecs, but they found that I need more requirements installed. After my confirmation, they installed the additional requirements for free. They always give me a nice and professional answer to my emails and they did the job very fast, very accurate and very professional.
FYI, they do offer Server Management, Plesk Support, Server Maintenance, Server Monitoring, Server Troubleshooting and support at a affordable rates. They also provide Server Optimization, Plesk Support, Linux Support, cPanel Support and Plesk Support.
Unknown said…
Thanks for sharing the information. It is very useful for my future. keep sharing
Signature:
Versión en descargar facebook en español a los países hablan Español: facebook entrar agora , facebook en español para and facebook entrar direto

Popular posts from this blog

Affiliate module for Interspire Shopping Cart

How to fix: Outlook on iOS/iPhone: "No Internet Connection"

Dealing with Nginx 400 Bad Request HTTP errors