Clement Nedelcu's Journal

Chinese OCR: translating scanned or photographied Chinese text to any language

2012-01-23T13:37:00.002+01:00

Having lived in China for almost 3 years now I am able to recognize a good bunch of characters, I can type in Chinese on the computer too but writing is much easier than reading since it doesn't require you to actually memorize the characters, you just type in pinyin (phonetics). That is not enough to understand a full, complex text.
After months of research I've finally figured out how to recognize chinese characters automatically from a picture, in order to copy/paste the text into a translator such as Google translate or others. The solution was right under my eyes all this time: Microsoft Office 2007. I had no idea that Office 2007 came with such features. I've always known of expensive solutions such as Ominpage Pro, but I refused to resort to purchasing the app considering its price and how little I would need it.

OCR, which stands for Optical Character Recognition, is the principle of proceeding to the digital analysis of an image to extract the characters/text that it contains, in order to be able to manipulate the text on a computer.

The solution I describe is for PC/Windows users only. If you're interested on doing the same, real time and in a much simpler way directly from your iPhone, I recommend the excellent Pleco. I've tried out the demo version and am seriously considering purchasing the full version.

TLDR: there are basically four steps involved in the process:

Take a photo with your digital camera, or scan your document with your scanner
(Optional) Convert your photo to a TIF image
Open the TIF image with Microsoft Office Document Imaging, run the OCR
Export the text to Microsoft Word then translate it

This tutorial requires the following software to be installed on your computer:

Microsoft Office 2007 (though this supposedly works with Microsoft Office 2003): installed with "Document Imaging" and "Picture Manager", both are components that you can select during the setup process. If you don't have those two installed on your computer, modify your Office setup to include them.
Chinese language support for Microsoft Office, which isn't exactly something you come across easily. I have the chance to work in China and we have licenses for the Chinese version of Microsoft Office, so I've had no trouble. As an alternative you can get the Microsoft Office 2007 Multi-Language pack and install Chinese support as well as a bunch of other languages if you're interested.

Step 1: Taking a picture or scanning a document

I don't need to remind you how you take pictures with a digital camera. Nor how to save pictures from a website with your favorite web browser. If you are going to use a scanner though, and that is probably the solution that will get you the best results, you can probably skip the next step if your scanner supports saving as TIF/TIFF documents.

To illustrate this tutorial I've chosen to work with a photo taken with my iPhone 4S. It's a document I've taken from a random advertisement booklet found at a friend's place. I tried to get a clear shot of the text to make sure OCR works as accurately as possible.

Step 2: Converting to TIF/TIFF image

Unfortunately, and I must admit I find this quite odd myself, the tool we're going to use for performing the OCR does not support anything other than the TIF format. So if your picture was saved under any other format (JPG typically, like mine) you'll have to convert it. There are plenty of ways to do so.

Since you have Microsoft Office installed on your computer, you should have everything it takes. Right-click your JPG image and "Open with..." - "Microsoft Office Picture Manager". Go to "File" - "Export..." and select the TIF format.

Step 3: Performing the OCR

The actual OCR (Optical Character Recognition) is performed by Microsoft Office Document Imaging. Open the tool, which should be located in your Start Menu under Microsoft Office / Microsoft Office Tools / Microsoft Office Document Imaging.

Before performing the OCR you need to specify the document language. To do so, open the "Tools" menu, go to "Options" - "OCR" and select "Chinese" in the drop-down list. The next steps are simple...

Open the file... Click on the OCR button... Click on the "Send Text to Word" button... press OK and you're done!

The text should be more or less faithfully transcripted depending on the quality of the original picture. Now onto translating it to something actually legible to the average westerner :-)

Step 4: Translation to English or other languages

There are tons of translators out there but I'm going to stick to Microsoft Office since that is what we've been using from the start. Yes, you can translate Chinese directly from within Word 2007 if you follow the simple instructions described below:

Select the text you want translated
Right-click the selected text and in the menu, go to "Translate" - "Translate..."
Select the input and output languages and click the little green arrow
You'll be taken to Microsoft's online translation service, which provides a surprisingly accurate translation of my original text.

Before:

After:

Note: the original document IS about shady management techniques. That's all I had.

Voila, you've successfully translated a document written in another language, based on a simple photo and Microsoft Office. Ah, isn't technology wonderful?

Outlook 2007: replying to multiple emails or senders

2011-11-08T09:54:00.000+01:00

Today, I received about 60 emails from my students who were handing in their assignments. Normally, I would just ask them to assume that I have received their emails correctly, but since we had trouble with the Internet connection, most of my students wanted to make sure I had received their work in due time. So I decided to send a kind confirmation reply to all the students that emailed me their work.

So here I go, opening Microsoft Outlook 2007, selecting multiple emails, right click to find the option "Reply to all" ... horror! It's not there! There's no apparent way to reply to all my students at once. What's worse is there isn't even an option to collect email addresses from people who emailed me (I would then have created one email with all recipients in BCC).

How it works, in a nutshell
After Googling it for about an hour, turns out there is no regular solution to perform such a basic operation. All the solutions I found involved installation of 3rd-party software OR installing and running complex macros. Uh, no thanks. I found some kind of simple workaround that requires no third-party software or macros. It's all in there, using advanced features.

Before I detail each step, here is the principle:

We create a "Message Template". This template will contain the email to be sent to everyone.
We place emails we wish to reply to into a specific category
We create a rule that automatically sends the email template we created in 1) to all the senders of the emails we selected in 2)
We apply that rule and send out the emails.

All of the above four steps are extremely simple and require no technical skills. Read on.

Creating a message template

Follow the simple instructions given below.

Instructions, as described on the image:

1) Write a regular email without a recipient

2) Click "Save As"...

3) Save it on your file system, making sure to select "Outlook Template" as a file type ("Save as type...")

Categorizing emails

The next step is to select the e-mails you want to reply to, and place them into a category.

As you can see, all you have to do is:

1) Select some emails

2) Right-click on the selection

3) Unfold "Categorize" and select a category, for example "Red Category".

Creating a rule

This is the trickiest part so I'll write more detailed instructions. Creating and managing rules is done from the "Rules and Alerts" window. In order to access this window, on the main menu bar of Outlook 2007 go to "Tools" then "Rules and Alerts..."

To create the correct rule, follow the rule creation wizard which shows up when you click the "New Rule" button:

Selecting a Template. Under the "Start from a blank rule", select "Check messages when they arrive". Click next.
Selecting Conditions. Locate the "Assigned to category category" item and check it. As you check the box, a new item appears in the rule description. Click the underlined word "category" to be able to select which category of emails should be replied to. Click next.
Selecting Actions. Locate the option "Reply using a specific template" and check it. As you check the box, a new item appears in the rule description. Click the underlined part "a specific template", which will reveal the template selection window. At the top of this window, in the "Look in" drop-down list, select "User Templates in File System". Then you will be able to browse for the directory containing your template created in the first part of this tutorial. Once your template is selected correctly, click Next.
Selecting Exceptions. If you want to exclude certain emails from getting replied to, select your conditions here. Click Next.
Specify a name for your rule. Below the rule name input box, you will find two checkboxes: "Run this rule now..." which you must enable; it will apply the rule to the folder currently selected in Outlook. Make sure to disable the "Turn on this rule" box, unless you want to automatically reply to similar messages in the future. Click Finish.

Upon clicking Finish, the rule will be applied: all the emails that you categorized will be replied to with the email template you selected previously. Click "Send/Receive" to send out the emails.

Conclusion

Voila! Was it so hard? I don't believe so. But Microsoft should have made it a lot easier to reply to multiple emails, it seems to me that this is quite basic functionality and it should have been available from the start.

Windows 7: restoring old-style logon screen tutorial

2011-10-31T10:38:00.000+01:00

I've found a lot of information on the subject on various blogs and forums, strangely everyone seems to come to the conclusion that it is impossible to get back to the old style logon screen that appeared like this under Windows XP:

Why would anyone do this?
Why would you go back to something like this? There could be thousands of reasons. The default logon screen of Windows 7 looks like this:

If, like me, you work in a computer-unfriendly environment such as a school, you'll find that a lot of people are unable to locate the "Switch user" button or simply wouldn't just dare to click it. All they were told is to enter their username and password and they never expected to have to do anything else.

So how do you restore the good old logon screen with a simple text box for the user name, and a simple text box for the password? It's actually a lot simpler than some people would think.

Local security policy management console
Press Windows+R to get the "Run" dialog, and enter "secpol.msc" in the text box then click OK. You are opening up the Local security policy management console.
In the option tree on the left, go to "Security Settings" / "Local Policies" / "Security options".
In the list of settings on the right, locate "Interactive logon: Do not display last user name". Double click this setting and in the box that shows up, select "Enabled" and click OK. Restart your computer... and voila!
You can also change a couple of other settings, such as disabling the annoying "Ctrl+alt+del" screen, the option right below the one I just described.

Once you're all done you'll be seeing a screen like this when you start up your computer. If you didn't join a domain obviously you won't be seeing the domain part.

Nginx market share soon to hit 10% mark

2011-10-25T06:33:00.000+02:00

According to multiple sources such as:

NetCraft's July 2011 Web Server survey
W3tech's Web server usage statistics as of October 25

the amazing and lightning-fast web server known as nginx is just about to hit the 10% market share mark in the next couple of months. This is a fantastic milestone for the author, Igor Sysoev, who probably didn't imagine that the application (which he originally developed on his own) would meet such fame.

Diagram courtesy of W3Techs.com (article linked above). Its market share went from a little over 5% to almost 10% over the last year as you can see.

I wouldn't go so far as to say that the release of my Nginx HTTP Server book (released Summer 2010) was one of the factors that contributed to the popularity of the Russian web server. It is my belief that the sole quality and efficiency of the software is what made its success.

The book was and is being translated into three languages:

In Japanese
In Chinese (to be published soon)
In Korean (release date unknown)

Nginx is already a popular web server in Russia and Asia; some of the most popular websites in these regions are powered by nginx; namely Yandex, 163 (China's #1 e-mail platform), Soso.com, Renren.com, and more internationally Wordpress, among many others. My own websites GBAtemp.net, FileTrip.net, ShopTemp.net are powered by nginx as well.

PPTP server fix for iOS problems (pptpd/PopTop)

2011-10-13T06:27:00.000+02:00

Ever since I was offered an iPod touch 4th gen I have been unsuccessful in my attempts to connect to PPTP based VPN servers from iOS. Apparently this is a well known issue since iOS 4.3.3 or earlier, that did not get fixed in the iOS 5 update.

Having set up my own PPTP-based VPN servers using Poptop (also known as pptpd) under CentOS, I always found it strange that my servers would function perfectly fine under all versions of Windows, but completely refuse to work under iOS and reportedly MacOS X as well.

Symptoms were the following:

Initially, the connection to the server starts ("Connecting... Starting... Authenticating... ") and appears successful for a second but then immediately drops, with a vague error message.
After a few tweaks that I read on DD-WRT's PPTP server configuration page (pertinent given that DD-WRT uses poptop as well) the situation changed but still failed to solve all of my problems: I was able to connect normally, without any error message, however any network communication failed and timed out. Whenever I tried loading a page it would just keep loading forever. Any app that connects to the Internet did the same- loading forever.

Finally after a combination of multiple tweaks I finally got it to work! The solution is given below but I'd like to credit people first. First I used the DD-WRT tweaks as I previously said, then I followed the tips of two users who posted on this page, sid2 and jeremy207. Massive thanks to both of them!

Here's what got it to work for me. You need to open up the options file for pptpd usually located here: /etc/ppp/options.pptpd
At the very bottom of the file, insert the following lines:

nopcomp
noaccomp
mtu 1400
mru 1400
default-asyncmap

After saving the file, make sure to restart pptpd properly (I stopped it and started it again completely) and try connecting from iOS again. Worked for me! Hope it will for you as well.

How Paypal rips off online businesses

2011-09-02T06:37:00.000+02:00

Paypal has just "resolved" a dispute that a customer filed against our small-scale online business and I am absolutely disgusted by the outcome. Not because of the amount that they've stolen from us (11.25€, we will survive without it), but for what it represents and the inherent risks. I should never have been charged that fee. Let me explain.

A regular purchase?
A few months ago a customer purchased the most expensive VPN plan (72€) on my website VPN1Euro.com. The account was immediately set up, and the customer started using it. A few months later, the customer requested a full refund for the payment (through an automated Paypal dispute), without contacting me prior to sending the request to Paypal. I responded to the dispute by telling the entire truth:

the customer purchased the account legimitately (IP address matches the location entered in the Paypal billing address, nothing indicates that there was some kind of Paypal account fraud)
the customer used the account for a while
the customer never contacted our services to request anything
we offer refunds on prorated basis. Basically if you purchase a 1-year plan, you can get a refund after a while. But if you've already used your account for 2 months, you'll only be refund of the remaining 10 months (why would anyone refund a service that has been duely consumed?). That's what happened here. Had the customer contacted us to obtain a refund we would have sent it without question.

Buyers abusing the system
From the moment the customer filed the dispute, Paypal froze the money (the whole 72€). I replied immediately, but it took over 3 weeks for Paypal to process my response. The outcome was just unveiled to me today: not only the customer is getting ALL of their money back, but I am ALSO being charged for a complementary fee called "settlement fee".

So let's summarize what happened here:

customer purchases expensive VPN plan
customer makes use of VPN plan
two months later, customer gets all of their money back
seller pays an additional 11.25€

Not only have we provided the service correctly, but we are also being charged for the payment reversal! This is completely absurd and Paypal won't budge on the matter.

A flawed system
With such a system any ill-intended customer can just purchase anything, THEN get their money back, THEN have the seller charged for, well, whatever we're being charged for ("settlement fee"). So I'll say it again: the amount lost in this case is obviously neglectable. But what if it happens again? We haven't receive a single cent for a service that was correctly provided, and we've also been charged for it. What can businesses do against such malpractices? The only answer I'm thinking of right now is: don't use Paypal. As if the 4.5% transaction fees weren't enough, they're now stealing money from us. That's great.

China upgrades its Great Firewall to regulate VPNs

2011-05-19T12:40:00.000+02:00

I have lived in China for two years now and I must say I am absolutely disgusted by the new censorship measures enforced by the Chinese government. Especially this new one.

Here's how the story begins. Late April, before the Easter holidays, I was still enjoying live TV streaming from french channels through my regular VPN. It wasn't too fast but good enough to stream correctly without much buffering. When I got back from holiday early May, I noticed that all my VPNs still worked but the speed had become incredibly bad, I could barely access websites - let alone stream live video. I thought it would be temporary or that maybe peering between China Telecom and Europe was saturated.

But it got worse. At work (I work at a foreign school) ever since I got back from holiday, our connection would get shut down every now and then. Nearly every single foreign website (most IP addresses outside of China) would be unreachable for hours. We decided to investigate the issue with our ISP, China Telecom. Unfortunately, it turns out that the reason our connection gets blocked is because the Great Firewall detected that some people in our school were using VPNs.

Today I got to talk with some China Telecom employee (#1 Internet Service Provider in China) and he informed me of the new measures taken by the government to enforce the Internet censorship laws. Here is a summary of the points that I remember hearing from the guy:

the GFW as they call it (GFW for Great FireWall) is controlled by the government up there in Beijing, China Telecom themselves can't do anything about it. All Internet traffic goes through the firewall.
they recently upgraded their systems to take further action to enforce Internet censorship law... sometime between late April and early May.
previously, their systems only enforced a "black list" kind of censorship. They kept a massive list of blocked domains, IP address, keywords, and so on.
since the upgrade, they are able to modify routing tables in order to actually prevent any communication with foreign IP addresses when they detect usage of a VPN.
both campuses of our school are having the same issue, even though they are in completely different areas. It is reported that other businesses/offices/organizations are experiencing the same issue as of late.
when it comes to the general consumer, when usage of VPN is detected, bandwidth gets severely capped. I have no idea how they proceed, but anyhow that's what the China Telecom dude said.

Are you experiencing issues too? I would like to get additional information and testimonies from people all over China. While I am reporting pretty much exactly what the guy said, it could have been bullshit from China Telecom trying to clear their name seeing as Internet in Shanghai is crappy.

As a reminder, the Chinese censorship completely disable access to the following websites (including all related services): Facebook, Youtube, Dailymotion, Twitter, any form of video streaming hosted outside of China, any form of pornographic content, random searches on the Internet, global news websites from mainstream media, and so much more.

I don't know what they are trying to achieve with this. There will always be ways to bypass the "great firewall", network technology always adapts itself. Tunnelling can be done in all sorts of ways, no matter the port, no matter the protocol, no matter the IP address, no matter the encryption method... tunnelling is merely a principle, a concept that they will never be able to block fully. They're just making it a lot less convenient right now, and the speed's worse than before... but still doesn't prevent anyone from accessing blocked sites.

Affiliate module for Interspire Shopping Cart

2010-12-13T10:27:00.000+01:00

Hello,
I have already implemented this module on two shops rather successfully. The concept is quite simple:

websites that want to get rewarded through affiliation must link to your website using an identifier in the URL. If your website is http://website.com/, then your links must be of the form http://website.com/?ref=123 where 123 is the affiliate ID.
when a visitor clicks the link, the affiliate ID will be placed in a cookie that will expire after 15 days (this can be configured)
when the visitor purchases an item: if the affiliate ID cookie is still enabled, the affiliate will be credited of a certain amount, it can be a share of the sale or a fixed amount.
affiliate accounts are simple customer accounts, there is no modification to make regarding accounts. When a sale is made, they receive store credit. They can either spend the store credit to purchase items on the store, or they can manually request for a withdrawal (managing withdrawals is up to you).

I will detail the implementation of this module. It's actually quite simple. You will need:

a tiny modification of the orders SQL table if you want to log individual sales
modifications in a few of the PHP scripts of Interspire Shopping Cart
an additional PHP script that I have provided for download.

Finally I have to mention that I have only applied this on Interspire Shopping Cart 5.x, I've never used it on Interspire Shopping Cart 6 but I don't see any reason why this wouldn't work. You can just try it out and revert the changes if it doesn't work.

Download and install the PHP module

Download the module called affiliate.php. You need to place it in the /lib/ folder of your website. You then need to include the module into Interspire Shopping Cart. To do so, open the /init.php which is at the root of your website. At the end of the file, add this line:

require_once(dirname(__FILE__).'/lib/affiliate.php');

Modification of the SQL table

Begin by modifying the orders SQL table that will allow us to log the sales made by affiliates. The modification is very small and it will NOT affect anything in your store. It will not break anything so don't be worried about it. I've done it multiple times like I said. You need to add a single field at the end of the table: "ordaffiliate". It's an "int" field with a size of at least 10. It will be used to hold the ID of the affiliate who got the sale.

ALTER TABLE `isc_orders` ADD `ordaffiliate` INT( 11 ) NOT NULL COMMENT 'ID of the affiliate who got the sale'

Logging sales properly

Next step is: when someone places an order on the website, we need to store the affiliate ID in the "ordaffiliate" field of the order, and also credit the affiliate for the sale. Open the /includes/class/class.checkout.php file. Find this text in the source code: "$pendingOrder = array" it is not a complete line but you should only get one result.
Before the end of the array, in other words before the ); add this line:

            'ordaffiliate' => isset($_COOKIE["ref"]) ? intval($_COOKIE["ref"]) : "0",

What it does is it checks if the "ref" cookie is set; if it's set then it will save its value in the database. If it's not set it just stores 0. NOTE: if you insert this line as the last line of the array, don't forget to add a comma to the line above! Like this:

            'ordstatus' => $orderStatus,

            'ordaffiliate' => isset($_COOKIE["ref"]) ? intval($_COOKIE["ref"]) : "0"

);

Another modification must be applied to this file: /lib/entities/entity.order.php
When you open it, you will see a massive array of strings like this:

        $schema = array(

                "orderid" => "int",

                "ordtoken" => "text",

...
Anywhere in the array, add this line:

                "ordaffiliate" => "int",

Once that is done, you can get to the next step...

Giving affiliate credit at the right time

You don't want to be giving affiliate credit when someone hasn't paid for their order yet. So we'll only give the credit when the sale is complete. Open /lib/orders.php and find this function: function UpdateOrderStatus.
Scroll down in the function until you find this:

    if (OrderIsComplete($status)) {

            $updatedOrder['orddateshipped'] = time();

}

We are going to change this block a bit and replace it by this:

    if (OrderIsComplete($status)) {

            $updatedOrder['orddateshipped'] = time();

            // Give affiliate credit!

            require_once("affiliate.php");

            if ($order['ordaffiliate'])

                GiveAffiliateCredit($order);

}

We call the "GiveAffiliateCredit" function only if the order is complete/shipped, and if the order has an affiliate ID defined.

At this point the module is functional. Affiliates will receive store credit for the sales they make. Now what you would like is to allow your affiliates to visualize the sales they have made! We're going to need another modification...

Integrating an "affiliate account" link on the account page

The first modification that you have to make will be in the HTML templates of your store design. Open your Admin control panel, and click the "Store design" link at the top right of the page. On the Store Design page, click the "Browse Template Files..." button. Open the "account.html" page by clicking it in the menu on the left. In the account.html page you will find a bullet list (<ul> ... </ul>) corresponding to the links from the account page. Add a new item in the bullet list (<li> tag) like this:

<li><a href='account.php?action=affiliate#affiliate' title='Manage my affiliate account'>Affiliate account</a> - Manage your affiliate account and view your sales</li>

There is another modification you need to make. After the </ul> tag, add this little bit:

%%GLOBAL_AffiliateAccount%%

Save the template and close the window. We are now going to create that affiliate page.

The Affiliate Account page

At this point if you click the link "Affiliate account" that you have inserted on the account page, it won't take you anywhere. It just takes you back to the regular account page. We need to edit the /includes/classes/class.account.php file and make the following modification:

Find the line where it says "switch ($action)" there should be only one in this file
Right below this line, add the following code:

                    case "affiliate": {

                        DisplayAffiliateAccountPage();

                        $this->MyAccountPage();

                        break;

}

To ensure that it works correctly, go to your account page and click the "Affiliate account" link. Here is a screenshot of the Affiliate Account section on the "My Account" page, with one affiliate sale.

Configuring the module

If you open the affiliate.php file that I've provided before, you will find interesting configuration settings at the beginning of the file.
define( "AFFILIATE_ENABLED",             true );     // enable or disable the affiliate module

define( "AFFILIATE_COOKIE_DURATION",     14);         // how many days should the affiliate cookie remain active?

define( "AFFILIATE_COMMISSION",         5 / 100);     // commission that affiliates make when a sale is made. Here it's 5%

define( "AFFILIATE_FIXED_AMOUNT",        10);        // if you want to use a fixed amount IN ADDITION TO THE COMMISSION, define the amount here.

define( "AFFILIATE_REASON", "AFFILIATE-CASHBACK");     // When an order is made, an entry in the "customer_credit" table is added. This is the credit "reason". If you change this after some sales were made, the sales won't be listed (see DisplayAffiliateAccountPage function below)

define( "AFFILIATE_SELF_AFFILIATE",     true);        // If the affiliate account is the same as the customer account (the customer is the affiliate) do we credit the affiliate account?

define( "AFFILIATE_INCLUDE_SHIPPING",     false);     // Include shipping costs in the commission calculation?

Feel free to change the value of each setting according to the affiliate policy you want to implement.

That's it! Your affiliate module is fully ready and functional.
Clem

Get a VPN account for just 1€ a month

2010-12-12T10:44:00.000+01:00

What is a VPN?

A virtual private network (VPN) is a computer network that uses a public telecommunication infrastructure such as the Internet to provide remote offices or individual users with secure access to their organization's network. It aims to avoid an expensive system of owned or leased lines that can be used by only one organization. It encapsulates data transfers between two or more networked devices which are not on the same private network so as to keep the transferred data private from other devices on one or more intervening local or wide area networks. There are many different classifications, implementations, and uses for VPNs.

VPN1EURO.com provides VPN accounts for prices starting at 1€ per month. This is probably the best deal on the web when it comes to ad-free and reliable VPNs!

Check out those VPN accounts for 1€ at VPN1EURO Internet Services! You will be getting an account on a fast, secure and reliable server and enjoy all the benefits offered by their centralized gateway. You can upgrade to VPN Gold Services at any time by contacting their customer support.

Nginx support for Plesk: my offer to Parallels

2010-12-11T17:49:00.000+01:00

Hello everyone.

If like me, you enjoy using Plesk but would like to see Nginx as a web server instead of Apache, you are probably going to want to read the following message. I have emailed Parallels about possible Nginx support in Plesk, offering my help for the implementation (you could say I am qualified for this, being the author of the only book ever written about Nginx).

------------------------------------------------------

Dear Sir,

My name is Clement Nedelcu, technology consultant and university lecturer in Jiangsu (China).
I would like to begin by apologizing because I presume you are not the right person to contact for such an enquiry.
I trust that you will forward this message to the proper person and I thank you in advance for it.

I am a long time Parallels Plesk user. I have recently published a book about a groundbreaking new web server called Nginx.
You will find more information about my book on Amazon: http://www.amazon.com/Nginx-HTTP-Server-Clément-Nedelcu/dp/1849510865
I am certain that the Plesk development team has already heard of Nginx. It is much faster and much more efficient than Apache (the only Linux web server supported by Plesk, as far as I know).

Let’s get to the point. This might sound like a bold request, but since Parallels apparently has no intention of supporting Nginx in future iterations of Plesk, I would like to officially offer my help to the development team towards adding Nginx support to future Plesk versions.

I am not afraid to openly claim that Nginx is contributing to making the Web faster. Unfortunately, major server management panels such as Plesk, CPanel or ISPConfig contribute to holding back the spreading of this little gem of a web server. But many of the most famous websites worldwide have already adopted Nginx:
·         WordPress
·         Rambler
·         Hulu
·         YouPorn
·         GBAtemp
·         MochiMedia
·         …

If you are interested in implementing Nginx support in future (or current) versions of Plesk, I invite you to ask for my help. I am genuinely motivated by the idea of seeing more web servers running Nginx, because it has done a fantastic job at replacing Apache on all my web servers.

I shall hope that you will look into the matter and get back to me on this. I will equally be publishing your answer (or non-answer, for what it’s worth).

Sincerely,
Clement NEDELCU
http://cnedelcu.net

----------------------------------------

I will make sure to post the answer here, if I ever get one.
Post comments if you approve of this. If you disapprove, tell us why!
Clem

Nginx rewrite rules for Interspire Shopping Cart

2010-12-04T09:11:00.001+01:00

It's been a while since I made my last post on this blog, but that's because I've been busy with work! Anyway, today I've chosen to publish a simple finding that I've come up with myself (not that it was any difficult anyway). It could be useful for people who want to run a web shop, particularly the excellent Interspire Shopping Cart. It comes with a set of rewrite rules for Apache to enable search-engine friendly URLs, but nothing for Nginx unfortunately.

Here is the Apache .htaccess file provided with Interspire Shopping Cart. I'm only pasting the section that we are interested in, in other words the Rewrite Module section:

    RewriteEngine On
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule . index.php

In order to achieve the same results in Nginx, you simply need to enable this location block:

        location / {
            try_files $uri $uri/ /index.php?q=$uri&$args;
        }

The try_files directive will attempt to serve the first parameter ($uri, the URI of the client request), then the second one ($uri + slash, could be a folder), but if neither is found, it will redirect the request to index.php and specify the requested URI as parameter in the URL together with the original arguments. It's as simple as that! The PHP scripts will handle the actual rewrites themselves.

Here is the full virtual host configuration which I used on a client's server:

    server {
        listen 80;
        server_name .website.com;
        root /var/www/website.com;
        index index.php;

        location / {
            try_files $uri $uri/ /index.php?q=$uri&$args;
        }

        location ~ \.php$ {
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
        }
    }

IMPORTANT NOTE: when you enable those rewrite rules in Nginx, Interspire does not automatically detect friendly URLs as being "available". So you need to go to your Interspire control panel, in the store settings, and force it to "Enable search-engine friendly URLs" instead of "Enable if available".

Enjoy!

"Nginx HTTP Server" book published

2010-07-22T11:32:00.001+02:00

Nginx HTTP Server was published and is now available for purchase on various locations. It will begin to appear in store in the coming weeks. You can already purchase the eBook from the publisher's website at the link below:
https://www.packtpub.com/nginx-http-server-for-web-applications/book
The eBook can be purchased for 23.79€ whereas the book is 31.49€.
There is also a preview chapter available on the website for anyone to read!
Please feel free to leave your comments, feedback and questions if you have read the book.

Nginx HTTP Server: the book

2010-07-12T07:32:00.002+02:00

Hello,
I am glad to to announce the first Nginx book on the market: Nginx HTTP Server, written by... me. :-)

NGINX HTTP SERVER
"Adopt Nginx for your web applications to make the most of your infrastructure and server pages faster than ever."
Focused on the primary aspect of Nginx (HTTP serving), the book covers the following topics:

Get started with Nginx to serve websites faster and safer
Learn to configure your servers and virtual hosts efficiently
Set up Nginx to work with PHP and other applications via FastCGI
Explore possible interactions between Nginx and Apache to get the best of both worlds
A step-by-step guide to switching from Apache to Nginx
Complete configuration directive and module reference

There are 8 chapters and 3 appendices, listed here:

1. Preparing your work environment: getting ready to work under a command-line environment, with a reminder on the basic tools and commands.
2. Downloading and installing Nginx: downloading the source, the prerequisites, building the application, installing it and learning to solve installation issues.
3. Nginx core configuration: configuring the core modules and optimizing your setup for your current hardware architecture.
4. HTTP configuration: configuring the HTTP core module (virtual hosts, http/server/location blocks...)
5. Module configuration: configuring and using additional modules such as the rewrite module, SSI module and all other first-party modules.
6. Setting up Nginx to work with PHP and Python: learn to use the FastCGI module to set up PHP with Nginx, then Python.
7. Setting up Nginx as a reverse proxy: discovering the Proxy module and setting up a reverse proxied architecture with Nginx as frontend and Apache as backend.
8. Switching from Apache to Nginx: a complete guide to switching from Apache to Nginx, covering everything including converting Apache rewrite rules to Nginx.
Appendix A: full directive index. A list and description of all configuration directives, sorted alphabetically. Module directives are also described in their respective chapters too.
Appendix B: full module reference. A detailed list of all available modules.
Appendix C: troubleshooting. Discusses the most common issues that administrators face when they configure Nginx.

By covering both early setup stages and advanced topics, this book will suit web administrators interested in solutions to optimize their infrastructure, whether they are looking into replacing existing web server software or integrating a new tool cooperating with applications already up and running. If you, your visitors, and your operating system have been disappointed by Apache, this book is exactly what you need.

The book will be available online as eBook and of course in libraries in the major english speaking countries: USA, UK, India, Canada (to be confirmed) and many more. You will find more information and descriptions over at the publisher's website: click here to see the official page about the first Nginx book.

PS: In fact, it appears that there is already a book about Nginx, but it was written in Chinese and never translated and published on the western markets.

Visual Studio 2010 web development bible

2010-07-03T06:13:00.000+02:00

Julien Dollon, whose blog is available here, recently finished writing his book about web development under Visual Studio 2010.

Knowing the professionalism and the talent of Julien, this book should turn out to be an excellent read for all of us web developers interested in the .NET platform. The book is already available for pre-orders on Amazon.
Congratulations Julien!

Nginx & PHP via FastCGI important security issue

2010-05-26T13:40:00.004+02:00

A critical security issue has recently been pointed out on servers that run Nginx and PHP via FastCGI. The issue allows anyone to execute their own PHP code on the system, I don't think I have to remind you of the consequences this could have. I will attempt to provide a simple explanation of the issue and more importantly how to fix it.

What is the issue?
I would like to begin by discussing the nature of the problem: it is not caused by Nginx itself - it is not a bug or a security breach in itself. Actually, it is the way that people usually configure Nginx FastCGI options to work with PHP, and how PHP reacts to that configuration. Pretty much everyone adopts the same configuration without being aware of the issue.

The issue itself can be understood simply, then I will explain why PHP behaves that way. Most dynamic websites allow for a reason or another uploading of files. Say, I'm running a forum-based community, users can upload images to use as personal photo or avatar. The photo gets uploaded and you get the following URL:
http://myforum.com/uploads/photo1234.jpg
The breach consists in appending an additional path element to the URL, making it end in .php:
http://myforum.com/uploads/photo1234.jpg/anything.php

Under certain conditions (and unfortunately with default settings), your photo1234.jpg gets processed as PHP file. So you could upload a PHP script renamed as .jpg, upload the image, then execute the script on the server.

If you want to know instantly if your server is vulnerable to this attack, there is a simple way to know. Find a regular file on your server, such as http://myforum.com/robots.txt. Examine the HTTP headers of the response:
HTTP/1.1 200 OK
Server: nginx/0.7.64
Date: Wed, 26 May 2010 10:56:01 GMT
Content-Type: text/plain
Content-Length: 43
(...)

Now add /test.php after the URL: http://myforum.com/robots.txt/test.php:
HTTP/1.1 200 OK
Server: nginx/0.7.64
Date: Wed, 26 May 2010 10:56:01 GMT
Content-Type: text/plain
Content-Length: 43
(...)
X-Powered-By: PHP/5.2.3

The X-Powered-By header was added by PHP which shows that the file was processed by PHP. Now visit that URL http://myforum.com/robots.txt/test.php in your web browser. What do you see:
- do you see the robots.txt file ? if so, your server is vulnerable.
- do you see an error page (403, 404, 500, 502...) or just a simple message "No input file specified" ? if so, your server is not affected by the problem.

Why does this happen?
There are two main reasons why this happens. First let's have a look at the data Nginx transmits to PHP.
A regular FastCGI/PHP configuration would be as follows:
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /var/www/vhosts/myforum.com/httpdocs$fastcgi_script_name;
fastcgi_param PATH_INFO $fastcgi_script_name;

When requesting an URL like http://myforum.com/uploads/photo1234.jpg/anything.php to Nginx, here is the data that gets sent:
fastcgi_param SCRIPT_FILENAME /var/www/vhosts/myforum.com/httpdocs/uploads/photo1234.jpg/anything.php;
fastcgi_param PATH_INFO /robots.txt/test.php;

So far, no problem. PHP is supposed to load a file anything.php, in the directory /var/www/vhosts/myforum.com/httpdocs/uploads/photo1234.jpg/. Naturally, this directory should not exist, and anything.php shouldn't exist either, so we should be getting a 404 error.
However, that's where the problem comes in. The PHP option cgi.fix_pathinfo, when enabled (and it is usually enabled by default) will transform these two parameters. The SCRIPT_FILENAME becomes /var/www/vhosts/myforum.com/httpdocs/uploads/photo1234.jpg, which means the .jpg file actually becomes the request filename, and it gets treated as PHP. And PATH_INFO becomes /anything.php. The original purpose of this option was to allow such kind of URLs: index.php/param1/param2/...
But when combined with Nginx, this turns into a major issue.

How do I fix it?
Well, the simplest thing you can do is open up your php.ini configuration file, and insert this directive in the main section:
cgi.fix_pathinfo=0
Then restart PHP-FPM or whatever FastCGI manager you're using.

Unfortunately in some cases that is not possible a solution, since perhaps other scripts on your server make the most of this option. So you could do mainly employ two different solutions on the Nginx side.

First, you could check that the requested URI actually exists, before passing the request via FastCGI:
location \.php$ {
    if (!-f $request_filename) {
        return 404;
    }
    fastcgi_pass 127.0.0.1:9000;
    [...]
}

This solution is efficient and a few of us Nginx+PHP have retained it.
Otherwise, if you think it's too consuming in terms of resources, you could check the URI to meet the following requirements:
- if the URI contains a dot, then a slash (example: image.jpg/...)
- if the URI ends with ".php" (example: image.jpg/test.php)
- then return a 403 error.
location ~ \..*/.*\.php$ {
    return 403;
}
location ~ \.php$ {
    fastcgi_pass 127.0.0.1:9000;
    ...
}

Alternatively, you could make sure that PHP is only enabled in certain directories, where file uploads are not allowed:
location ~ ^/(scripts|sources|src)/.*\.php$ {
    fastcgi_pass 127.0.0.1:9000;
    ...
}

Thanks for reading. And if you find this vulnerability on servers that do not belong to you, contact the server administrator immediately to report the problem!

The problem was discovered here: http://www.80sec.com/nginx-securit.html
And discussed here: http://www.pubbs.net/201005/nginx/39767-nginx-0day-exploit-for-nginx-fastcgi-php.html

Thanks to Martin F. for reporting the issue!

Dealing with Nginx 400 Bad Request HTTP errors

2010-05-09T18:34:00.000+02:00

Today I'll write about something I experienced personally, on my websites.
Some visitors reported that they were getting a "400 Bad Request" Nginx error randomly when visiting pages. And when they start getting that error, they can't access the site anymore: it'll output the same error no matter the page, until you "clear your cache and cookies".

The error is easily understandable and is likely to be caused by... too much cookie data.
Every time a visitor loads *any* page/content/file of your website, it sends the cookie data to the server.
Cookie data is sent under the form of 1 header line starting with "Cookie: ".

Basically, Nginx by default is configured to accept header lines of a maximum size of 4 kilobytes.
When a line in the headers exceeds 4 kilobytes, Nginx returns the '400 Bad Request' error.
Cookie data sometimes gets big, so it causes the error. It particularly happens on forums like vBulletin, Invision and others.
So why does it happen only for some web browsers (Firefox, Chrome...) and not others? Because those browsers do not limit the amount of data a cookie may store. Or maybe they do, but the limit is higher than the default 4k of Nginx. Other browsers limit the amount of cookie data so they do not have the issue.

There is a simple fix for that. The large_client_header_buffers directive of Nginx allows you to define size of buffers that will contain large headers like those big fat cookies.

The directive specifies: the amount of buffers, and the size of buffers. You basically need to increase the size.
In your http block (or your server block, if you want to apply the setting at the virtual host level), insert this directive with a size larger than 4k (actually the default size can be 8k depending on your system, so let's make it... 16k):

http {

   [...]

   large_client_header_buffers 4 16k;

   [...]

}

Save your configuration, reload nginx by running /usr/local/nginx/sbin/nginx -s reload and it should now be fine. If you ever get the "400 Bad Request" again, you could either increase this value once more or look into the code and see why cookies get so big.

Downloading MMS streams in Linux (CentOS, Ubuntu, Debian, Fedora...) with mmsclient

2010-05-03T05:33:00.000+02:00

Hello!

So I was looking into solutions for saving a MMS stream on my server. My connection at home isn't fast enough so I cant watch most of the streams, I'd rather have those downloaded by my dedicated server and then I download the file off my server.

I first looked into mimms: http://savannah.nongnu.org/download/mimms/
Unfortunately it was written in Python and I was missing Python 2.5. After struggling with my system I couldn't get it to work (missing dependencies one after the other).

So I kept looking and I found mmsclient.
The official website: http://ole.tange.dk/projekter/kontroversielt/www.geocities.com/majormms/ (actually a copy of the site, which was hosted on Geocities previously).
Scroll down to the bottom of the page where it says "mmsclient". You will find a link to download it. I've mirrored the link here just in case: http://gbatemp.net/up/mms_client-0.0.3.tar.gz

To install mmsclient, follow these simple steps:
1) Download the package:

wget http://gbatemp.net/up/mms_client-0.0.3.tar.gz

2) Extract it:

tar xzvf mms_client-0.0.3.tar.gz

3) Configure, make, make install:

cd mms_client-0.0.3

./configure

make && make install

That's it! Now you can download mms streams. Just use the "mmsclient" command, followed by the URL of the stream you want to download:

mmsclient mms://example.com/stream.wmv

Additionally, you will notice that it outputs a lot of data to the command line window. To make it quiet, add >/dev/null at the end:

mmsclient mms://example.com/stream.wmv >/dev/null

Enjoy your downloaded streams!

Setting up and using memcached & memcache on Linux CentOS 5/Plesk 9

2009-09-30T09:03:00.001+02:00

The second part of my tutorial on optimizing your server for hosting high traffic websites: installing and configuring memcached and the memcache php extension for your server. This is a little easier than the first step (setting up nginx as reverse proxy, see article below) and can be applied to any kind of dynamic website.

What's the point ?
On highly dynamic websites such as forums, news sites or any user content based website, the database server load is often very high. The more traffic you get, the more cluttered your database server becomes, sometimes rendering your website completely unavailable to visitors. Using a data caching daemon will allow you to save some data in memory instead of fetching the data from the database every time. You should know that memcached is used by major websites such as Wikipedia, SourceForge, SlashDot... need I say more?

What is memcached ?
Memcached is the daemon running on your server. Its usage is extremely simple, there are no configuration files, all you do is start the daemon on a given port, and your websites will connect to this daemon to store data in memory. Yes, the data is stored in your RAM. So when starting memcached you'll have to decide how much RAM memcached will be allowed to use. If you start memcached with a 1GB memory space, memcached will store this much data; when the cache is full some of the older data will begin to disappear from the cache.

What is memcache ?
Memcache, in our case, is the PHP extension that will allow us to connect to and make use of Memcached. This PHP extension is not part of the default ones so you'll have to download and install it (see step 2). It provides classes as well as functions that I must admit are very easy to use and understandable. In this article, I provide a mysql+memcache wrapper class for anyone to use.

What is the difference between memcached and memcache ?
Well if you've read the two points above, you should already know. In short, memcached is the daemon running on your machine; memcache is the PHP extension allowing you to make use of memcached.

1. Setting up memcached

I haven't found memcached in my repositories (might aswell try # yum install memcached just in case?) so I'll download the source and compile it. First go get the latest version from the official website.

# wget http://memcached.googlecode.com/files/memcached-1.4.1.tar.gz

# tar zxvf memcached-1.4.1.tar.gz

# cd memcached-1.4.1

# ./configure

If like me you get this message "libevent is missing" or something, you can run this command:

# yum install libevent-devel

And then run configure again:

# ./configure

Install memcached:
# make install
That's it, you're set! That was pretty easy wasn't it? We'll now see the command line arguments for starting memcached:
# memcached -d -m 1024 -l 127.0.0.1 -p 11211 -u nobody
The arguments are:
-d : start as daemon, running in the background
-m 1024: allow memcached to use up to 1024 MB of RAM (1GB)
-l 127.0.0.1: listen on local interface
-p 11211: listen on port 11211
-u nobody: run as user "nobody"

If you're not sure how much memory you should allocate to memcached, try running this command first:

# free

It will tell you how much free RAM you've got left.
Note that upon starting memcached, if all is OK, you will see no output message. To see if memcached is correctly started, run this command:

# ps aux | grep memcached

You should be seeing something like this:

nobody 13133 0.0 0.0 43580 732 ? Ssl 07:11 0:00 memcached -d -m 128 -l 127.0.0.1 -p 11211 -u nobody
user 13143 0.0 0.0 4152 648 pts/0 R+ 07:11 0:00 grep memcached

2. Setting up memcache PHP extension
The memcache PHP extension should be found in the classic repositories, so try this command:

# yum install php-pecl-memcache

If you're lucky (why should you be unlucky anyway?) the install will work fine and you'll be seeing these messages:
Installed: php-pecl-memcache.i386 0:2.2.3-1.el5_2
Dependency Installed: php-pear.noarch 1:1.4.9-4.el5.1
Complete!
Just for reference, here's a link to the official memcache website, if you need to grab the sources.

Let's see if memcache was installed properly. First restart the httpd:
# service httpd restart
Then place a simple php file on your website containing the following code:

phpinfo();  

Open the PHP file in your browser (eg. http://mydomain.com/phpinfo.php ) and have a look at the output. If you can find a "memcache" section looking like the following picture, it means memcache was successfully installed.

We will now have a look at the memcache configuration file. First locate your php module configuration files folder, in my case /etc/php.d/ . You should find the newly installed "memcache.ini" configuration file. Open it up to see a list of configuration keys and their meaning.

The default options are just fine, but if you're interested, you should know that memcache offers load-balancing features through the "allow_failover" configuration key. I'm not going to make use of this feature so I will not be editing any of the settings.

3. Using memcache in your code
Unfortunately, installing both components isn't enough. You'll have to edit your code in order to make use of the caching features. Be reassured though, it couldn't be easier! There are a couple of functions you'll need to use, nothing complex.
If you want to find out the complete listing of the memcache php functions, visit the official website. Basically we'll be using 5 methods:
- Memcache::connect($host, $port, $timeout): connect to your daemon
- Memcache::get($key) : fetch data from your cache
- Memcache::set($key, $var, $flag, $expire): store data in your cache
- Memcache::delete($key): remove data from your cache
- Memcache::close(): disconnect.

You can cache any data that you want:

$mc = new Memcache;

$mc->connect("localhost", 11211);

$saved_data = $mc->get("saved_data");

if (!$saved_data) {

  $saved_data = file_get_contents("myfile.txt");

  $mc->set("saved_data", $saved_data, MEMCACHE_COMPRESSED, 60*60*24*7); // store for 7 days

}

echo $saved_data;

$mc->close();

Applied to MySQL queries:

$mc = new Memcache;

$mc->connect("localhost", 11211);

$news_articles = $mc->get("news_articles");

if (!$news_articles) {

  $news_articles = array();

  $query = "SELECT * FROM news_articles ORDER BY article_id DESC LIMIT 0,10";

  $result = mysql_query($query);

  while($row = mysql_fetch_assoc($result)) $news_articles[] = $row;

  $mc->set("news_articles", serialize($news_articles), MEMCACHE_COMPRESSED, 60*60*24*7); // store for 7 days, but don't forget to rebuild the cache when a new article is posted!

} else {

  $news_articles = unserialize($news_articles);

}

// Display articles.. 

$mc->close();
As you can see in the example above, I use the "serialize" and "unserialize" php functions. Why is that? The reason is because the Memcache::get() function always returns a string. So if you want to store an array of data (or an object), you'll have to serialize said array, and unserialize it after having read it from the database.
If you know a better workaround for this problem please feel free to leave a comment.

4. Wrapper class for memcache & mysql
I have just written a simple wrapper class for MySQL, making use of the powerful caching system offered by memcache. You can download the class here, I included a simple example for testing the class.
The principle is very simple: when executing a query, the script will check if the query result is already in the cache. If the data is in the cache, it is returned immediatly (no query executed). if the data is not in the cache, the query is executed, and the results are then placed in the cache with the specified "time to live".

Here are the wrapper class functions:
function MySQLMemcache($mysql_info, $memcache_info, $autoconnect=true, $enable_logging=true);
function connect();
function disconnect();
function dataQuery($query, $usecache=true, $ttl=0);
function nonDataQuery($query);
function fieldDataQuery($query, $field, $usecache=true, $ttl=0);

More documentation is provided inside the actual php file.

Thank you for reading, feel free to leave a comment if this article has been helpful to you.
I'm finished with my server optimization thematic.
TTFN!

Setting up nginx as reverse proxy on Linux Centos 5.2 (and Plesk 9)

2009-09-29T16:33:00.001+02:00

Following my blog article on optimizing your web server by using nginx and memcached, I'll now detail the first step: setting up nginx as reverse proxy on your server. This is going to be a bit tricky, and you'll be getting your hands dirty, so be warned.

What does this consist in?
Well basically, your website will be served by two daemons: nginx for the static content (images, js, css, html...) and Apache for the dynamic content. Nginx will be listening on port 80, will serve static content to visitors, and redirect any dynamic data query to Apache, running on another port -- in our case we'll be using port 8080.

What is nginx?
Nginx is a lightweight open-source http daemon (http server). It is said to be extremely fast, a lot more than Apache, and I have to admit by personal experience this seems to be very true. Using nginx for serving static content dramatically improved the speed of my high traffic website. Actually, some major websites such as Wordpress.com, *cough* Youporn.com, use nginx exclusively for serving web content.

Major issues
This configuration is a bit tricky and can be difficult to achieve particularly if you have numerous domains & subdomains. There are several issues with this configuration:
1) I happen to be using Plesk 9 (admin control panel) for easy domain & subdomain management. Unfortunately it doesn't seem to be compatible with nginx at the moment, it only works with Apache. So we'll run into a few problems very soon.
2) We'll have to work out the configuration files manually (including vhosts - virtual hosts configurations) so be careful about what you're doing or you might run into annoying problems.
3) Plesk rebuilds the virtual hosts configuration files every time you make the slightest change in the web configuration.

Step 1: download and install nginx
nginx requires the PCRE library. If it's not installed on your system, run the following command:

# yum install pcre

# yum install pcre-devel

Try these libraries aswell just in case they aren't on your system already:

# yum install zlib

# yum install openssl

# yum install openssl-devel

# yum install gcc

Visit the official nginx website (nginx.net) and find the latest version. From your shell, run this command:

# wget http://sysoev.ru/nginx/nginx-0.7.62.tar.gz

Unzip the files with the following command

# tar zxvf ./nginx-0.7.62.tar.gz

Change directory to the nginx folder

# cd nginx-0.7.62

Run the following commands:

# ./configure

If you get no errors, you're all set, go on with the next couple of commands. If you get an error, try to make sure all the libraries are installed.

# make

# make install

2. Nginx base configuration
Congrats if you've made it this far! Now let's have a look at the base configuration of nginx. By default, the main configuration file should be found here: /usr/local/nginx/conf/nginx.conf or /etc/nginx/nginx.conf
Open it up and we'll have a look at some of the settings:
- worker_processes : the amount of processes that will be ran. In most architectures, 1 process = 1 core; so if you want to fully make use of your multi-core CPU, might aswell use as many processes as your CPU has cores. In my case my CPU's a quad-core, so I'll be using 4 worker processes.
- worker_connections : how many connections a process will accepted simultaneously.
For more configuration keys, I suggest having a look here ! Excellent article.
If you can't be arsed, here's the configuration I'll use:
- user apache apache; # might aswell use the same user and user group as apache! this will allow nginx to have read permissions on the same files as apache
- #tcp_nopush on -- leave this commented.

tcp_nodelay        on; # to be inserted below #tcp_nopush on;

- keepalive_timeout 30; # should be enough unless your site's really slow

- gzip on; # yeah, why disable it? unless you have a really crappy CPU...

- gzip_proxied any; # proxied requests will also be gzipped

- gzip_comp_level 2; # set gzip compression to 2 (from 1-fast to 9-highly compressed)

There isn't much more to configure here, so we'll start configuring Apache. But before doing so, there's one little additional configuration directive we'll add to nginx.conf, which will make handling virtual hosts a lot easier. In the nginx configuration folder, create a new folder: sites (

/usr/local/nginx/conf/sites/ ).

In the configuration file, below the configuration directives you've put above, insert that new one:

include /usr/local/nginx/conf/sites/*.conf;

3. Configuring Apache
This is where it'll get dirty. If like me you run Plesk, you probably already have some vhost configuration files all over. You'll have to edit these configuration files one by one, after having modified the main conf file.

- Open your main apache configuration file, probably located somewhere like: /etc/httpd/conf/httpd.conf. Find the "Listen" directive at the beginning of your configuration file. It's probably already set to listen on port 80, so change it to port 8080, and add the line below.

Listen 8080

NameVirtualHost X.X.X.X:8080

Replace X.X.X.X by your actual server IP address. Save and close the file.

- Open your vhosts configuration files one by one, we're going to make some changes. If like me you're using Plesk, the config files for each domain should be located here: /var/www/vhosts/mydomain.com/conf/httpd.include
Replace all references of port 80 by 8080.
Example:
< VirtualHost 49.32.113.160:80 > => < VirtualHost 49.32.113.160:8080 >
ServerName mydomain.com:80 => ServerName mydomain.com:8080
Do the same for all domains and subdomains that use port 80. We can't allow Apache to use port 80 as it'll be used by nginx! No need to edit the 443 references though, we'll still use Apache for all our https content.

- Once you've edited the configuration files of all your websites, reload your httpd service: service httpd restart.
An error may (or may not) appear upon restarting: [warn] VirtualHost 49.32.113.160:8080 overlaps with VirtualHost 49.32.113.160:8080, the first has precedence, perhaps you need a NameVirtualHost directive.
No need to worry, the fix is simple. Pick one of the vhosts configuration file. Find a vhost directive section such as this: , and add below: NameVirtualHost X.X.X.X:8080 where X.X.X.X is your server's IP address. Save the conf file and reload httpd.

If your httpd reloads without warnings or error, you can proceed to the next section. Otherwise, read carefully the steps I've described above to see if you missed anything.
You can test your changes by accessing your website on port 8080, for example: http://www.mydomain.com:8080/ . Your website should load, even though there might be some display errors due to the port change.

Major issue: when you make any change to the web configuration in Plesk, Plesk rebuilds the vhosts configuration files, which means you'll have to make these changes every time you modify the configuration! There may be some way to prevent this, if you know any, please let me know by posting a comment, I'd be very grateful.

4. Configuring nginx as reverse proxy

So far, we've only installed nginx, and made Apache listen on port 8080 instead of 80. If you stop here, everything's pretty much broken. So read on.
The next step is to configure nginx in order to redirect dynamic content requests to Apache, and return them to the user properly. Start by creating a new file in the nginx configuration folder (same folder as your nginx.conf). Name this file proxy.conf. In this file we'll define the proxy settings. I won't detail each of the settings, this would take ages and you might aswell use the settings below as they should be valid for most sites:

proxy_redirect     off;
proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
client_max_body_size       10m;
client_body_buffer_size    128k;
proxy_connect_timeout      90;
proxy_send_timeout         90;
proxy_read_timeout         90;
proxy_buffer_size          4k;
proxy_buffers               4 32k;
proxy_busy_buffers_size     64k;
proxy_temp_file_write_size 64k;

Credit: papygeek.com
Paste the above lines in the proxy.conf file you've created. We'll be using this file to configure the proxy options in each of our virtual hosts. That's not all though, there is a problem introduced by the proxification of our architecture: how is Apache going to know the real client's IP address? Since nginx will forward the http requests, Apache will be receiving the nginx IP address, in other words, your local IP (your server's IP address). In order to fix this problem, an apache module was created: mod_rpaf.

Begin by installing said module:

# wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz

# tar zxvf mod_rpaf-0.6.tar.gz

# cd mod_rpaf-0.6

# make rpaf-2.0 && make install-2.0
If you run apache2, replace "apxs" by "apxs2" in the command below. If apxs/2 isn't installed on your machine, run this command first: yum install httpd-devel

# apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
# a2enmod rpaf

# service httpd restart

5. Configuring the virtual hosts

Let's now see the final part of this tutorial: configuring the virtual hosts for nginx. First go to your "sites" folder, which you created in the nginx configuration folder (default should be /usr/local/nginx/conf/sites/ ). We'll do this the clean way: for each domain hosted on your machine, create a new .conf file.

Here is the configuration for the "mydomain.com" domain (and thus the content of your mydomain.conf) :

server {
listen      80;
server_name www.mydomain.com;
access_log      off; # Apache already creates access logs, I'd disable them unless you really need them
error_log       /var/log/httpd/nginx.errors.mydomain.com.log warn; # error log, level "warn":
# Forward requests to Apache! This is the key to our system
location / {
    proxy_pass         http://www.mydomain.com:8080/;
    include /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
}
# Select files to be deserved by nginx
location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root /var/www/vhosts/mydomain.com/httpdocs; # the httpdocs folder of your domain
    expires           7d; # caching, expire after 7 days
}
}

# same configuration, without the www.

server {
listen 80;
server_name mydomain.com;
access_log off;

error_log /var/log/httpd/nginx.errors.mydomain.com.log warn;

location / {
    proxy_pass         http://mydomain.com:8080/;
    include /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
}
location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root /var/www/vhosts/mydomain.com/httpdocs;

expires 7d;

}
}

# subdomains! replace "mysubdomain" by your subdomain name

server {
listen 80;
server_name mysubdomain.mydomain.com;

access_log off;

    error_log       /var/log/httpd/nginx.errors.mysubdomain.mydomain.com.log warn;
location / {
    proxy_pass         http://mysubdomain.mydomain.com:8080/;
    include /usr/local/nginx/conf/proxy.conf; # the proxy.conf file
}

location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico|txt|srt|swf|zip|rar|html|htm|pdf)$ {
    root /var/www/vhosts/mydomain.com/subdomains/mysubdomain/httpdocs; # httpdocs folder of your subdomain
    expires           7d;
}
}

Feel free to copy the subdomain section as many times as you have subdomains.
Try your nginx configuration by running the following command:
# /usr/local/nginx/sbin/nginx -t
You should be receiving this message, provided you've done it correctly:
the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
configuration file /usr/local/nginx/conf/nginx.conf test is successful

6. Setting nginx as service & startup
I've written an init.d script for you to use. Nearly all the credit goes to Slicehost for writing the original Ubuntu one; the one I wrote is for CentOS, although it should work for other systems: download here. Unzip it and place it in your /etc/rc/init.d/ folder. If you have placed the nginx binary in a different folder, you'll have to open up the script and change the $DAEMON path. Give it execute permissions ( chmod +x /etc/rc/init.d/nginx ), after which you'll be able to use the following commands:
Starting the server: # service nginx start
Stopping the server: # service nginx stop
Reloading the configuration: # service nginx reload
Restarting the server: # service nginx restart

If you wish, you can add the "service nginx start" command in the /etc/rc.local file, this will allow you service to be ran on startup.

Well, I guess that's about it!
The next article will deal with memcached, so stay tuned!
Clem

The final solution: nginx+apache2 and memcached

2009-09-26T15:18:00.002+02:00

Been a while!
But I'll make up for the huge time gap: this post will probably be one of the most useful I'll ever post.

I happen to be running a high traffic website, have been running it for about 5 years now. Over the past few years though, my website has known a major traffic increase which resulted in my servers being regularly cluttered and my website inaccessible. My website profile: an Invision Powerd Board based website (heavily modded though), running under PHP 5 and MySQL 5. Servers are hosted in France at OVH.com.
At first, my reasoning was quite simple: spend more money on a more powerful server. I ran about 5 or 6 server upgrades over the years. I must say it worked at first, since I was running low-end servers. But for the last couple of months the traffic became way too high, which resulted in my website being completely inaccessible for a part of the world (for visitors in remote countries such as Canada, connections frequently timed out) and just plain slow for everyone else. At that time the traffic was: nearly 60K unique visitors/day, about 10 million visits/month.
My server setup: a quad-core with 8GB of ram for Apache, and a quad-core with 4GB of ram for MySQL, both using SATA2 RAID0 HDDs. Connected to eachother with a 1 Gb link.

Well, I've finally settled for a solution that seems to be working great. The website's fast for everyone, even for me over there in China.

1. Running PHP with Fast-CGI

My website is a community based website, which means the site is strongly dynamic. Every page served is PHP, strictly no HTML. The "default" option for serving PHP is to use PHP as an Apache module. The problem with this solution is that for every page served, a new Apache/httpd process has to be loaded in memory. With high traffic website this isn't necessarily a good solution especially if your server doesn't have much RAM.
So the first thing I did was to switch PHP from Apache module to Fast-CGI module.

Those tutorials should help you set up PHP as fast-cgi module for Apache:
[english] http://www.fastcgi.com/drupal/node/5?q=node/10
[french] http://www.sos-dedie.com/2009/01/15/apache-2-worker-et-php-fastcgi/

2. Setting up nginx for serving static content

Apache will be serving dynamic content via PHP as FastCGI module. But on top of Apache, we'll be using another webserver, a very lightweight one, for serving static content. Basically, this means php pages will be served by Apache, but other content (images, javascript, css, static html...) is served by nginx, which is extremely fast and reliable for such things.
How to put such a set up in production? Simple as that: nginx on port 80, Apache on port 8080 (or another) and nginx is configured to redirect all dynamic content to Apache. It's called "using nginx as reverse-proxy".

Here are a few articles about it:
[english] http://kovyrin.net/2006/05/18/nginx-as-reverse-proxy/
[english] http://wiki.joyent.com/accelerators:nginx_apache_proxy
[french] http://www.papygeek.com/software/optimiser-son-serveur-web-avec-nginx/
Both english articles don't cover vhosts issues, so if I ever get comments asking me how to proceed, I'll post a new article about it.

3. Memcached and memcache

I had a bit of trouble figuring that one out as I couldn't really find any article explaining the difference between memcache and memcached. So here's the deal.
- Basically "memcached" (note the trailing "d", which stands for "daemon") is a process that runs on your machine and that allows you to easily cache data in memory--RAM. It's basically a simple and efficient cache manager. It listens on a given port and you can connect to it via...
- memcache: this is the name of the PHP module that allows you to make use of memcached. You're going to need to install this because it doesn't come with PHP! Memcached and memcache can be found in the usual repositories (eg. rpmforge)

[english] http://www.danga.com/memcached/ < Memcached official website
[english] http://www.php.net/memcache < Memcache (PHP module) official website

Installing and configuring both isn't the only thing you have to do. You're going to have to make use of the memcache PHP module functions. That's the trick! But I'll guide you through it.

Here are a couple of handy functions:
memcache_connect ($host, $port, $timeout) : connects to the memcache server you've set up on your machine.
memcache_get ($key) : gets a string from the cache. Returns null if the string was not found.
memcache_set ($key, $data, $flag, $ttl) : save a string into the cache.
memcache_delete ($key) : deletes the string from the cache

Now how to use these functions: this couldn't be any simpler.
- Begin by connecting to the memcached server using memcache_connect()
- Before running any SQL query, ask yourself: can this query be cached? In theory, most queries can be. In my case, I used the memcache functions to optimize my portal page (index.php) which is basically a simple news article display. In other words, the content almost never changes, so this kind of query can definitely be cached.
Here is a simple code example:

// Retrieve data from cache
$articles = memcache_get( "news_articles");

// if $articles is null, it means the data isn't in the cache store
if (!$articles) {
$articles = array();
$result = mysql_query("SELECT * FROM news ORDER BY news_id DESC LIMIT 0,10");
while ($row = mysql_fetch_assoc($result)) $articles[] = $row;

// Save the data into the cache.
// Since memcache_get can only return a string, you'll have to serialize the data before saving it into the cache.
// The data is saved for 1 week as defined with the last parameter.
memcache_set( "news_articles", serialize($articles), MEMCACHE_COMPRESSED, 60*60*24*7);
}
// $articles is not null
else {
$articles = unserialize($articles); // unserialize the data
}
// You now have a fully loaded $articles array, ready for display!

Conclusion

I managed to reduce the amount of SQL queries of my main page from an average of 15 to... 5. Every single page of my website loads nearly instantly even during high influx of visitors. Lately I had about 3000 users online simultaneously, and I didn't notice any slowdowns.
So I can safely say that these 3 points described above actually solved all my issues.

What a relief!
Clem

Visual Studio 2008 Survey

2008-12-10T17:52:00.003+01:00

My friend Fabien Lavocat from Microsoft France recently sent me an email regarding a Visual Studio 2008 survey they have just started.
The survey is of course available in English.
If you have a couple of minutes to kill, please answer the survey as this will help Microsoft improve their future editions of Visual Studio!

Click here to access the survey

Thanks for your time!

Dotnet-France

2008-12-09T14:30:00.002+01:00

For french developers willing to obtain Microsoft Developer certifications, a colleague of mine (Julien Dollon) has just published a very useful website that will help you prepare for the certification exams:

Dotnet-france.com

I wish Julien good luck with his website and trust it will be a great help a lot of developers.

[MySQL] Using SQL_CALC_FOUND_ROWS and FOUND_ROWS with MySQL

2008-11-07T13:11:00.003+01:00

Hello,
for this second blog article I've decided to explain this neat little feature of MySQL: SQL_CALC_FOUND_ROWS and FOUND_ROWS().
This article is about MySQL only, it is likely that these keywords/functions exist in other SQL-based languages but I've only ever used them with MySQL.

What's the point, you ask?
When working with (my)SQL databases, you often find yourself using the LIMIT keyword, in order for example to limit the results of a large search. Let's study a particular case.
- You're working on the "employees" table, which contains, say 1000 data rows.
- You wish to display the list of employees in a particular department, 10 by 10.
- You wish to display the amount of employees in this department.

What would you normally do?
Here's a rough example:
$result = mysql_query("SELECT * FROM employees WHERE department='sales' LIMIT 0,10 ");
$total = mysql_num_rows($result);
-> This is incorrect because $total will never exceed 10 due to the LIMIT clause.

You could also do this:
$result = mysql_query("SELECT COUNT(*) cnt FROM employees WHERE department='sales'");
$count = mysql_result($result, 0, "cnt");
$result = mysql_query("SELECT * FROM employees WHERE department='sales' LIMIT 0,10 ");
-> But this means you have to process two SQL queries.

So how do you fix this?
Using the SQL_CALC_FOUND_ROWS keyword in your SQL query will allow you to fetch the total amount of results without being limited by the LIMIT clause.
Here's how you use it:

$data_result = mysql_query("SELECT SQL_CALC_FOUND_ROWS * FROM employees WHERE department='sales' LIMIT 0,10");
$count_result = mysql_query("SELECT FOUND_ROWS() cnt");
$employees_count = mysql_result($count_result, 0, "cnt");
echo "$employees_count employees found in the sales dept. Showing first 10: ";
...

As you can see, using the SQL_CALC_FOUND_ROWS keyword allows you to save the amount of results before applying the LIMIT clause; and this amount can be retrieved with a simple "SELECT FOUND_ROWS()" after the query is executed.

I've found this to be pretty useful when designing search pages, directories, and so on...
If you want more information about the FOUND_ROWS() function, and other useful functions aswell, check this page on the official MySQL documentation.

Okay, that's all folks!

WPF: drawing a video using MediaPlayer, VideoDrawing and DrawingBrush

2008-10-19T18:38:00.004+02:00

I've decided to start this blog for a simple reason: I was recently doing some research on various subjects, and was unable to find any clear documentation. And here I was thinking the World Wide Web had all the answers. Perhaps my contributions will help some of my fellow developers!

This first article is about how to draw a video using the MediaPlayer class in WPF (C# only - my apologies to VB.NET users: I'm allergic to VB). I was in the middle of preparing a training session at Avanquest Software when I read something in the WPF MOC: using the MediaElement control isn't the only way you can draw a video on a surface. Doh! I investigated a little further on the subject but couldn't find anything on the subject.

What does this do?
This article will explain how to use the MediaPlayer class to play a video on a surface. To demonstrate the wide range of possibilities offered by this mechanism, I will explain how to draw the same video clip on 9 different controls simultaneously.

MediaPlayer
First, you need to instantiate the MediaPlayer class. This class lets you manage the playback of a media file by exposing methods such as Play, Pause, Stop, etc. The constructor doesn't accept any parameter. Then use the Open() method to specify the media you wish to play. In our example we'll be playing a video.

MediaPlayer mp = new MediaPlayer();
mp.Open(new Uri("c:/video.avi"));

VideoDrawing
Then initialize a new instance of the VideoDrawing class. This class is used to do the actual drawing on a surface. Instantiate a new object and affect its "Player" member to the MediaPlayer instance you've declared above. Finally, affect the "Rect" member to specifiy the zone where the video will be drawn. Since we're going to be using the video source as a background for our controls, the video will be stretched to fit the control space so the value you specify here don't matter -- just make sure to specify a value greater than 0 as Width and Height.

VideoDrawing vd = new VideoDrawing();
vd.Player = mp;
vd.Rect = new Rect(0, 0, 100, 100);

DrawingBrush
At this point, nothing is drawn on the window. You need to instantiate a DrawingBrush based on your VideoDrawing instance:

DrawingBrush db = new DrawingBrush(vd);

Your "db" variable is now an instance of DrawingBrush, meaning you can affect any property that is of type Brush, such as Control.Background, etc. In my example, I've got a grid with 9 cells, each of these cells contains a button.

foreach (Button b in videoGrid.Children)
b.Background = db;

Once this is done, all you have to do is to call the Play() method of the MediaPlayer instance.

mp.Play();

I've provided a quick demo source code with this article. It plays the file located at c:\video.avi so make sure to either have a file with the same name or, of course, change the location of the file you wish to play.

Download demo project