2012-11-28

LimeSurvey Nginx rewrite rules

I just downloaded LimeSurvey, an amazing free open-source project for creating your own online surveys, and was about to install it on my own Nginx-powered server. To my surprise, there are no available Nginx rewrite rules for it on the official project website, and I didn't even find any blog post detailing such rules. So here I go again, filling the voids.

This post details:
- the Nginx rewrite rules translated directly from the provided .htaccess file, the small server configuration file containing the rewrite rules for Apache.
- what to do after installing LimeSurvey

Nginx rewrite rules

The original rewrite rules for LimeSurvey are contained in this simple .htaccess file:

<IfModule mod_rewrite.c>  
RewriteEngine on
    # if a directory or a file exists, use it directly
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    # otherwise forward it to index.php
    RewriteRule . index.php
</IfModule>
# General setting to properly handle LimeSurvey paths
# AcceptPathInfo on

Before you copy the Nginx-compatible rules below, you should ideally try to understand what the above does. The first two lines (IfModule... RewriteEngine on) enable the rewrite rules only if the rewrite module is enabled. Normally it should be the case by default on Nginx so you don't need to worry about that.

The following lines (RewriteCond...) specify conditions for the URL to be rewritten. The first line (!-f) ensures that the requested URI doesn't really exist. For example, if the visitor requests /document.txt and that file actually exists on your server, then the URI doesn't need to be rewritten at all; the file can be served as is. The second line (!-d) does the same for folders; if the requested URI corresponds to an existing folder, it will serve it normally, by redirecting the user to its index page, or displaying an automatic index or something. Eventually if both RewriteConditions are met, then Apache will rewrite the URI by internally redirecting the request to index.php (and automatically appending the arguments to the URL).

We could write a similar script for Nginx with if's and rewrite instructions, but the best practice is to use the try_files directive:

location / {
try_files $uri $uri/ /index.php?q=$uri&$args;
}

What this does is:
1) Nginx attempts to serve $uri (the requested URI)
2) If the above doesn't exist, it attempts to load $uri/ (note the slash for a potential folder)
3) If the above doesn't exist, it loads /index.php?q=$uri&$args (the index.php page, transferring the request arguments too)

An average working (tested) server block for serving LimeSurvey would be:

server {
listen 80;
server_name mywebsite.com;
root /var/www/mywebsitefiles;
index index.php;
location / {
try_files $uri $uri/ /index.php?q=$uri&$args;
}

location ~* \.php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param PATH_INFO $fastcgi_script_name;
include fastcgi.conf;
}
}
Make sure to replace my example values by your own; if you are unsure about the meaning of the directives you should consult the Nginx wiki. Save your configuration file and reload Nginx to apply the changes.

Note: if you allow file uploads from users, you should better secure the "location ~* \.php$" line by specifying from which folders PHP files may be executed. Otherwise your server might be at risk depending on the files you allow your users to upload (ie. if you allow them to upload PHP files).

After installing LimeSurvey

Whether you install LimeSurvey after or before applying the Nginx rewrite rules doesn't matter: the LimeSurvey install script will detect that your server isn't running Apache's mod_rewrite anyway, so it will automatically switch off the "fancy URLs" functionality.

In order to enable it, open /application/config/config.php and change the following values:
'urlFormat' => 'get', 
change this line to:
'urlFormat' => 'path',
And then:
'showScriptName' => true,
to the following line:
'showScriptName' => false,

After doing so, try visiting yourwebsite.com/admin/ and you should be taken to Limesurvey's administration panel. If your browser isn't loading the page properly you might have to clear your browser cache.

I have tested this myself and haven't run into issues so far.

2012-11-02

Shanghai Air Quality Index Widget (from US Embassy data)

I live in Shanghai and air pollution is a major concern, especially in institutions such as schools or hospitals. It is important to give proper directives to children and elder: when the air pollution reaches a certain threshold, they should stay inside as much as possible and avoid any form of physical exercise outside.

There are mainly two indices for air quality in Shanghai: the Chinese government's data and the official website of the US Consulate in Shanghai. The latter is based on an RSS feed updated every hour.

In order to facilitate the display and comprehension of the data from that RSS feed, I have designed a widget (running under Windows XP/Vista/7/8) that is meant to display Shanghai's air quality index in real time - again, based on official data from the US Embassy. The widget was designed to be ran on display monitors that stay on 24/7, so it displays at the topmost - it always shows on top regardless of other windows. The data is automatically refreshed every hour with a timer. If you start the widget at 8:05am, it will refresh the data at 8:05am, 9:05am, 10:05am, and so on.

If you are interested in the widget, you can download it from this link. Warning: this application requires the .NET Framework 4.5 to be installed on your machine. It won't run without it!

You may also download the source code here (developped with Visual Studio Express 2012 for Windows Desktop).

2012-09-26

JWPlayer: remembering position in video, to resume playback later

As part of my recent developments on my upload site Filetrip.net, which comes with interesting video playback capabilities, I've added the possibility for visitors who watch a video once and interrupt playback to return to the video later on and resume where they left off.

In short: you are watching a video, you suddenly or accidentally close the tab or visit another site. When you go back to the video page and start playing again, the video will resume where you left off.

I was wrong to assume that this was an upcoming feature for JwPlayer, seeing as one of the authors stated that they "don't support this explicitely". So I designed it myself with minimal coding. Note that this has been tested on JWPlayer 5 and above, but I cannot guarantee that it will always work in the future.

Here is how I am going to detail this tweak:
1. Introduction and requirements
2. Remembering the position
3. Setting the starting offset
4. Cookie data limitations

Introduction and requirements

Let me explain how I got this to work: the solution is really very simple.
First, I set a javascript timer that makes an API call to the player asking for the current position in the video. Then, I save that position in a cookie. Finally, when the visitor reloads the page, the backend (example: a PHP page) tells the player the starting offset. Note that this third step could be done in Javascript, client side, but it's really up to you.

If you have read and understood the above correctly, the requirements are:
- cookies, being able to store them somehow. I will provide a javascript function for setting cookies.
- JWPlayer 5 or above, with its javascript API

Remembering the position

Somewhere in your page, insert that bit of code:

<script type="text/javascript">
function setCookie(c_name,value,expiredays) {
var exdate=new Date();
exdate.setDate(exdate.getDate()+expiredays);
document.cookie=c_name+ "=" +escape(value)+ ((expiredays==null) ? "" : ";expires="+exdate.toUTCString());
}
</script>

This is a function that will allow you to store some data on the visitor's computer. We will be using it to remember the position in videos.

Now, below that function (before the </script> closing tag), insert the following lines of code:
jwplayer().onPlay(function() { setTimeout("rememberPosition()", 5000); });
jwplayer().onComplete(function() { setCookie("mv_{$pid}", 0,-1); });
function rememberPosition() {
  if (jwplayer().getState() == "IDLE") {
    setCookie("mv_{$pid}", 0,-1);
  } else {
    setCookie("mv_{$pid}", Math.round(jwplayer().getPosition()),7);
    setTimeout("rememberPosition()", 5000); 
  }
}

  • The first line tells the player to call the "rememberPosition" function five seconds after the playback starts.
  • The second line tells the player to remove the cookie when the video playback is complete.
  • The "rememberPosition" function saves the position in the cookie for 7 days, and calls itself again in five seconds. If the user stops the video somehow or the playback finishes without triggering the onComplete event, the cookie is removed all the same.
  • Note: regarding the jwplayer().getState() == "IDLE" part, the IDLE state means "video stopped" or "playback complete". But it is different from the PAUSED state: in fact it is important to remember the position when the visitor paused the video. However once the visitor finished watching the video, we can delete the cookie permanently.
Now, if you've noticed, the name of the cookie value is mv_{$id}. The {$id} part means you should insert the ID of your video, or something that uniquely identifies your video, so that when your visitor plays another video the playback doesn't resume at the offset saved from another completely different video. You can also change the frequency (here, 5000 miliseconds) and the number of days the cookie should exist, 7 days in my example.

Setting the starting offset

Now, we have to tell JWPlayer to start the playback where the visitor left off - that is, in the value stored in the visitor's cookie. I realize that this could easily be done in Javascript with a getCookie function (you can find that on the web) and call that when the page loads or something. For some reason it was easier for me to do it with PHP.

From within the player page, find the javascript declaration of your JWPlayer instance. Here is mine:
  jwplayer('container').setup({
    'flashplayer': '/js/player59.swf',
    'id': 'playerID',
    'width': '100%',
    'height': '100%',
    'file': '<?= $path ?>',
    'image': '<?= $path ?>.jpg',
    'controlbar': 'over',
    'start': '<?= intval($_COOKIE['mv_'.$id]) ?>',
    'skin': '/js/beelden.zip',
    'backcolor': '#000000',
    'screencolor': '#000000',
    'provider': 'http'
  });

Note the <?= ?> PHP tags to insert the file path, and more importantly the 'start' parameter, which will indicate the starting offset for the video. 

Cookie data limitations

If your visitors are going to be watching a lot of videos, you might want to avoid using cookies for remembering positions. Remember:
1) cookie data is sent in every request to your HTTP server, even images, CSS, javascript and all. So it does take a bit of bandwidth.
2) there is usually a server-defined limit on the size of cookies. Things get big... You might want to keep the expiration date for your cookies as low as possible so that it doesn't accumulate over time.
3) if the visitor's cookie gets too big, you might get one of those Error 400 - Bad Request on your web server.

I hope you enjoyed my little tweak, feel free to post your own contribution in the comments!

2012-06-21

AVCLevels PHP Library: read and set profile level from H264 AVC-encoded video (MP4, MKV, ...)

I was stunned this morning to find out about AVC levels (H264/MPEG-4 AVC). Basically, when you encode a video using the widespread H264 codec for the video stream, the encoder sets a flag called the "level". Wikipedia says that the level is a specified set of constraints indicating a degree of required decoder performance for a profile. The problem I have with this is: some software and equipment restrict their own capabilities based on that level! They check the level, and if the value doesn't please them, they refuse to even attempt to play the video.

What is this all about?

See, I had two MP4 videos. One would play fine on my iPhone, the other refused to play (iTunes refused to copy it into my library, claiming that my iPhone did not support the video format). I inspected both files with MediaInfo, they were both encoded properly with AVC (H264) and AAC audio, similar bitrates, same resolution. After hours of testing and research I stumbled upon this blog post. Turns out that the only thing preventing my iPhone to play the second video was the "level" flag that was set to 51. The first file's flag was 31. Further researches indicated that my iPhone 4S supports H.264 video up to 720p, 30 frames per second, Main Profile level 3.1 with AAC-LC audio up to 160 Kbps per channel, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats.


So I manually edited my video using some hex editor, set the "level" byte from 51 to 31, and voila: the video played fine on my iPhone. Wow. 


A good solution for Windows

I discovered a nify little Windows application that allows me to set the level: H264 Level Editor. Easy to use, the program does the job, and more. Make sure to back up your files before setting the level.

Problem is, I have hundreds of videos that I need to edit out. And since the videos are on my web servers, I need a solution that runs under Linux. I didn't find anything like that. I expected MP4Box to be able to do so, or ffmpeg, but since this is a downright "hack" (a workaround) these noble apps won't help - at they will only re-encode the videos which takes hours. Come on! All I need to do is change a single byte...

My AVCLevels PHP library

Since I didn't find any program that would do it, I programmed a simple PHP library that will do the job. Again, all it does is detect the flag from your video file, and read or set the "level" byte.

Download the AVCLevels library from Filetrip (Free. This is the source code, feel free to use it any way you want). Use it at your own risks! This isn't guaranteed to work on all videos, though I have tested it with over 100 videos and it always worked fine. Maybe I was lucky, anyhow DO back up your files before using this.


Using it couldn't be simpler. There are no particular requirements, just PHP 5 and higher.

// Include library
require_once("AVCLevels.php");
 
// Set file path
$file = "./test.mp4";
 
// Retrieve & display level
$level = AVCL_RetrieveLevel($file);
echo "Level: ".$level;
 
// set level to 31 (AVC Profile Level 3.1)
AVCL_SetLevel($file, 31);

That's all there is to it really. If you want an iOS compatible video, set the level to 31, since that's what it takes. Anything higher won't work. Lower profile levels should work, but 31 seems to do the trick. I don't know what level is recommended for the PS3 but I know Sony's been using the same practices (31 should work for the PS3 I suppose).

2012-05-29

Top 3 fixes for: Internet Explorer won't start, IE7, IE8, IE9 opens then closes immediately

I have been having this issue for years and I found the final solution just yesterday! So I thought I would post it here and hopefully help out thousands of other people like me.

Before I explain the solution let me describe the problem in detail. For some unknown reason, I have always had this problem on my main computer with Internet Explorer 9: whenever I want to run it, a blank window appears for 0.1 second then closes instantly, without any error message or anything. The solutions I describe are valid for Internet Explorer 7 (IE7), Internet Explorer 8 (IE8) and Internet Explorer 9 (IE9). I am running Windows 7 but the solutions also work for Windows Vista and Windows XP.

There are three solutions here, among which two are the official solutions proposed by Microsoft. Let me tell you right off - Microsoft's solutions had no effect, so you might want to skip to the last one immediately.

  1. Solution 1: deleting all IE settings so as to remove potentially blocking add-ons
  2. Solution 2: uninstalling and reinstalling IE completely
  3. Solution 3: running a script that cleans your registry and re-registers missing DLLs
Read on for a detailed description of each of these three solutions.

Solution 1 (by Microsoft)

The problem is perfectly described on a support page entitled "Internet Explorer opens, flashes, and then closes immediately when you start it". That's exactly what happens here, except their solutions had no effect for me. Maybe you'll be luckier than I was?

So the first solution is to reset Internet Explorer to its default settings. After all, the problem you are having might be caused by an add-on that causes IE to crash or something. In order to proceed, either click the above link to download a tool that will clean up your IE settinsg, OR run the tool from your machine (yeah, the tool should already be installed on your machine):
1) Open the Internet Options applet from the control panel
2) Go to the Advanced tab
3) Click the "Reset Settings" button
4) Make sure the "Delete personal settings" box is ticked and press OK.
5) Try running IE again and see if that fixed it.



Solution 2 (by Microsoft)

If that didn't help, you can try their second recommendation, which is basically to uninstall and reinstall Internet Explorer. The process is simple and there is actually no installer to run.
Uninstalling:
1) Open the Programs and Features applet from the control panel (Add/Remove programs in XP)
2) Click the "Turn Windows Features on or off" link on the left
3) Untick Internet Explorer in the list of programs that shows up
4) Restart your computer
Reinstalling: do the exact same, except you tick the box.
If you have an older version of Internet Explorer, I recommend downloading Internet Explorer 9 (offline installer, not one of those stupid web installers that always fail).



Solution 3 by Kai Schätzl - the ultimate fix!

If like me none of the above worked, you must try Kai Schätzl's delicious script. You have no idea how thankful I am, Mr. Kai. Basically, all you have to do is download one of the below scripts:
- If you are running Windows 32-bit, download this: Internet Explorer 7/8/9 repair script for Win32
- If you are running Internet Explorer 32-bit version on Windows 64-bit, download this: IE 7/8/9 repair script for IE 32bit on Win64
- If you are running Internet Explorer 64-bit on Windows 64-bit, download this: IE 7/8/9 repair script for IE on Win64

Once you have that downloaded, extract the ZIP archive, you will find a .cmd file (command-line batch file). It contains instructions that will repair your system and allow Internet Explorer to finally work again. DISREGARD the filename: it's called "IE8" but it works with IE7, IE8 and IE9. To run it, right-click the file and click "Run as administrator".



After it finishes running, start Internet Explorer and voila! You can now use the world's worst web browser again. Download Google Chrome now!

2012-01-23

Chinese OCR: translating scanned or photographied Chinese text to any language


Having lived in China for almost 3 years now I am able to recognize a good bunch of characters, I can type in Chinese on the computer too but writing is much easier than reading since it doesn't require you to actually memorize the characters, you just type in pinyin (phonetics). That is not enough to understand a full, complex text.
After months of research I've finally figured out how to recognize chinese characters automatically from a picture, in order to copy/paste the text into a translator such as Google translate or others. The solution was right under my eyes all this time: Microsoft Office 2007. I had no idea that Office 2007 came with such features. I've always known of expensive solutions such as Ominpage Pro, but I refused to resort to purchasing the app considering its price and how little I would need it.


OCR, which stands for Optical Character Recognition, is the principle of proceeding to the digital analysis of an image to extract the characters/text that it contains, in order to be able to manipulate the text on a computer.

The solution I describe is for PC/Windows users only. If you're interested on doing the same, real time and in a much simpler way directly from your iPhone, I recommend the excellent Pleco. I've tried out the demo version and am seriously considering purchasing the full version.

TLDR: there are basically four steps involved in the process:

  1. Take a photo with your digital camera, or scan your document with your scanner
  2. (Optional) Convert your photo to a TIF image
  3. Open the TIF image with Microsoft Office Document Imaging, run the OCR
  4. Export the text to Microsoft Word then translate it

This tutorial requires the following software to be installed on your computer:

  1. Microsoft Office 2007 (though this supposedly works with Microsoft Office 2003): installed with "Document Imaging" and "Picture Manager", both are components that you can select during the setup process. If you don't have those two installed on your computer, modify your Office setup to include them.
  2. Chinese language support for Microsoft Office, which isn't exactly something you come across easily. I have the chance to work in China and we have licenses for the Chinese version of Microsoft Office, so I've had no trouble. As an alternative you can get the Microsoft Office 2007 Multi-Language pack and install Chinese support as well as a bunch of other languages if you're interested.
Step 1: Taking a picture or scanning a document
I don't need to remind you how you take pictures with a digital camera. Nor how to save pictures from a website with your favorite web browser. If you are going to use a scanner though, and that is probably the solution that will get you the best results, you can probably skip the next step if your scanner supports saving as TIF/TIFF documents.

To illustrate this tutorial I've chosen to work with a photo taken with my iPhone 4S. It's a document I've taken from a random advertisement booklet found at a friend's place. I tried to get a clear shot of the text to make sure OCR works as accurately as possible.


Step 2: Converting to TIF/TIFF image
Unfortunately, and I must admit I find this quite odd myself, the tool we're going to use for performing the OCR does not support anything other than the TIF format. So if your picture was saved under any other format (JPG typically, like mine) you'll have to convert it. There are plenty of ways to do so.

Since you have Microsoft Office installed on your computer, you should have everything it takes. Right-click your JPG image and "Open with..." - "Microsoft Office Picture Manager". Go to "File" - "Export..." and select the TIF format.

Step 3: Performing the OCR
The actual OCR (Optical Character Recognition) is performed by Microsoft Office Document Imaging. Open the tool, which should be located in your Start Menu under Microsoft Office / Microsoft Office Tools / Microsoft Office Document Imaging.

Before performing the OCR you need to specify the document language. To do so, open the "Tools" menu, go to "Options" - "OCR" and select "Chinese" in the drop-down list. The next steps are simple...

Open the file... Click on the OCR button... Click on the "Send Text to Word" button... press OK and you're done!
The text should be more or less faithfully transcripted depending on the quality of the original picture. Now onto translating it to something actually legible to the average westerner :-)

Step 4: Translation to English or other languages
There are tons of translators out there but I'm going to stick to Microsoft Office since that is what we've been using from the start. Yes, you can translate Chinese directly from within Word 2007 if you follow the simple instructions described below:
  1. Select the text you want translated
  2. Right-click the selected text and in the menu, go to "Translate" - "Translate..." 
  3. Select the input and output languages and click the little green arrow
  4. You'll be taken to Microsoft's online translation service, which provides a surprisingly accurate translation of my original text.
Before:

After:
Note: the original document IS about shady management techniques. That's all I had.

Voila, you've successfully translated a document written in another language, based on a simple photo and Microsoft Office. Ah, isn't technology wonderful?

Search This Blog