2018-09-29

How to fix: Nginx proxy or fastcgi cache always MISS or EXPIRED

I've recently had an issue with my Nginx setup: it wasn't caching or serving cached pages properly. Every time I would visit, I'd get "MISS" or "EXPIRED".


What I found online is that you have to instruct Nginx to ignore browser request headers relative to caching, therefore allowing Nginx to determine whether it should serve cache, all by itself.

This is done by inserting the following directive:

# ignore headers to let Nginx control its own cache
fastcgi_ignore_headers Cache-Control Expires;

However, I already had this in my configuration and it wasn't helping.

After investigating the issue, I determined that my PHP scripts were updating cookies at every request. Nginx was forced to execute the PHP scripts via FastCGI (ie. not serving them from cache) because the cookie data had changed. When PHP sets a cookie, Nginx will not hit the cache, which is normal behavior. I wouldn't recommend changing this behavior. What I did is fix my PHP scripts so that they wouldn't update cookies at every request. Once I did that, I started hitting the cache again!


One last thing: if you turn off fastcgi or proxy buffering, caching may not work properly, so make sure buffering is on (it should be on by default).

2018-05-03

Clickatell vs MessageBird vs Nexmo vs Plivo vs Tropo vs Twilio: outbound SMS pricing comparison

DISCLAIMER: I am not affiliated in any way to any of those companies. I'm an independent software developer. This article is based on pricing data collected on May 2018.

As I was looking for a provider to send SMS internationally from a web application (one-way), I was recommended a variety of services by friends and colleagues. My main concern was the pricing: which of them would offer me the cheapest rates in the most countries? Obviously the answer isn't that simple, so I decided to collect pricing information from major providers to compare. I tried including SMSGlobal in this comparison but they wouldn't provide me with a full list so I had to exclude them.

Methodology

There are three important pieces of information that you absolutely need to grasp before you continue reading this article:

• The prices I have collected do not include bulk discounts. Most, if not all providers offer such discounts if you talk to their sales department. The rates I have retained in my comparison are standard rates and have been collected off the provider's sites. In half the cases I had to obtain the list by talking to the sales department directly - I asked for standard rates for low volumes.

• Some providers offer a flat rate per country, while others offer different rates based on the destination carrier. In order to establish a comparison that would be as fair as possible, for those operators that have different rates per carrier I compared prices based on AVG/MIN/MAX: AVG being the average rate for all carriers combined, MIN being the rate of the cheapest carrier, and MAX being the rate of the most expensive carrier.

•  This article compares providers solely based on pricing. But keep in mind that the pricing isn't the only thing that matters to select a provider: delivery rates, reliability, customer support, API, documentation, etc. can be equally important.

Contenders

The contenders are: (listed in alphabetical order)

1) Clickatell: based in South Africa. Flat rates. Pricing sheet obtained from sales department on May 2nd, 2018. Prices were given in USD.

2) Messagebird: based in the Netherlands (Europe). Flat rates. Pricing sheet obtained from sales department on April 30th, 2018. Prices were given in USD.

3) Nexmo: originally based in France (Europe) but now owned by Vonage (USA). Flat rates. Pricing sheet obtained from sales department on May 1st, 2018. Prices were given in EUR, converted to USD based on EUR/USD exchange rate from XE.com on May 2nd, 2018.

4) Plivo: based in the USA. Carrier-dependent rates. Pricing sheet obtained from website on May 2nd, 2018. Prices given in USD.

5) Tropo: owned by Cisco, based in the USA. Carrier-dependent rates. Pricing sheet obtained from website on May 2nd, 2018. Prices given in USD.

6) Twilio: based in the USA. Carrier-dependent rates. Pricing sheet obtained from website on May 2nd, 2018. Prices given in USD.

I collected all data from these 6 providers and combined it into a single SQL table. I spent a good amount of time fixing the name of countries to get a coherent list - my apologies if any mistakes were made as I'm not an expert in geopolitics. I also deleted a few rare countries that were covered by only 1 provider. Then I built an Excel file and determined the provider that offers the cheapest rates for each country.

Results

The final Excel table looks like this (download link can be found at the bottom of the article):


The providers that offer the best rates are... *drum rolls*:


1. Messagebird (flat rates)
They offer the cheapest rates in:
• 134 countries if we look at flat or average carrier rates of competitors
• 122 countries if we look at flat or minimum carrier rates of competitors
• 137 countries if we look at flat or maximum carrier rates of competitors

2. Tropo (carrier-dependent rates)
They offer the cheapest rates in:
• 50 countries if we look at flat or average carrier rates of competitors
• 44 countries if we look at flat or minimum carrier rates of competitors
• 50 countries if we look at flat or maximum carrier rates of competitors

3. Clickatell (flat rates)
They offer the cheapest rates in:
• 23 countries if we look at flat or average carrier rates of competitors
• 20 countries if we look at flat or minimum carrier rates of competitors
• 23 countries if we look at flat or maximum carrier rates of competitors

4. Plivo (carrier-dependent rates)
They offer the cheapest rates in:
• 19 countries if we look at flat or average carrier rates of competitors
• 37 countries if we look at flat or minimum carrier rates of competitors
• 16 countries if we look at flat or maximum carrier rates of competitors

5. Twilio (carrier-dependent rates)
They offer the cheapest rates in:
• 2 countries if we look at flat or average carrier rates of competitors
• 8 countries if we look at flat or minimum carrier rates of competitors
• 2 countries if we look at flat or maximum carrier rates of competitors

6. Nexmo (flat rates)
They offer the cheapest rates in:
• 1 country if we look at flat or average carrier rates of competitors
• 1 country if we look at flat or minimum carrier rates of competitors
• 1 country if we look at flat or maximum carrier rates of competitors

Downloads

The Excel file with the full list of countries is available for download here

You can also download the source code for my script, the file contains:
• The original CSV/PDF/Excel files from providers with the rates
• Database dump with the SQL table containing properly imported and reformatted data
• The source code of my script. Bear in mind this script was written over 2 hours, I am publishing it for reasons of transparency, so please no judging of the dirty code (PHP 5.6).

If you notice any inconsistency in my data, script, or methodology please comment and I will do my best to make the necessary fixes.

2018-02-21

Nginx HTTP Server - 4th edition now available

Thanks to the amazing work of Martin Fjordvald, the 4th edition of Nginx HTTP Server (which was, at the time of the 1st edition, the first-ever book about Nginx) is as of today available for purchase in libraries and online. Congratulations Martin!


This book is a detailed guide to setting up Nginx in ways that correspond to actual production situations: as a standalone server, as a reverse proxy, interacting with applications via FastCGI, and more. In addition, this complete direct reference will be indispensable at all stages of the configuration and maintenance processes. This book mainly targets the most recent version of Nginx (1.13.2) and focuses on all the new additions and improvements, such as support for HTTP/2, improved dynamic modules, security enhancements, and support for multiple SSL certificates. 

This book is the perfect companion for both Nginx beginners and experienced administrators. For beginners, it will take you through the complete process of setting up this lightweight HTTP server on your system and configuring its various modules so that it does exactly what you need quickly and securely. For more experienced administrators, this book provides different approaches that can help you make the most of your current infrastructure.

2018-02-14

How to fix: Outlook on iOS/iPhone: "No Internet Connection"

Microsoft Outlook is probably one of the best (if not the best) email client for iOS to date. Sadly, you may run into a random annoying problem which basically renders the application useless: it displays a message saying "No internet connection", at which point you can't receive or send emails anymore. The number of people having this issue is incredibly high, as I discovered when it happened to me yesterday:


After reading all of these posts and many more, there did not appear to be a particular solution for this. I had already checked the following:
  • my Internet was of course working properly, on both WiFi and 4G
  • I made sure to allow data/internet usage for the Outlook app
  • I made sure my email accounts were working properly (they work fine on the Outlook desktop app & web)
  • I tried enabling or disabling push notifications but that did not help
  • I tried to use the 'reset account' functionality in the Outlook app settings for each of the accounts
  • I made sure I had enough storage space left on device, which I did, over 10 GB left. And the Outlook app itself was only using about 200 MB.

None of the above proved useful.

The only solution?

Eventually, here is what I did to get the app to work again:

  1. Removed all my accounts one by one from the Outlook app
  2. Went to the iOS settings, "Storage & icloud" settings, manage storage, selected the 'Outlook' app and tapped 'Delete' in order to delete the application from my iPhone
  3. I re-downloaded the application from the App Store
  4. I set up my accounts into the app again

Thankfully, the 'no internet connection' message hasn't shown up again until now. It's really surprising that considering the amount of people having this issue (not even recently) Microsoft have yet to address the issue properly. If someone is reading this, my iPhone is running the latest version of iOS 10, and I always keep my apps up to date so I had the latest version of Outlook for iOS installed at the time of writing this article.

2017-01-04

VPNAutoConnect 1.0 for Windows: automatically connect and reconnect to a VPN

I am releasing a handy little tool I have been working on called VPNAutoConnect. It does a couple of simple things Windows won't let you do (at least not easily):
- automatically reconnects to a VPN when the connection is lost
- connects to a VPN on computer start-up
- logs VPN connection events for debugging


I designed this tool because I have been using a VPN for a while but I randomly get disconnected. I wish Windows offered an option to automatically reconnect but strangely there isn't an easy, straightforward way to do that (as of today). With this tool, not only my VPN connection is started when I boot my computer, but it always reconnects automatically after I get those random disconnects.

This is a free open source program designed with Visual Studio Community 2015. It requires the .NET framework 4.5.2, and it makes uses of the DotRas library.
- Download program from here: vpnautoconnect-1.0.zip (SHA1)
- Download sources from here: vpnautoconnect-1.0.src.zip (SHA1)
NB: use at your own risks. Tested under Windows 8.1 only.

There are 3 command-line arguments that control the behavior of the program, allowing you to achieve automatic VPN connections on computer startup for example.
--quiet: enables the 'do not display ballon tips' option
--timer=N: sets the timer to N seconds
--connection=XXX: automatically connect/reconnects to XXX when the application starts. If your connection has spaces in its name, try using quotes (ie. --connection="My VPN")

2016-12-18

Simple VPN autoconnect for Windows 8 and 10

Since Windows 7, for some reason VPN support in new Windows versions has annoyingly decreased in quality. You can't create shortcuts to VPNs on the desktop or in the start menu anymore; you can't set up a VPN connection to automatically connect on start up; and with Windows 10 there is even a bug (at the time of writing this article) that makes you click on another connection item in the list before the Connect button appears.

The simplest way to achieve automatic VPN connection on computer start-up is to create a batch/cmd file with the following command:
rasdial "VPN Connection" username password

Where:
- rasdial is the name of the command-line utility that will perform the connection
- replace "VPN Connection" by the actual name of the VPN connection
- replace username by your actual VPN user name
- replace password by your actual VPN password

If you want the VPN connection to be started when you log on, simply place this batch script into your startup folder: press Win+R to run 'shell:startup', this will open a folder in which you can place programs or shortcuts to be ran on startup.

UPDATE: or use this tool - VPN AutoConnect

2016-09-06

Solution for "DOMException: Failed to load because no supported source was found" error in Chrome with HTML5 media API

After updating Chrome to release 53 this week, my web application was no longer able to use the media API to capture video with a webcam. This is the third time that I am forced to update my code due to changes in Chrome... this is getting a little annoying!

The error message I was getting in my javascript console is:
DOMException: Failed to load because no supported source was found

I'll keep it short. The problem was the way I set the source of my media element.
I was setting the source like this:

navigator.getUserMedia(videoObj, function(stream) {
   CaptureImageStream = stream;
   CaptureImageVideo.src = stream;
   CaptureImageVideo.play();
}, errBack);

It seems like this is no longer supported. I changed it to this and it now works again:

navigator.getUserMedia(videoObj, function(stream) {
   CaptureImageStream = stream;
   CaptureImageVideo.src = window.URL.createObjectURL(stream);
   CaptureImageVideo.play();
}, errBack); 

I hope this helps!
Clement

2016-07-28

Nginx: set up a LetsEncrypt SSL certificate with auto-renewal in 3 easy steps

Unless you have been living under a rock for the past year, you should know by now that you can get SSL certificates free of charge from LetsEncrypt, without registration, and with automatic renewal! This is one of the best thing that's happened to web admins and the web in general in the recent years. The certificates are authentic and work great in all browsers (you get the little green lock icon like everywhere else).

Let's get straight to the point. The three steps are summarized here:
1) Download LetsEncrypt (the application) for your Linux server
2) Run the application to generate a certificate for your domain and set up the monthly auto-renew cron job
3) Add the certificate to your Nginx configuration.

Step 1: download LetsEncrypt

Install git if you haven't done so yet:
# apt-get install git

Use git to get the application and store it somewhere (ie: /root/temp)
# git clone https://github.com/letsencrypt/letsencrypt /root/temp/letsencrypt

Step 2: generate your certificate

The first time you run the command below, you will be asked to provide an e-mail address to be associated to the domain or subdomain, in case you should ever need to recover the key or something. The next time you run the same command (to renew the certificate) it won't be asked.

So run the following command to generate the certificate:
# /root/temp/letsencrypt/letsencrypt-auto certonly -a webroot --agree-tos --renew-by-default --webroot-path=XXX -d YYY
Where: XXX is the full path to your website's root folder. For example /home/www/website.com
And YYY is the domain name or subdomain name (ie: website.com, or something.website.com)

At the time of running the command, your domain must be available to visitors already, because LetsEncrypt's servers will make a verification to ascertain that you are actually the owner of the domain. It will place a hidden /.well-known/ folder with some files in it at the root of your web directory (specified above with --webroot-path).

Certificates generated by LetsEncrypt are valid 3 months at the moment. Also, wildcards are not supported; you can get certificates for domain names and subdomains one by one. Since we don't want to manually renew the certificate every month for every domain and subdomain, we can set up a simple cron job to be ran monthly (it should be run every 3 months ideally but LetsEncrypt are talking about reducing the validity of their certificates so I'd rather not be caught off guard).

Open /etc/cron.monthly and create a new file, make sure to chmod +x to give executable permissions. The first line of the file should be:
#!/bin/sh
And then put the exact same command you ran before to generate the certificate. You can also add  >>/root/temp/certificate-update.log at the end of the command if you want to keep a log of the updates (although I imagine LetsEncrypt generates its own log somewhere too).

Step 3: configuring Nginx

After running the command that generates the certificates, you should have several files in /etc/letsencrypt/live/website.com/ (replace website.com by your own domain). We are going to need just two of them for Nginx: fullchain.pem and privkey.pem.

The beginning of your server block should look like this:
server {
server_name www.website.com website.com;
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/website.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/website.com/privkey.pem;
ssl_session_cache shared:SSL:20m;
[...]

Don't forget that last line, otherwise Safari and iOS devices won't be able to visit your website (see my recent blog post about it). Save configuration, reload, visit your site and voila, you're done. To make sure the automatic renewal script works, try running it now with the new configuration.

2016-07-27

Nginx and LetsEncrypt SSL certificate problem with iOS and Safari (fixed)

I have recently started using LetsEncrypt as my main SSL certificate supplier, it's amazing! With the auto-renew cron task, I have literally 0 work to do to keep certificates up to date, and of course, it's free.

I recently noticed an issue though: when I visit my websites over HTTPS on my iPhone (and reportedly the problem exists with Safari on Mac OS X as well), the sites simply will not load. This is the error I get:


The error reads: "Safari cannot open the page because the network connection was reset. The server may be busy". After trying to sort out the problem for hours and Googling up the error everywhere, I finally stumbled upon this forum thread. Well thank you Mr. Duckson because that really did fix the problem!

The solution: in your server {  ... } block, insert the ssl_session_cache directive with whatever value you deem fit. Example from Nginx documentation: ssl_session_cache shared:SSL:10m;

Save your configuration, reload Nginx, try it again - voila, it's fixed. Kudos to koren on IRC for finding the solution, you have amazing Google skills, my friend. If anyone has an explanation as to why this directive is required for Safari/iOS browsers specifically with LetsEncrypt certificate, I'm all ears!

2016-03-08

HTML5 video capture from browser: error with "stop() not a function"

A quick blog post about recent changes in the MediaStream API support in modern browsers. I had previously developed a function allowing users to capture images from their webcam, based on tutorials found on the web.

Here was my code to initialize the capture canvas:

function CaptureImageBeginCapture() {
   CaptureImageCanvas = $("#ImageCaptureCanvas")[0];
   var videoObj = { "video": true },
      errBack = function(error) {
         console.log("Video capture error: ", error.code);
      };

   CaptureImageVideo = $("#ImageCapturePreviewFrame")[0];
   CaptureImageContext = CaptureImageCanvas.getContext("2d");

   // Put video listeners into place   if(navigator.getUserMedia) { // Standard      navigator.getUserMedia(videoObj, function(stream) {
         CaptureImageStream = stream;
         CaptureImageVideo.src = stream;
         CaptureImageVideo.play();
      }, errBack);
   } else if(navigator.webkitGetUserMedia) { // WebKit-prefixed      navigator.webkitGetUserMedia(videoObj, function(stream){
         CaptureImageStream = stream;
         CaptureImageVideo.src = window.webkitURL.createObjectURL(stream);
         CaptureImageVideo.play();
      }, errBack);
   }
   else if(navigator.mozGetUserMedia) { // Firefox-prefixed      navigator.mozGetUserMedia(videoObj, function(stream){
         CaptureImageStream = stream;
         CaptureImageVideo.src = window.URL.createObjectURL(stream);
         CaptureImageVideo.play();
      }, errBack);
   }
}
And this code was used to react to the pressing of a "Capture" button which froze the image and then saved it.
// Snapping up the picture$("#ImageCaptureSnap").click(function() {
   CaptureImageContext.drawImage(CaptureImageVideo, 0, 0, 640, 480, -95,0,448,330);
   CaptureImageStream.stop();
   $("#ImageCaptureStep1").hide();
   $("#ImageCaptureStep2").show();
});
Unfortunately, the MediaStream.stop() function recently became depreciated. After a quick search, I found only one article explaining what I need to do to fix it. The fix is to get a "track" instance from your MediaStream and stop the track instead of stopping the MediaStream itself.
// Snapping up the picture$("#ImageCaptureSnap").click(function() {
   CaptureImageContext.drawImage(CaptureImageVideo, 0, 0, 640, 480, -95,0,448,330);
   try { CaptureImageStream.stop(); } catch(e) { }
   try {
       var track = CaptureImageStream.getTracks()[0];
       track.stop();
} catch(e) { }
$("#ImageCaptureStep1").hide(); $("#ImageCaptureStep2").show(); });
Hopefully it won't change further, but as we know HTML5 APIs are evolving all the time.

Search This Blog