One of the functions of this server is to run a few small mailing lists, some of the members of one of these lists receive their email via a corporate network which happens to silently drop any messages which happen to contain whatever the corporate network operators happen to define as profanity.

As the list infrequently happens to contain profanity, and it’s quite inconvenient to have messages dropped without notice, I figured it’d be fairly trivial to work something out, and saw this as an opportunity to teach me something about Exim.

I’ve used the Exim configuration to add a small script into the router pipeline, which can operate on messages, changing their content and headers. You could realistically use this script to do whatever you wanted, I based my configuration on the standard Exim configuration for Spamassassin, which is a fairly large example of magic email fiddling.

Read on for the full configuration and instructions.

Continue reading →

One of the aims of creating this server was to test ways of making Linux servers I manage more secure in general. Two-Factor authentication is a method often employed to harden-up and increase the security of logins, but before now I’ve assumed there’d be some monetary cost to this.

It turns out, you can use just a standard Debian package along with a free* smartphone app to setup two-factor authentication for SSH, with about 5 minutes actual work.

* – As Matthew notes in the comments, both client and server for Google Authenticator are Open Source and based on an Open Standard, and available from http://code.google.com/p/google-authenticator/ 

Method

So, this is rediculously easy. Also it won’t affect SSH Key logins (which are already two-factor if you have a passphrase on your private key), so your normal login processes shouldn’t be affected, you should only need to enter the login code when you’re logging in from somewhere you don’t have access to your SSH key.

1. Install the ‘libpam-google-authenticator’ package – this is in the standard Debian (squeeze) repositories: aptitude install libpam-google-authenticator
2. As the user you’re going to login as (in my case, ‘james’), run the command ‘google-authenticator’ – this will generate the necessary keys and provide a QR code for you to scan.
3. Install the Google Authenticator app, this is free and available for at least iOS and Android.
4. In the Authenticator app, add a new identity and choose the ‘scan a barcode’ option, then scan the QR code from the console.
5. Edit the configuration for SSH (/etc/ssh/sshd_config) to enable ChallengeResponseAuthentication, just fine the line this is (presumably) disabled on, and change ‘no’ to ‘yes’.
6. Reload/Restart SSHD
7. Edit /etc/pam.d/sshd and add the following line:
   auth required pam_google_authenticator.so
8. That’s it! Just try a login without your key, and you should be prompted first for your password, then for the code, just enter the numbers from the app!(If you want to test this, you can disable keys on a one-off basis as follows:
   ssh -o PubKeyAuthentication=no user@host.name

Official instructions for this can be found here! (don’t miss the ChallengeResponse bit though!)

The command-line client provided by the libpam-google-authenticator package also interestingly makes use of a command-line QR code viewer, which I’d never seen before. Check it out!

I decided that since I was using Nginx for Ruby apps, I might-as-well get rid of Lighttpd altogether (which was previously in-use for serving static files, HTTPS and PHP).

This wasn’t massively simple, I figured since HTTP was already behind a proxy (Varnish), HTTPS might as well be also for clean-ness’ sake, for this I chose Pound, a very simple HTTPS proxy. This just sits in-front of Nginx proxying SSL traffic. In this way, Nginx isn’t exposed in any way to the public.

Making Nginx work with PHP is slightly more effort than making Lighttpd do the same, whereas Lighttpd is capable of spawning it’s own FastCGI (php-cgi) processes, Nginx has none of the same functionality. Luckily, a fairly simple method is provided for doing this yourself.

Anyway, I present a diagram which documents the new software layout. In this diagram, black lines represent standard HTTP traffic, red is HTTPS, and blue shows traffic passed over an internal UNIX socket.

TLDR: The next few paragraphs can be mostly ignored by the general reader, feel free to skip to the solution below, if you simply have an SD card which you can read a bit of, but which dies before you’re able to access the full contents.

I’ve recently had the chance to learn about some fun tools for data recovery, in this instance through trying to recover some photos from a (presumably physically) damaged SD card for a relative over the holiday period.

My experience here is all based on this incident, related only to recovering photo data from a damaged SD card, although the tools and practices could possibly work for other forms of file recovery, on different kinds of media (notably spinning platter hard drives).

Additional Note: Since I made this work and posted about it on Twitter, I was pointed in the direction of this article, which covers the actual file recovery part, and covers a tool which I made use of in recovering the data from this SD card (photorec).

The Problem

The card I was handed had a problem which I’d not seen before. When presented to iPhoto in a card reader, the software would show that there were 309 photos and would attempt to begin an import, around half of the way through, OSX would throw up the “Removable media unplugged without being Safely Removed” dialogue, and the card would disappear from the mounted device list, and it’s device entry under /dev would disappear.

It seemed as if a small part of the physical SD card was damaged, which would cause the operating system to see the device as disconnected when that part of the card was read.

By iterating over the SD card byte-by-byte, I was able to discover that the faulty problem data appeared to occur at a consistent location on the card. By skipping to $errorByte+(some number of kilobytes) before starting the byte-by-byte read, I was able to continue getting valid data off the card (up until another small area of error bytes later on the card).

TLDR: read the disk byte-by-byte, found two small areas of the disk which would cause issues, was able to read the rest of the disk without a problem

This essentially left me with the knowledge that most of the data was probably intact, the errors appeared to be confined to two small (<1MB) areas of the physical SD card, around about the 53% and 85% marks. Unfortunately, this made recovering the data tricky. I wrote a bit more code to seek() past the bad bits and write the rest of the data to files, although this didn’t seem to provide me with anything I could restore from.

Solution: Ddrescue to the um… rescue!

Through some googling, I was found this page, which highlighted the *excellent* tool Ddrescue.

Unlike the standard GNU dd, which basically just copies “some” blocks from somewhere to somewhere else, ddrescue is designed to recover as much data as possible from a known-faulty device.

The way this is done is described here. In my case, ddrescue was able to scan over the card until it hit the bad section, then was able to continue past that section, even reading backwards from the end of the file until the end of the erroring sections to narrow down the bad areas as much as possible, all t

So then, although I’ve gone over the way that WordPress and it’s requisite caching plugins are installed, I’ve neglected to go over how the PHP code is actually interpreted and served to visitors.

Interpreting the PHP code is lighttpd, running the fastCGI module for the PHP interpreter. Lighty’s configured to listen on a public port for SSL traffic and a firewalled non-public port for HTTP.

Sitting in front of lighttpd, on the standard public HTTP port is varnish, this is a caching HTTP proxy which forwards requests to lighttpd as required, caching static HTTP requests as needed. Varnish is very configurable, using it’s own particular configuration language which essentially allows you to define exactly what you want to happen to incoming HTTP requests.

In this instance, Varnish is configured to cache incoming HTTP requests when they meet a set of standard parameters which attempt to ensure that only “cacheable” requests will be cached (i.e non-personal data, such as GET requests without cookies).

I’ve made the following changes to the configuration for this particular deployment:

• Requests with URLs which include ‘/wp-admin’, which is the WordPress administration interface will not be cached. This interface is very seldom used, and caching it would provide little-to-no benefit, and problems where it comes to returning old data. In effect, this rule isn’t really a problem, as most of the requests for the administrative interface will contain cookies anyway.
• Requests for the “fake-CDN”, which can be detected due to having a particular hostname (in this case, media.tlyk.eu) automatically have their cookies stripped. This allows media (which will always be “static” for our purposes) to be cached every time it’s requested from the cache. This will help to increase our cache hit rate.
• Requests for ‘/favicon.ico’ are discarded with a HTTP 204 response code, as these aren’t in use in this deployment.
• Headers are added to indicate the cache status of requests (X-Varnish etc…)

So then, as a Linux systems administrator at a web hosting company, I tend to spend a fair amount of time tweaking installations of web applications like WordPress to make them run quickly, making sure that they’re secure, and that they’re maintainable and securely backed-up.

As you’d imagine, in setting up a personal webserver/blog installation, like the one you’re looking at right now, I like to try and employ a bunch of the same things which I’d normally use at work.

The point of this post is to try and at least list what’s in use here, partly for my own reference in the future, but also for the benefit of anyone who finds this post via Google, or the site archives. If you’re reading this information from any domain which does not end in tlyk.eu  – then something weird has happened.

I’ll start at the very top of the stack, with the software which actually provides the page content. This post is going to be fairly long…

Continue reading →