This is fairly straight forward and surprisingly simple:

First, get the dependency out of the way:

yum install httpd-devel

Once you have that package (and its dependencies), you can then install mod_rpaf.


(you may want to go to just to verify that is still the current version)

tar zxvf mod_rpaf-0.6.tar.gz
cd mod_rpaf-0.6
apxs -i -c -n mod_rpaf-2.0.c

You can then create the file /etc/httpd/conf.d/mod_rpaf.conf, with the following content:

LoadModule rpaf_module modules/

# mod_rpaf configuration

RPAFenable On
RPAFsethostname On
RPAFproxy_ips X.X.X.X
RPAFheader X-Forwarded-For

Be sure to replace X.X.X.X with your proxy IP.

Restart apache and then check your logs to see that you are now seeing the IPs of your visitors rather than the private proxy IP.

I’m usually someone who learns new things because I become irritated with “the way things are”. The following tip is a perfect example. I grew tired of my usb drive going into sleep mode and causing issues with some of the things I use it for, so I decided to see if I could tell it to never go to sleep.

Turns out? You can.

Redhat flavors?

yum install sdparm


sudo aptitude install sdparm

Once that is installed, you can point it at your USB device. In my case, it was /dev/sdc. If you’re unsure what device it has been labeled as, consult the output of “dmesg” and find it.

So, point sdparm to it with the command: sdparm -a

[root@pinja ~]# sdparm -a /dev/sdc
/dev/sdc: Seagate FreeAgentDesktop 100F
Power condition mode page:
PM_BG 0 [cha: n, def: 0, sav: 0]
STANDBY_Y 0 [cha: n, def: 0, sav: 0]
IDLE_C 0 [cha: n, def: 0, sav: 0]
IDLE_B 0 [cha: n, def: 0, sav: 0]
IDLE 0 [cha: n, def: 0, sav: 0]
STANDBY 1 [cha: y, def: 1, sav: 1]
ICT 0 [cha: n, def: 0, sav: 0]
SCT 9000 [cha: y, def:9000, sav:9000]
Power consumption mode page:
ps_id 0 [cha: n, def: 0, sav: 0]
SAT ATA Power condition mode page:
APMP 0 [cha: n, def: 0, sav: 0]
APM 0 [cha: n, def: 0, sav: 0]

Notice STANDBY_Y has a 0 beside it? That means standby is disabled (because I shut it off earlier). The actual flag can change a bit from drive to drive (some just say STANDBY, for example) so just pay attention to what is listed there. If you find it has a 1, you will need to run the commands:

sdparm --command=start /dev/sdc
sdparm --clear STANDBY_Y -6 /dev/sdc
sdparm -save -6 /dev/sdc

Another “sdparm -a /dev/sdc” should then show you the cleared value and we’ve saved it so it will persist through a reboot.

Category: advice, hardware, linux

Ran into this one today for a client and it has been such a long time since I had to do any NAT routing with iptables, it took me a bit to suss it out again how it was done properly. In light of that, documenting it here so it is easier to find 5 years from now when I need to do it again. 😉

Set forwarding in sysctl.cnf:

net.ipv4.ip_forward = 1

Apply it:

sysctl -p

Then set the two rules. One to route the traffic to the remote host and the other to route it back:

# iptables -A PREROUTING -t nat -p tcp -d –dport 3306 -j DNAT –to-destination

# iptables -A POSTROUTING -t nat -p tcp -d –dport 3306 -j SNAT –to-source

Presto! All traffic to “” on port 3306 is now being routed out to 3306 on “”.

Category: iptables, linux

Let’s face it, “500 internal server error” from apache is about the most annoying, unspecific thing you run into on a linux box. It could be anything (and usually is) and the logging associated with it is next to useless. The only thing worse is Perl’s unspecified error logging. :p

So…how do you find out what is wrong? Simple.

Run the following command from another server/location:

$ telnet 80
Connected to
Escape character is ‘^]’.

it will give you a blank prompt after that escape character line. Type the following:

GET / HTTP/1.1

hit return ONCE.

Now, go to your server where the website is hosted. Do the following command:

netstat -natp | grep “”

You should get back something like:

tcp 0 0 ::ffff:blah.blah.blah:80 ESTABLISHED 25051/httpd

That bit just before the httpd is what you want. That is the process id of the apache process you are connected to. Now run:

strace -s 6666 -p 25051

Where the 25051 is the number that was actually in your output. In case you are wondering, the -s sets the number of characters each line can be. If you don’t set this, you end up with truncated lines that make it nearly impossible to tell what is really going on. So I just do the -s and a large number to be safe.

Now go back to your other window and just under your GET command, type:


hit enter twice and then go watch the output in strace.

Now, I know what you are thinking. I thought the exact same thing the first time I ever tried to use strace. OMG WHAT THE HELL DOES ALL OF THAT MEAN??? Strace output can be VERY wall of text. Just take a deep breath and then actually look at what it is telling you. Strace shows you every call the process makes. Every file it opens and reads. Everything it did is recorded right there, so if you start at where the process dies and move backwards, you can generally put it all together. It just requires taking the time to read each line and try to understand what it is telling you.

Trust me, once you get the hang of it? This will become the most valuable tool you have for troubleshooting “what the hell is apache doing???” issues and other obscure problems were a process isn’t doing what you think it should be, but you don’t get any relevant errors to point you in the right direction. Strace is easily one of my favourite tools. Live it, love it, use it.

If you’ve ever had a system drift pretty wide on the time, you are aware that ntp can’t update the time after a certain amount of drift. I’ve found this to be a particular problem on some systems from a reboot, where the time never gets manually set and so it stays off kilter and just keeps drifting more and more.

On “Redhat” flavor boxes, you can edit


and change

OPTIONS=”-u ntp:ntp -p /var/run/”

to be

OPTIONS=”-x -u ntp:ntp -p /var/run/”

That -x is a very tiny change, but a huge effect. What this does is when you stop/start ntp (or it starts on a reboot of your system), it does the equivelent of

ntpdate -u time.server.of.choice

ie, forcing the manual update against your chosen time server. No more manually fixing drift that has gotten too wide. From a reboot the time is set to a value that ntp then can automatically update and keep updated moving forward.

Try running

service ntpd restart

and you’ll see it do the manual time update.

# service ntpd restart
Shutting down ntpd: [ OK ]
ntpd: Synchronizing with time server: [ OK ]
Syncing hardware clock to system time [ OK ]
Starting ntpd: [ OK ]

Category: advice, linux, ntp

To add a new tmpfs mount in a specific location:

mkdir -p /path/to/directory/
mount -t tmpfs -o size=512M,mode=0744 tmpfs /path/to/directory

and then add the following to /etc/fstab:

tmpfs /path/to/directory tmpfs size=512M,mode=0777 0 0

The potential returns this can give something like your session (if you’re still using files and not, say, memcache) or cache files is pretty significant. As always, test in your own environment before rolling out into production. And in case you weren’t aware, a reboot will wipe the contents of that folder…after all, it is a ram disk, not actual file storage.

To grow the size of an existing tmpfs:

mount -o remount,size=2G /path/to/directory

And don’t forget to modify /etc/fstab if you want it to be permanent!

Category: file systems, linux

Update: 09/03/2012

Lsync has been added to Redhat’s repository, so those of you using redhat can do:

yum install epel-release
yum install lsync

to get lsync installed even easier than the below guide.

I’ve mentioned Lsync here before, but I figured I could do a bit more documentation on this service.

Lsync is a very handy software package for doing near instantanious updates of files/directories from a central location to many external locations. Think star topology. It works particularly well in the cloud where you tend to find large number of web servers that need the same content across them at any given moment. It will NOT work if you can not isolate your application to doing uploads to a single server in such a load balanced configuration. You were warned. That said, as long as all your file uploads are going to a single server, lsync can keep your other web servers current and up to date with, at most, a 20 second delay.

All that being said…here is how you install it.


Category: linux, replication

This is a short and easy way to track down the source of a compromised PHP script that is spamming out of your system.

Change the /etc/php.ini sendmail path to:

sendmail_path = /usr/local/bin/sendmail-php -t -i

and then create that file with the following contents:


logger -p sendmail-php: site=${HTTP_HOST}, client=${REMOTE_ADDR}, script=${SCRIPT_NAME}, filename=${SCRIPT_FILENAME}, docroot=${DOCUMENT_ROOT}, pwd=${PWD}, uid=${UID}, user=$(whoami)

/usr/sbin/sendmail -t -i $*

chmod +x that file and restart apache.

Now when a php script is called and sends a piece of mail, it is logged to the log file and you can then trace back what is legitimate and what isn’t. Spammers tend to be very…brute force…so it is usually pretty apparent, fairly quickly, which script is being abused.

Category: linux, php, spam

Ok, folks…

I realize that you are operating a business. I realize that keeping costs low is the name of the game. Honest, I do.

That being said? You need to understand something. Black Friday/Cyber Monday is not a surprise holiday that was announced 2 days ago. You’ve known this day was coming for months. You know it and I know it. So when you call me 24 hours before Black Friday, desperate to scale out your solution NOW NOW NOW because your site is going to tank otherwise?

Well, you have no one to blame but yourself. Sure, I’m going to do my best to help you. That’s what I do, but you need to understand how many clients I have. How many clients I have that think just like you do, all of whom are calling me 24 hours before Black Friday to make everything better.

In your own best business interests, plan to scale out your configuration 2-3 weeks before Black Friday and take it back down 2 weeks or so after NYE. Just put that in your budget every year. Understand that it is the name of the game. Because when you and thousands of others push administrators like me to rush scale out your configurations in a 24 hour window of time with no real opportunity to test?

Well, you’re just inviting trouble when the traffic begins to flood in.

Scale early, test often.

Just some friendly advice from a guy who has watched companies like yours make the same mistake year after year after year.

Category: advice, linux

Using lsync to keep your code current between machines but it has suddenly stopped and you can figure out why?

The problem is most likely the fact that inotify has a limit set on the number of files it can watch:


If you exceed that number, lsync stops updating. Increase this value and restart lsync.

Category: linux


gives good tech
Kale is one of the smartest people I know

Racker Hacker
Major is always good for leet deetz