Let’s face it, “500 internal server error” from apache is about the most annoying, unspecific thing you run into on a linux box. It could be anything (and usually is) and the logging associated with it is next to useless. The only thing worse is Perl’s unspecified error logging. :p

So…how do you find out what is wrong? Simple.

Run the following command from another server/location:

$ telnet yourdomain.com 80
Trying yourdomain.com…
Connected to yourdomain.com.
Escape character is ‘^]’.

it will give you a blank prompt after that escape character line. Type the following:

GET / HTTP/1.1

hit return ONCE.

Now, go to your server where the website is hosted. Do the following command:

netstat -natp | grep “ip.address.of.the.server.you.ran.telnet.from”

You should get back something like:

tcp 0 0 ::ffff:blah.blah.blah:80 ::ffff:ip.address.of.the.server.you.ran.telnet.from:19417 ESTABLISHED 25051/httpd

That bit just before the httpd is what you want. That is the process id of the apache process you are connected to. Now run:

strace -s 6666 -p 25051

Where the 25051 is the number that was actually in your output. In case you are wondering, the -s sets the number of characters each line can be. If you don’t set this, you end up with truncated lines that make it nearly impossible to tell what is really going on. So I just do the -s and a large number to be safe.

Now go back to your other window and just under your GET command, type:

host: yourdomain.com

hit enter twice and then go watch the output in strace.

Now, I know what you are thinking. I thought the exact same thing the first time I ever tried to use strace. OMG WHAT THE HELL DOES ALL OF THAT MEAN??? Strace output can be VERY wall of text. Just take a deep breath and then actually look at what it is telling you. Strace shows you every call the process makes. Every file it opens and reads. Everything it did is recorded right there, so if you start at where the process dies and move backwards, you can generally put it all together. It just requires taking the time to read each line and try to understand what it is telling you.

Trust me, once you get the hang of it? This will become the most valuable tool you have for troubleshooting “what the hell is apache doing???” issues and other obscure problems were a process isn’t doing what you think it should be, but you don’t get any relevant errors to point you in the right direction. Strace is easily one of my favourite tools. Live it, love it, use it.


Leave a Reply

Categories


gives good tech

tech.superhappykittymeow.com
Kale is one of the smartest people I know

Racker Hacker
Major is always good for leet deetz