Category Archives: troubleshooting
So, yesterday, a co-worker and I were scratching our heads about a plesk issue we ran into. A client had been upgraded from an older plesk to plesk 10.4.4. In the process, their chrooted SFTP accounts stopped working. Attempting to su to those users gave:
# su – username
system error: No such file or directory
When I picked up the escalation, I went right to the usual chroot issues. Permissions, ownership, etc. No dice. I noticed that the users shared a UID with another user, but after a quick manual change to see if that impacted it, still no dice. Another co-worker found that the issue was indeed the UID, but you had to run a series of commands afterwards. I wanted to document this here so that future people searching google for this error will see something other than the useless “cron” related posts on the plesk forums that I was wading through yesterday. :p
1) Change the UID of the user
2) /usr/local/psa/admin/sbin/chrootmng --remove --source=/var/www/vhosts/chroot --target=/path/to/chrooted/users/home/directory
3) rm -rf /path/to/chrooted/users/home/directory/etc
(for some reason --remove leaves this folder behind)
4) /usr/local/psa/admin/sbin/chrootmng --create --source=/var/www/vhosts/chroot --target=/path/to/chrooted/users/home/directory --setup-user username
That should get your chroot SFTP users back up and running if you encountered the same error. Big thanks to the co-worker that found the ultimate order of items and wasn’t thrown off the track of the UID issue like I was… and may our frustration save the rest of you out there in google land a bit of time.
However, be warned! Plesk no longer supports chrooting for users other than the primary FTP user of the domain. As a result of this, you are going to run into permission issues with your subusers that you used to have sharing the UID. You’re going to need to learn about setting up ACL rules to counteract this.
Let’s face it, “500 internal server error” from apache is about the most annoying, unspecific thing you run into on a linux box. It could be anything (and usually is) and the logging associated with it is next to useless. The only thing worse is Perl’s unspecified error logging. :p
So…how do you find out what is wrong? Simple.
Run the following command from another server/location:
$ telnet yourdomain.com 80
Connected to yourdomain.com.
Escape character is ‘^]’.
it will give you a blank prompt after that escape character line. Type the following:
GET / HTTP/1.1
hit return ONCE.
Now, go to your server where the website is hosted. Do the following command:
netstat -natp | grep “ip.address.of.the.server.you.ran.telnet.from”
You should get back something like:
tcp 0 0 ::ffff:blah.blah.blah:80 ::ffff:ip.address.of.the.server.you.ran.telnet.from:19417 ESTABLISHED 25051/httpd
That bit just before the httpd is what you want. That is the process id of the apache process you are connected to. Now run:
strace -s 6666 -p 25051
Where the 25051 is the number that was actually in your output. In case you are wondering, the -s sets the number of characters each line can be. If you don’t set this, you end up with truncated lines that make it nearly impossible to tell what is really going on. So I just do the -s and a large number to be safe.
Now go back to your other window and just under your GET command, type:
hit enter twice and then go watch the output in strace.
Now, I know what you are thinking. I thought the exact same thing the first time I ever tried to use strace. OMG WHAT THE HELL DOES ALL OF THAT MEAN??? Strace output can be VERY wall of text. Just take a deep breath and then actually look at what it is telling you. Strace shows you every call the process makes. Every file it opens and reads. Everything it did is recorded right there, so if you start at where the process dies and move backwards, you can generally put it all together. It just requires taking the time to read each line and try to understand what it is telling you.
Trust me, once you get the hang of it? This will become the most valuable tool you have for troubleshooting “what the hell is apache doing???” issues and other obscure problems were a process isn’t doing what you think it should be, but you don’t get any relevant errors to point you in the right direction. Strace is easily one of my favourite tools. Live it, love it, use it.