It is hard to believe that it was 10 years ago that I started with my current employer. 10 years in the same job is a lifetime in the current “stay in a job for a year or two and move on” culture; particularly in the tech market. What can I say? I love my job. I love the “fireman” aspect of it; the sense of being under the gun that I experience on a nightly basis:

“Hey Alex, there’s 20 seconds on the clock and you’ve got a down server running some application you’ve never even heard of….what are you gonna do?”

*cracks knuckles*

One of the things that makes me love my job so much is the huge variety of things I have to deal with. No day is ever the same. Ever. Personalities, issues, new trends and a whole gamut of other things- they all change from day to day. I thought that an interesting way to celebrate my 10 year anniversary would be to create a list of things I’ve “learned” in that time that don’t change.

To start, I am going to look at the things that you, the person dealing with your systems administrator, should know and/or keep in mind:

1. You came to me for help.

This may seem obvious at first glance, but it is number one on my list for a reason. When a client is upset about the current situation they have found themselves in and I provide them a solution (or solutions), all too often they become argumentative and want to do things the way they want to do them. In the name of professionalism, I never actually say it, but I always find myself thinking: “You came to me for help. If your way worked, would you be talking to me right now?” I’m looking to provide you solutions, solutions you came to me and asked for, so why would you not then listen to me? Keep an open mind and somewhere between your requirements and my experience, we will find an answer.
2. I don’t suggest things by accident.

If you’ve asked me to set your ssh to permit root login and to set the root password to 123456, I’m going to suggest alternatives to you. Depending on what the request is, I’m going to word some suggestions more strongly than others. Sometimes I will flat out refuse (such as the root/ssh example I gave) and tell you why. Please listen to me. I’m not doing this because it’s a fun way to pass the time or I am bored and want to play games with you and your business. I’m doing it because I’ve seen 1,000 customers do exactly what you are about to do (including ignoring my advice) and I know how this story ends. I know what phone call I am going to receive in a few days and what you will then need me to do so that we can correct the exact mistake I tried to help you prevent. When I try to head that problem off at the pass? Let me.
3. I’m here to help you…

…so please don’t lie to me. I’m not going to smack your hands (like some UNIX version of the Catholic nuns with their legendary rulers) if you have made a mistake. We’ve all made mistakes. The quickest way for me to resolve the problem is to know exactly what you did. When I ask if you’ve changed anything recently, the answer I get 100% of the time is: “Absolutely not!” The truth of the matter is that probably 60% or more of the time? You actually did. It just took me longer to find it and prove it because you didn’t tell me up front. I’m your partner in this, not your judge, so help me to help you.
4. I’m not an expert in everything.

No one is. Please try to remember this when you call up, super excited, asking about FlibbertyGibbet version 3.4 and demanding that it be put in place now, now, now. If you can give me the time to look at it, I can assess if we can help. However, you also need to understand that there is simply too much application software out there for me to innately know, install, configure, and maintain every single bit of software out there. I love the chance to learn something new and if you’ve ever presented me with that opportunity, you’ve probably heard the words: “You know what? I’ve never even heard of that before. Give me about an hour to do some research and I’ll get back to you.” Most of the time, I can do it. Occasionally, I’ve had to suggest alternatives. Some clients are more gracious than others in how they respond to the latter. So try to bear all of this in mind. After all, It is linux…there’s always about 20 different ways to do exactly what you are trying to do! It may well be that a more established application that we support would be better than a brand spanking new one that we don’t, purely in terms of the long term supportability of your production environment. Newer is not always better.
5. Plan for what could happen.

I know you think your application is rock solid. It might well be 99% of the time. That 1% of the time is what you should be planning for. I actually can not stress this one enough. At some point when I am working with a client, I will suggest to them various backup strategies and there’s always this moment, I can hear it in their voice across the phone: “Oh, I’ll get to that someday.” It kills me every time I hear it, because those are almost always the same clients I see that come back to me with a failed hard drive or a compromised server and they do not have backups. There’s no magic wand I can wave to save the day in that case. When it comes to your business and its on-line life? BE A PESSIMIST. What’s the worst that can happen and what do you do if that happens? You may never need that plan, but if you do? You have no idea how happy you’ll be that you actually have it.
6. Hardware fails.

It doesn’t matter how long you scream at me on the phone, upset that a hard drive died, it isn’t going to change this base fact: If it has moving parts it will die some day. “Some day” can be sooner and “some day” can be later. There’s nothing you or I can do about it except to plan for that event and hope it is later. So please, please, please remember this fact and look at #5 again. And then read #7.
7. RAID = redundancy / Backups = recovery.

I think that a lot of my clients think of their RAID as a disaster recovery plan, which is why I include this here. It’s an easy misconception to make. If you have multiple drives, with data spanning those drives, and you can just replace any one drive if it fails without losing the data; surely that’s better than a backup, right? No. Too many clients do not consider what happens when all of the drives fail. Trust me, there are ways that all of the drives in a RAID can fail or be corrupted (see #6 above) and all the RAID in the world isn’t going to solve a situation where all of the drives are unusable. The hope is that you never have to use a backup, but make sure you have it for when you need it, even if you have a RAID configuration.
8. Kindness goes a long way.

I’d say that a good 50% of what I do on any given night is done as a favor. It isn’t technically supported, but I do it because I know it is important to you and your business. You came to me asking for help and I’m where I am because I want to help. It is hard-wired into me at the most basic of levels. Seriously. You don’t do this job for 10 years if you aren’t interested in helping people. The customers that know how to say thank you? That appreciate the work I’ve done for them? They are remembered. It is worth pointing out that the reverse of that statement is true, as well. I’m not asking for you to fall all over me, fawning. I’m am saying that if I’ve gone out of my way for you, it is nice to hear a “thank you” in return. You’d be surprised how many people just answer with another laundry list of what they want done next, taking for granted that the favors will continue. This particular bit of information works well in every aspect of life, actually. Your parents were right: “please” and “thank you” go a long, long way.
9. Your server is a child that never sleeps.

I’ve been telling clients this one for years. Some listen, some don’t. Your server is a 3 year old child that never sleeps. It needs constant attention, monitoring and maintenance. There is no situation that involves you uploading code and then walking away, never looking at it again, that doesn’t end badly. Most OS vendors have included automatic updates to their server software at this point, but most servers aren’t just running OS provided software. I’d suggest that you make a list of all 3rd party applications you’ve installed on the server, from basic php form mail scripts to the modules that you load inside of your CMS. Anything extra you installed on that server should be listed. Bookmark the homepage of each item on that list and make it a point to visit those web pages, at least once a month (if not once a week). If your CMS includes an automatic update method, check it regularly and use it. This will generally keep you updated with security releases and bug fixes as they are released for the CMS you have installed. If you are not staying on top of what versions you are running, I assure you that there is a malicious user out there that is.
10. Don’t be afraid to ask questions.

I’m not a god. I make mistakes. If you think I’ve overlooked something, or that I’ve not given something my full attention, call me on it. Make me make you understand. There’s a fine line between arguing for arguing sake and genuinely feeling like you think the problem wasn’t addressed fully. If you ever feel it is the latter, say so. Maybe I need to include more documentation, log entries to back up what I am saying, etc, but whatever it is, you are entitled to it. Don’t let any systems administrator tell you otherwise.

Now, lest people think I am some smarmy systems administrator that thinks he’s above reproach, let me also do a top 10 of things that I’ve learned from the other side of this equation.

Here is an equally important list of advice for systems administrators to remember in dealing with their clients:

1. Give every problem your full attention.

The reverse of this is last on the list above, but first on this one. Why? Because this is a rough one for experienced admins. Even for me. 10 years of “how do I stop spam?” questions can lead to a certain…sureness…on your part that you know exactly what the answer to the question is without even looking at it. Always look. Always test. Always bring proof to the table of what you are saying rather than just going from your gut. If you can’t prove to the client what you are saying, why should they listen to you? It is easy to mistakenly dismiss a legitimate issue because you seen 50 issues that sound the same in the last 3 days. Make sure you are looking at this issue, the one before your eyes, and not just making assumptions based on the other issues, now past.
2. Never stop being willing to learn.

You’re human. Habits form. You create tracks and ruts in the world of *nix that you feel comfortable in. Or perhaps you’ve just never really explored something because it hasn’t been required. Never be afraid to try. You might fail, sure, but you could also succeed. You’ll never know until you try. The urge to do what I call “hot potato” with something you don’t think you know well enough can be huge. So let me ask this: if no one anywhere was ever willing to look at anything except what they already knew? Who would ever learn it? Be brave. Dive in. You’ll surprise yourself far more often than you will disappoint yourself. I can not tell you the number of times I’d had a gut reaction of “I don’t know this, I’ll leave it for someone else,” that then became “Man, this was easy. Why haven’t I done this before now?”

3. There is no such thing as a hard “no”.

Tell yourself and mean it: You do not get the luxury of simply saying “no” and ending a discussion. Your client came to you for answers. Provide them. You may not be able to provide the exact answer they were looking for, but that is where your experience comes into play. Look at what they are trying to do, understand it and then figure out if there are other options that may work for them that can be done instead of what they originally asked for. If so, you’ve just solved a problem rather than slammed a door. One is far better than the other when it comes to long term client happiness.

4. Always make a backup.

Always. Always. Always. I can not stress this one enough. It doesn’t matter if you’re just make a quick tweak that you’ve done a thousand times before or if you’re running some big huge complicated bash script you just banged out to save yourself 3 hours of work….always make sure you can undo what you just did. Nothing sucks more than that momentary stomach punch of: “oh man, what did I just do?” from a typo’ed command or malformed scripting. Except perhaps having to tell a customer that you did it and can’t undo it. Always make a backup and you’ll never have to worry about that.

5. Never be afraid to double check.

It is never a bad thing to be sure you understand completely what the end goal is, particularly if this is a production environment you are working on. If you are unclear, need more information or just flat out don’t fully understand what the customer is asking for: ask for that information. I know that there can sometimes be a fear of being seen as “dumb” for not having “gotten it” the first time and so people will make their best guess about what was desired and move forward from there. Get rid of that. What’s more important? Your ego? Or the blow it will take if you’ve misunderstood what a client wanted and caused issues with their production environment? Be willing to ask “dumb” questions and ensure you know exactly where you are trying to get to before you start the work.

6. Check before AND after.

Before doing any major work, always check how the application works. Get a baseline feel for it and how it looks/performs. When you’ve done your work, go back and look again. This one is subtle. It seems totally obvious and yet it is one that I have encountered repeatedly over the years, in one form or another. Always double check your work and ensure a client’s environment is functioning the way it was before you began the work. There’s nothing worse than doing planned work, signing off on it, and then 12 hours later finding out a secondary site on the server has been functionally broken and could have been fixed if someone had just looked. It can be easy for us to forget that this is someone else’s bread and butter, their livelihood, because we only deal with the server and are abstracted from the end business a bit. Get in the habit of doing a lay of the land first, then do your work and then verify that work hasn’t broken anything. Simple, yes. Invaluable? You better believe it.

7. Own your mistakes.

In my experience, a client responds better if you are completely upfront with them when you make a mistake. After all, it is very likely the mistake caused some sort of bump for them, depending on what the mistake was. They deserve, and are owed, an explanation. Trying to vague it away and talk circles around it breaks the trust between you and your client. Be upfront and honest; explain how and why it happened and why it won’t happen again. They know a mistake was made. You know you made it. Client trust and confidence is the key here and it is something that is very easily lost. Honesty is the best policy even if it means taking a blow or two to your ego in the short term.

8. Know when you’re out of your league.

10 years in this job and I still encounter things that I am simply not qualified to deal with. I’ll always make a try at it (see #2), but I also think that being the best, most honest assessor of your own strengths and weaknesses is a valuable time saver for both yourself and your client. I can research all day long, but I’m probably not going to become an Oracle DBA in that time. You should become adept at figuring out what you can do within a reasonable amount of time and what is going to be better referred onward to a specialist. Your clients aren’t paying for you to educate yourself on their dime. The goal here is the best and quickest answer for a client’s particular problem. Sometimes that answer isn’t you.

9. Know where to find the information.

You’re not the only person to ever see the issue you are looking at. Google is a system’s administrators best friend. Become adept at remembering where information is stored rather than trying to store it all in your own head through rote memorization. Above all….search. And search. and SEARCH. You’ll find the answer 90% of the time. For the other 10%? Well, there is always strace. Ah strace… you beautiful creature you.

10. Start with “is it plugged in?”

If you’ve been a systems administrator for any length of time, you are probably chuckling to yourself right now. How many times have you overcomplicated an issue? As your experience grows, you start to see problems as much more complicated things then they actually are. Your mind immediately leaps to the more challenging, functionally more interesting problems that are currently exciting you. As a result, you can completely overlook the basic/simple cause of the actual problem. Always start at the low end, simple things moving towards the complex; not the other way around.

And that’s it really. What 10 years of my job has boiled down to. Software comes and goes, OSes rise and fall and the new “hotness” soon becomes the old boredom…

…the only thing that actually stays the same is how to correctly approach a problem and how to work together, professionally, to get to the solution.

Category: linux

3 Responses to A Decade of “Wisdom”

  1. FJ!! says:

    Nr 3 is because people have been told repeatedly that if they make changes (open the case, switch out a part, change a setting) “the warranty is void” and they won’t get help any more. Even tech support, like at work, will very quickly say, when you have installed something ‘not on the list’, “You’re on your own, I am terminating the call.”

    Your employer may have very forgiving policies that your customers are simply not used to.

  2. kale says:

    This should be required reading for every new admin. Seriously, print it out and put it on new hire desks. Everything is 100% true, except for the latter #10 — it’s not “is it plugged in”, but always always ALWAYS “/etc/hosts hurr durr” as the answer to grossly overcomplicated issues. Every. Single. Time. I just look at /etc/hosts first now.

  3. Miguel says:

    Fully agree on every single point. That is experience distilled. A must read as kale said.

Leave a Reply


gives good tech
Kale is one of the smartest people I know

Racker Hacker
Major is always good for leet deetz