LiveCD project, revisited

Well, it's been a while, and I haven't updated you on the sometimes frustrating LiveCD project.  So I will take a few minutes and do so.

I finished the project more than a week ago on a Friday afternoon, an hour after I was supposed to go home.  I tested it in VirtualBox on my Linux workstation, and it booted and let the user activate the VPN and access the three web pages they were supposed to access.  It did not offer them an install option, and they had immediate access in the Unity Panel to only VPN, Firefox, & the Libre Office apps.  My intent had been to offer both Firefox and Chrome, but you would not BELIEVE how hard it is to shrink a filesystem small enough that 'remastersys' can shrink it down to an ISO file under 700MB so it can be burned to a CD.  Chrome is much bigger than Firefox, so Firefox won.

I also took the CD home and successfully tested it on my old Dell laptop, and my weekend was happy and relieved.

On Monday, I gave my boss the CD, and told him to have fun with it.  He knew I had stayed late on Friday to complete the project, not because he was pressuring me, but because I just wanted to finish it so I could start the new week by simply documenting the process of creating a LiveCD for future reference.  I know, documenting my work...  I'm going to lose my SysAdmin card if the others find out. :-)

10 minutes later he brought the CD to my desk and said it didn't work.  What the heck???

I put it in my workstation, and restarted it.  No VirtualBox test this time.  I did it all the way!  Worked like a charm!

I then realized that because it was moving at the speed of CD, just getting past the initial splash screen was taking over 35 seconds on my workstation, and it has 8 CPUs and 12GB of RAM.  He had been testing it on old notebooks.  I tested it successfully on my workstation, and then returned the CD to my boss.  I said, "Don't scare me like that!  It just takes a good long time to boot up because it's moving at the speed of CD and you're using very old laptops."  So he plugged it in and booted it up.  I stood and watched, counting internally, "One-one-thousand, two-one-thousand, three-one-thousand..."   He looked at me quizzically, and I pointed out to him that I had just crossed the 30 second mark and it would be a while.  After another minute or so, I got tired of standing and sat in his guest chair to wait this thing out.  After another 3 minutes I surrendered and asked him to give me both the CD and the laptop we were testing on so I could have it at my desk and not have to bother him again for testing.

Tail between my legs, I returned to my desk and to the drawing board.  That didn't go so well, did it? Dang!

I decided that since it worked on two other machines, the CD was okay.  It must be something about the way I created it.  So I burned a copy of the original LiveCD as it came from Ubuntu.  I was using Ubuntu Desktop 12.04, i386 version.  One CD later, I was booting the laptop.  Kind of.  It got about 10 seconds into the boot process and put up an error about the kernel requiring a PAE qualified CPU, and this one was not.  Aha!   So, I asked myself, what the heck is PAE???

I Googled it and found this bit of information on wikipedia.

"In computing, Physical Address Extension (PAE) is a feature to allow 32-bit x86 processors to access a physical address space (including random access memory and memory mapped devices) larger than 4 gigabytes."

Okay, that is understandable.  Older CPUs were unable to use that much memory, and it's a nice way to increase your computer's performance.

Here's the problem.  Ubuntu decided - unilaterally - to start using PAE only kernels starting in 12.04.  One page explaining it can be found here:
http://askubuntu.com/questions/122841/why-is-12-04-using-a-pae-dependent-kernel-by-default

It's Ubuntu's right to make such a decision.  I would have preferred that they add a bit of code to their install script that detects the CPU and determines whether it as PAE compliant, and if not, installs an non-PAE kernel for that machine.

So all my work was for naught.  I had to go out and hunt down a non-PAE version of Ubuntu 12.04.  I found an unsupported version at:  http://people.canonical.com/~diwic/12.04-nonpae/  ...and used it as my base, starting over from scratch.  Two days later, I had a tested, working copy.  I had tested it on the first of the two laptops where the original LiveCD had failed for the boss.  I had tested it on my workstation, both via VirtualBox and via actual booting to the CD.  I was happy.  I had even managed to kill two stoned birds by documenting - line by line - as I ran through the process, so I was well on my way to finishing the documentation.

I gave the CD to the boss, and tiredly made my way back to my desk to rest and play with my laurels. For about two minutes.  That's when my boss called me and told me the CD had caused a 'kernel panic'.  A kernel panic occurs basically when the kernel - the heart of the Unix/Linux operating system - encounters either a software or hardware problem which it feels to be insurmountable, and it throws up the panic message and stops.  Okee dokee.  So either the CD was flawed, or the hardware on the laptop was having an issue.

I brought the CD back to my desk and repeated my tests, booting both the other notebook from the boss, and my own workstation - both via VirtualBox and via booting from the CD.  No problems.  Okay.  That means a hardware issue.  Having just wasted weeks building a PAE kernel that failed because the hardware wasn't good enough, I quickly came to the conclusion that the hardware on the second notebook was probably too old for the kernel being used in 12.04, PAE or not.

So yesterday, I started over yet again, this time using Ubuntu Desktop 11.10  i386, and this morning successfully burned a CD that I am about to begin testing.  I have also downloaded Ubuntu Desktop i386 versions 11.04 and 10.10, so I can go back a year in kernel design if need be.  I don't like the security implications, but since this is a LiveCD, I don't see a heck of a lot a hacker can do, so I will test, and if it works, I'll let the boss decide.  Perhaps he can give the 12.04 CD to the users with the newer hardware, and regress back to the older version for those who refuse to buy a newer laptop or tower for their home use.

I hope you're learning a bit about what it is to be a Unix/Linux Systems Administrator!!

Creating a LiveCD for employees to use to VPN in from home

So, today I'm back to working on the LiveCD project.  Here is what I'm trying to accomplish:

Build a LiveCD that has icons already prepped on the desktop (actually the panel in Ubuntu) that enable the user to start a VPN connection to the office, open Firefox or Chromium with the primary 3 internal-only web pages opening with request for user ID and password.  Do NOT offer the install option either at boot or on the desktop afterward.

The primary reason for this is to ease the complexity users are currently facing when trying to VPN in from home.  The secondary reason, and my primary concern, is to keep them from introducing viruses into our network that they've managed to infect their home machines with.  By booting to a LiveCD that is known to be virus-free (gotta love Linux!), we can be certain that they are coming in clean.

Obviously, the laziest way would be to just hand them an Ubuntu LiveCD and a sheet of instructions for configuring the VPN connection, and explicit instructions regarding NEVER touching the install icon.  Problems?  First, we all know it is likely that the users would never read the instructions, would complain that they cannot configure the VPN, and would immediately overwrite their precious Windows partitions with Ubuntu because they hit the install icon.

The second best solution would be to put the VPN icon in the panel, and pre-configure the Firefox startup to open the three pages.  Problem?  The only way I've found to do this is with 'remastersys', which insists on including ubiquity, the startup script that includes the install option.  If I remove ubiquity from the source installation, remastersys automatically reaches out and downloads it before creating the iso file.

Third best option is building a new LiveCD from scratch, and it's starting to look like that's what I'm going to have to do.  Oh, well, let the adventure begin!

The process I'm going through is often the way it is in systems administration.  Disclaimer: I'm about to exaggerate in order to make a point!  And this has nothing to do with any religion!  It's just a story to make a point!

You decide to write a simple script to do a bit of math.  A couple of hours into the job, you discover that in order to truly understand this math, because we in America use Arabic numerals, you need to learn the entire Arabic language.  You sigh, google up some pages on the Arabic language, and then manfully (or womanfully) begin learning a whole new language.  A few days into that, you suddenly discover that in order to truly understand the Arabic language and how people think in that language, you'll need to memorize the entire Koran, word for word.  At this point, you think back to the simple little script you wanted to write, curl up under your desk in the fetal position, place your thumb in your mouth, and wait for the shaking to stop.

Here's hoping I get the LiveCD done this week.  It's already taken far too long.  Heavy, heavy sigh...

Benny Helms

My first blog

My first blog

I've never blogged before, so this will be a new experience for me.

Professionally, I'm a Unix/Linux Systems Administrator with many years of experience.  I dabble in most flavors of Linux, but my preference at this time is Debian and its derivatives.  I'm currently typing this on a workstation that is running Ubuntu 13.10 Desktop 64-bit.

How does one become a Linux Systems Administrator?  Almost always by accident.  You start playing with Linux to see what all the hype is about.   Then you dig deeper so you can do more and more. Then the people where you work begin to see you as the "guru" when it comes to all things Linux. You're answering questions left and right, most of which you have to go Google to find answers for, all the while learning and becoming more and more proficient.  Then your company decides to try out this "Linux Thing" and hears you are the Lord Of All Things Linux, and offers you the chance to set up a server and administer it, all while continuing to do your "real job" of course.  Before you know it, you are setting up more servers as the company sees profits rising (from lack of licensing fees) and downtime decreasing (from Linux's failure to crash every other day), and the company makes riding herd on these boxes your full time job.

It's like a dream come true!  You get to play all day with the latest stuff, and they PAY you for it!!

Careful what you wish for.  After a while you begin to realize that being the only SysAdmin at a company means you wear a lot of hats, and it begins taking more time than you have in a day.  You are the security chief, the web server administrator, the mail server administrator, you're riding herd on Samba file servers and print servers.  You start to notice that they expect you to know everything about anything that has ever been near a computer.  When the power goes out and comes back up, they expect you to reset all the clocks on all the microwaves, because "you're good with techie stuff" (not that you weren't already resetting all the microwave clocks; being a geek you just can't handle seeing that clock being incorrect!)

In the beginning, you're like a dry sponge, absorbing knowledge like you've been dying of thirst.  After several years you reach a point of saturation, and you begin wondering if you shouldn't have started this in the first place.  To stay on top of your game, the learning is endless, you'll grow weary of dealing with script kiddies and hackers, you'll sometimes have days where you are tired of supporting users who never seem to make any effort to figure out for themselves what is wrong before calling you.  It sometimes feels overwhelming, and you begin wondering what it would be like to be a forest ranger, out in the middle of the wilderness with no network signal and no cell phone signal.  Perhaps riding a horse in Montana in the middle of a huge ranch, keeping cattle safe.  Almost anything that has nothing to do with technology.

The problem is that you are too good at what you do, you still enjoy it on most days, you're still learning new things every day that keep the job exciting, and you've reached a salary level that you cannot match in any other field without investing a decade in training and experience.  You know you'll be a SysAdmin until you retire, and you need to adjust your mindset to see all the positive in your job, like helping people and keeping servers safe and secure.

Whenever I reach a place where I've allowed the negative to creep into my mind instead of focussing on the positive, I think of an episode of "The Jetsons" I saw as a kid.  George Jetson is complaining to his wife about how hard his day was at work, sitting at his desk pushing buttons.  At the time, I thought George was a whiner and should enjoy such an easy job, instead of having to dig ditches, or some other back breaking work.  Now I realize I have become George Jetson.  All I do all day is push buttons, and then I have the audacity to whine about it.  Granted, the stress level is more draining than the button pushing, but you understand my point.  We SysAdmins have a pretty decent way of making a living, even though many of us that have been doing it for more than a decade reach a point where they hate it, and would do almost anything else if they could earn a decent income.

You've just got to keep focusing on the positive, and ignore the negative.  That job on the horse in Montana has it's own problems, the winters are ridiculously cold (I know, I lived there for 11 years), and it can get lonely out there with only the cattle for company.  Every job will have a negative aspect and a positive aspect.  What you focus on is what you experience.  I focus on the positive, and I have come to thoroughly enjoy my profession, even after many years of doing this.

I think that's more than enough for my first blog posting.  I welcome feedback, especially from other SysAdmins.

Have a great day!

Benny Helms