[frgeek-michiana] Re: Server update gone bad

  • From: Tom Brown <tbrown@xxxxxxxxxxxx>
  • To: frgeek-michiana@xxxxxxxxxxxxx
  • Date: Thu, 05 Jan 2012 11:17:05 -0500

I tried to boot the server off an Ultimate Boot CD (UBCD) Phil carries in his kit. The goal was to find, copy and edit the GRUB conf file. btw, the contents of the Grub conf file are displayed on the console for about 1-2 seconds, too fast for me to pause the console output.


The server wouldn't boot off the optical drive even though we set it as the first boot device. The BIOS continued to boot off the hard drive. Phil and I disconnected and reconnected cables with no improvement. Assuming the BIOS doesn't see the optical drive or the drive is broken, we installed a DVD ROM from inventory. We must have disconnected the floppy drive because the server won't boot and throws the type of error you'd expect from a bad or disconnected FD. We ran out of time and will pick it up again tonight, starting with reconnecting the FD.

Here is what I transcribed from the end of the console on booting. It appears the BIOS is not using UUID. The init script finds hda (hda1, hda2) and hdc (hdc1, hdc2) so I’ll quote the init output from there.

<boot log snippet>

Begin: Mounting root files system ...
Begin: Running /scripts/local-top ...
Begin: Loading md: raid1 personality registered for level 1
Success: loaded module raid1.
done.
Begin: Assembling all MD arrays ...
md: md0 stopped.
md: bind<hdc2>
md: bind<hda2>
raid1: raid set md0 active with 2 out of 2 mirrors
mdadm: /dev/md0 has been started with 2 drives.
Success: assembled all arrays.
done.
Begin: waiting for udev to process events ... done.
done.
Begin: Running /scripts/local-premount ... kinit: name_to_dev_t(/dev/hda1) = hda1(3,1)
kinit: trying to resume from /dev/hda1
PM: Starting manual resume from disk
kinit: No resume image, doing normal boot...
resume: libgcrypt version: 1.4.1
done.
mount: mounting /dev/hda2 on /root failed: Device or resource busy
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev failed: No such file or directory
done.
mount: mounting /sys on /root/sys failed: No such file or direcotry
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn’t have /sbin/init.
No init found. Try passing init= bootarg.

BusyBox v1.10.2 (Debian 1:1.10.2-2) built-in shell (ash).
Enter 'help' for a list of commands.

/bin/sh: can’t access tty; job control turned off
(initramfs) _

</boot log snippet>

Tom --


Imre B wrote:
# blkid
will show disks with their UUID

On Fri, Dec 30, 2011 at 1:45 PM, Tony Germano <tony_germano@xxxxxxxxxxx> wrote:
Is the UUID used anywhere besides fstab and maybe the boot loader? It makes
sense to use it there when you have multiple drives because it guarantees
that your mounts will all work even if boot order changes for some reason.

Each file system is assigned a UUID upon creation. You can use the basic
tools for that file system to find out what the UUID is. It will never
change, even if you move the drive to a different server. It will also not
change if you clone the partition.

If you are able to boot from a cd, I think you can use tune2fs -l
/dev/<hdxx> to see the UUID for the file system on a given partition
(assuming it is ext2/3/4.)

This page also lists some other ways (I haven't tried them) to get the
UUID http://liquidat.wordpress.com/2007/10/15/short-tip-get-uuid-of-hard-disks/
The ls -l /dev/disk/by-uuid way looks pretty cool. I didn't know that
directory was there.

Tony


Date: Thu, 29 Dec 2011 17:46:07 -0500
From: tbrown@xxxxxxxxxxxx
To: frgeek-michiana@xxxxxxxxxxxxx
Subject: [frgeek-michiana] Re: Server update gone bad

Sorry to reply so late. I've been very busy over the holiday and out of
town for a few days.

I ran 'apt-get update' using the default 'stable' sources, but that
listed a lot of unverified packages which turned out to be Squeeze
packages so I never started the upgrade. After some research I realized
I needed to stay at Lenny by changing the sources from 'stable' to
'Lenny' as stable now means 'squeeze'. That seemed to work as the number
of unverified packages were very few.

I don't know UUID. I'll research it at the workshop tonight.

afaik, there is no data backup. I'll try booting with a rescue disk and
poke around. As a last resort, I will pull the drives and try to read
them on another system. I don't know yet whether the drives are SCSI,
SATA, PATA or some combination. I believe the drives were in some sort
of RAID config.

Tom --

Richard Zimmerman wrote:
Reminds me since we are on the subject...

Colo server IS LOCKED to "Lenny" and I have every intention of leaving
it there....

If a new server admin takes over, his or her call then...lists

BTW, also speaking from experience after I toasted my home server on a
blind 'dist-upgrade' option.... (from the Lenny to Squeeze upgrade)

***********************************
NEVER EVER USE IT AGAIN
***********************************

That BTDT thing I mentioned in the previous email also happened on my
home server but I got lucky and recovered it....

Now days I **ONLY** use the 'safe-upgrade' option...

Lessons learned the hard way about 8 months ago...

Goose

/ Where is the old RedHat 5.2 when you want it? It was simple and 'just
worked'
// Debian Lenny is the same way. Functional and 'just works'
/// Why keep screwing with it...???


On Sat, Dec 24, 2011 at 7:14 PM, Richard Zimmerman <ke4rit@xxxxxxxxx>
wrote:

If I'm reading it correctly. You pretty well screwed the server....

The new Debian forces everything to that NOT SO wonderful UUID Crap
for everything. If you then rolled back to Lenny, congrats, you own a
toasted server....

BTDT, you pretty screwed... That happened to me.... Happen to have a
backup of the server?

Goose

/ Happened on a server at work. FORTUNATELY, I had grab some dead
hardware out of the back room and had used it so it wasn't a
production server I toasted...

// Left all work servers @ Lenny and have been researching finding a
distro that DOES NOT use UUID. That crap is for the birds....

/// Maybe force it UUID in Debian for the win!

//// If I can't find a distro that doesn't use UUID, guess I'll be
retiring as a server admin....




On Sat, Dec 24, 2011 at 12:41 PM, Tom Brown <tbrown@xxxxxxxxxxxx>
wrote:

Mike started a hard drive wipe too late to complete by closing time
last
Thursday. I started a warehouse server update which also didn't
complete
before we departed so Mike and I left the systems running with the
intention
I would stop by the warehouse today and turn off the computers.

Mike's DBAN wipe completed so I turned off the PC. The server update
completed so I rebooted to test the update.

When the server first tried to boot it died trying to load drivers for
hdc
or the next item after that; I couldn't tell which. I turned the
server off,
booted and got the following errors.

----------

Running /scripts/inti-bottom ... mount: mounting /dev on /root/dev
failed:
No such file or directory
done
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. Try passing init=bootarg.

BusyBox v1.10.2 (Debian 1:1.10.2-2) built-in shell (ash).
Enter 'help' for a list of commands.

/bin/sh can't access tty; job control turned off
(initramfs) __

----------

A Google search didn't find anything in the first two pages of results
which
exactly applies to the problem and update back story.

The source lists for apt-get were set to "stable" repositories. Since
Debian advanced to v6.x 'Squeeze', stable was resulting in v6 packages
which
wouldn't verify. So I changed 'stable' to 'Lenny' (Debian v5.x) in the
sources list, and the update/upgrade sequence went much better (until
I
rebooted).

I'm not sure if there was a kernel update or not.

The server is running RAID and possibly LVM. I seldom use either so
I'm not
well versed on either. Also, Debian uses Grub (possibly several
versions of
Grub). Grub runs on my Ubuntu 10.04 LTS netbook install, but I never
touch
it.

Any ideas?

Tom --
** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish
for it
to be in the public domain. Email addresses are removed from the
archive but
other items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at
//www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list
website at //www.freelists.org/webpage/frgeek-michiana


--

--
Richard "Goose" Zimmerman, ke4rit
Mishawaka, IN



** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it
to be in the public domain. Email addresses are removed from the archive but
other items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at
//www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list
website at //www.freelists.org/webpage/frgeek-michiana

** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to 
be in the public domain. Email addresses are removed from the archive but other 
items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to 
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list website 
at //www.freelists.org/webpage/frgeek-michiana


** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to 
be in the public domain. Email addresses are removed from the archive but other 
items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to 
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list website 
at //www.freelists.org/webpage/frgeek-michiana

Other related posts: