[frgeek-michiana] Re: Server update gone bad

  • From: Tom Brown <tbrown@xxxxxxxxxxxx>
  • To: frgeek-michiana@xxxxxxxxxxxxx
  • Date: Fri, 06 Jan 2012 08:29:38 -0500

Phil is right. The PATA drives are connected to the main board IDE channels.


There are two expansion cards in the PCI daughter board (riser): an Ethernet card and a two-channel IDE card. Channel 1 of the IDE expansion card is connected to the DVD drive.

Phil and I wanted to boot from the original DVD drive to inspect and possibly edit the Grub menu list. The BIOS was not reporting the original optical drive so we replaced it. The BIOS continues to report the FD and the two PATA HDs, both connected to the main board, but still does not report the optical drive.

Last night I reseated the PCI riser bus, the IDE expansion card and the data and power cables to the optical drive. The BIOS still wouldn't report it although the LED in the optical drive lights up during bootstrap. Oddly, the Linux kernel detects and reports the optical drive. But, because the BIOS doesn't "see" the optical drive, it is not in the Boot Order menu, and the BIOS won't boot from it.

My options seem to be:

1. Replace the IDE/optical drive data cable.
2. Move the IDE/optical drive to channel 2 on the PCI expansion card.
3. Replace the DVD optical drive with a simpler CDROM drive.
4. Connect the optical drive to the slave data connector on one of the onboard PATA HD cables.
5. Replace the PCI IDE expansion card.

Tom --

Mike Cook wrote:
It sounds like the raid card isn't being used for raid which would seem to 
indicate that either no raid is being used for the hard drives or that the 
server is running software raid and using the raid card for additional ide 
channel(s). I have seen. some ide raid cards (Promise) that use the onboard ide 
channels  though. I would proceed carefully if we wish to salvage data.


-----Original Message-----
From: Imre <imreb321@xxxxxxxxx>
Sent: Jan 5, 2012 12:06 PM
To: frgeek-michiana@xxxxxxxxxxxxx
Subject: [frgeek-michiana] Re: Server update gone bad

Then it seems to me that's the problem. The motherboard bios is not
seeing the optical drive, therefore it cannot boot from it. The optical
is probably being recognized by the raid card bios only. But the system
will not be able to boot from the optical. You have to connect the
optical to the onboard ide.

On Thu, 2012-01-05 at 16:47 -0500, Phil wrote:
The hard drives are connected to the motherboard directly, one each on the IDE channels. The optical is connected the RAID (PCI IDE) card.
Sent via Android, G2

Richard Zimmerman <ke4rit@xxxxxxxxx> wrote:

I looked at the raid setup on the colo server (/proc/mdstat says raid
arrays fine :) )

here's what it reads out on cat /boot/grub/menu.lst

<readout>
## ## End Default Options ##

title           Debian GNU/Linux, kernel 2.6.26-2-openvz-686
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-2-openvz-686 root=/dev/md0 ro vga=791
initrd          /boot/initrd.img-2.6.26-2-openvz-686

title           Debian GNU/Linux, kernel 2.6.26-2-openvz-686 (single-user mode)
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-2-openvz-686 root=/dev/md0 ro single
initrd          /boot/initrd.img-2.6.26-2-openvz-686

title           Debian GNU/Linux, kernel 2.6.26-2-686
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-2-686 root=/dev/md0 ro vga=791
initrd          /boot/initrd.img-2.6.26-2-686

title           Debian GNU/Linux, kernel 2.6.26-2-686 (single-user mode)
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-2-686 root=/dev/md0 ro single
initrd          /boot/initrd.img-2.6.26-2-686

title           Debian GNU/Linux, kernel 2.6.26-1-openvz-686
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-1-openvz-686 root=/dev/md0 ro vga=791
initrd          /boot/initrd.img-2.6.26-1-openvz-686

title           Debian GNU/Linux, kernel 2.6.26-1-openvz-686 (single-user mode)
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-1-openvz-686 root=/dev/md0 ro single
initrd          /boot/initrd.img-2.6.26-1-openvz-686

title           Debian GNU/Linux, kernel 2.6.26-1-686
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-1-686 root=/dev/md0 ro vga=791
initrd          /boot/initrd.img-2.6.26-1-686

title           Debian GNU/Linux, kernel 2.6.26-1-686 (single-user mode)
root            (hd0,5)
kernel          /boot/vmlinuz-2.6.26-1-686 root=/dev/md0 ro single
initrd          /boot/initrd.img-2.6.26-1-686

### END DEBIAN AUTOMAGIC KERNELS LIST
</readout>

colo mounts root=/dev/md0

The error "mount: mounting /dev/hda2 on /root failed: Device or
resource busy" would make sense as it's attached to the array md0...

If you can get onto the array.... mount /dev/md0 look at the menu.lst file...

/dev/md1 should be the swap file IIRC...

Richard




On Thu, Jan 5, 2012 at 11:17 AM, Tom Brown <tbrown@xxxxxxxxxxxx> wrote:
I tried to boot the server off an Ultimate Boot CD (UBCD) Phil carries in
his kit. The goal was to find, copy and edit the GRUB conf file. btw, the
contents of the Grub conf file are displayed on the console for about 1-2
seconds, too fast for me to pause the console output.

The server wouldn't boot off the optical drive even though we set it as the
first boot device. The BIOS continued to boot off the hard drive. Phil and I
disconnected and reconnected cables with no improvement. Assuming the BIOS
doesn't see the optical drive or the drive is broken, we installed a DVD ROM
from inventory. We must have disconnected the floppy drive because the
server won't boot and throws the type of error you'd expect from a bad or
disconnected FD. We ran out of time and will pick it up again tonight,
starting with reconnecting the FD.

Here is what I transcribed from the end of the console on booting. It
appears the BIOS is not using UUID. The init script finds hda (hda1, hda2)
and hdc (hdc1, hdc2) so I’ll quote the init output from there.

<boot log snippet>

Begin: Mounting root files system ...
Begin: Running /scripts/local-top ...
Begin: Loading md: raid1 personality registered for level 1
Success: loaded module raid1.
done.
Begin: Assembling all MD arrays ...
md: md0 stopped.
md: bind<hdc2>
md: bind<hda2>
raid1: raid set md0 active with 2 out of 2 mirrors
mdadm: /dev/md0 has been started with 2 drives.
Success: assembled all arrays.
done.
Begin: waiting for udev to process events ... done.
done.
Begin: Running /scripts/local-premount ... kinit: name_to_dev_t(/dev/hda1) =
hda1(3,1)
kinit: trying to resume from /dev/hda1
PM: Starting manual resume from disk
kinit: No resume image, doing normal boot...
resume: libgcrypt version: 1.4.1
done.
mount: mounting /dev/hda2 on /root failed: Device or resource busy
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev
failed: No such file or directory
done.
mount: mounting /sys on /root/sys failed: No such file or direcotry
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn’t have /sbin/init.
No init found. Try passing init= bootarg.

BusyBox v1.10.2 (Debian 1:1.10.2-2) built-in shell (ash).
Enter 'help' for a list of commands.

/bin/sh: can’t access tty; job control turned off
(initramfs) _

</boot log snippet>

Tom --


Imre B wrote:
# blkid
will show disks with their UUID

On Fri, Dec 30, 2011 at 1:45 PM, Tony Germano <tony_germano@xxxxxxxxxxx>
wrote:

Is the UUID used anywhere besides fstab and maybe the boot loader? It
makes
sense to use it there when you have multiple drives because it guarantees
that your mounts will all work even if boot order changes for some
reason.

Each file system is assigned a UUID upon creation. You can use the basic
tools for that file system to find out what the UUID is. It will never
change, even if you move the drive to a different server. It will also
not
change if you clone the partition.

If you are able to boot from a cd, I think you can use tune2fs -l
/dev/<hdxx> to see the UUID for the file system on a given partition
(assuming it is ext2/3/4.)

This page also lists some other ways (I haven't tried them) to get the
UUID
http://liquidat.wordpress.com/2007/10/15/short-tip-get-uuid-of-hard-disks/
The ls -l /dev/disk/by-uuid way looks pretty cool. I didn't know that
directory was there.

Tony



Date: Thu, 29 Dec 2011 17:46:07 -0500
From: tbrown@xxxxxxxxxxxx
To: frgeek-michiana@xxxxxxxxxxxxx
Subject: [frgeek-michiana] Re: Server update gone bad

Sorry to reply so late. I've been very busy over the holiday and out of
town for a few days.

I ran 'apt-get update' using the default 'stable' sources, but that
listed a lot of unverified packages which turned out to be Squeeze
packages so I never started the upgrade. After some research I realized
I needed to stay at Lenny by changing the sources from 'stable' to
'Lenny' as stable now means 'squeeze'. That seemed to work as the number
of unverified packages were very few.

I don't know UUID. I'll research it at the workshop tonight.

afaik, there is no data backup. I'll try booting with a rescue disk and
poke around. As a last resort, I will pull the drives and try to read
them on another system. I don't know yet whether the drives are SCSI,
SATA, PATA or some combination. I believe the drives were in some sort
of RAID config.

Tom --

Richard Zimmerman wrote:

Reminds me since we are on the subject...

Colo server IS LOCKED to "Lenny" and I have every intention of leaving
it there....

If a new server admin takes over, his or her call then...lists

BTW, also speaking from experience after I toasted my home server on a
blind 'dist-upgrade' option.... (from the Lenny to Squeeze upgrade)

***********************************
NEVER EVER USE IT AGAIN
***********************************

That BTDT thing I mentioned in the previous email also happened on my
home server but I got lucky and recovered it....

Now days I **ONLY** use the 'safe-upgrade' option...

Lessons learned the hard way about 8 months ago...

Goose

/ Where is the old RedHat 5.2 when you want it? It was simple and 'just
worked'
// Debian Lenny is the same way. Functional and 'just works'
/// Why keep screwing with it...???


On Sat, Dec 24, 2011 at 7:14 PM, Richard Zimmerman <ke4rit@xxxxxxxxx>
wrote:


If I'm reading it correctly. You pretty well screwed the server....

The new Debian forces everything to that NOT SO wonderful UUID Crap
for everything. If you then rolled back to Lenny, congrats, you own a
toasted server....

BTDT, you pretty screwed... That happened to me.... Happen to have a
backup of the server?

Goose

/ Happened on a server at work. FORTUNATELY, I had grab some dead
hardware out of the back room and had used it so it wasn't a
production server I toasted...

// Left all work servers @ Lenny and have been researching finding a
distro that DOES NOT use UUID. That crap is for the birds....

/// Maybe force it UUID in Debian for the win!

//// If I can't find a distro that doesn't use UUID, guess I'll be
retiring as a server admin....




On Sat, Dec 24, 2011 at 12:41 PM, Tom Brown <tbrown@xxxxxxxxxxxx>
wrote:


Mike started a hard drive wipe too late to complete by closing time
last
Thursday. I started a warehouse server update which also didn't
complete
before we departed so Mike and I left the systems running with the
intention
I would stop by the warehouse today and turn off the computers.

Mike's DBAN wipe completed so I turned off the PC. The server update
completed so I rebooted to test the update.

When the server first tried to boot it died trying to load drivers
for
hdc
or the next item after that; I couldn't tell which. I turned the
server off,
booted and got the following errors.

----------

Running /scripts/inti-bottom ... mount: mounting /dev on /root/dev
failed:
No such file or directory
done
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target filesystem doesn't have /sbin/init.
No init found. Try passing init=bootarg.

BusyBox v1.10.2 (Debian 1:1.10.2-2) built-in shell (ash).
Enter 'help' for a list of commands.

/bin/sh can't access tty; job control turned off
(initramfs) __

----------

A Google search didn't find anything in the first two pages of
results
which
exactly applies to the problem and update back story.

The source lists for apt-get were set to "stable" repositories. Since
Debian advanced to v6.x 'Squeeze', stable was resulting in v6
packages
which
wouldn't verify. So I changed 'stable' to 'Lenny' (Debian v5.x) in
the
sources list, and the update/upgrade sequence went much better (until
I
rebooted).

I'm not sure if there was a kernel update or not.

The server is running RAID and possibly LVM. I seldom use either so
I'm not
well versed on either. Also, Debian uses Grub (possibly several
versions of
Grub). Grub runs on my Ubuntu 10.04 LTS netbook install, but I never
touch
it.

Any ideas?

Tom --
** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish
for it
to be in the public domain. Email addresses are removed from the
archive but
other items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at
//www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list
website at //www.freelists.org/webpage/frgeek-michiana



--

--
Richard "Goose" Zimmerman, ke4rit
Mishawaka, IN




** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for
it
to be in the public domain. Email addresses are removed from the archive
but
other items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at
//www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list
website at //www.freelists.org/webpage/frgeek-michiana


** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it
to be in the public domain. Email addresses are removed from the archive but
other items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at
//www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list
website at //www.freelists.org/webpage/frgeek-michiana



** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it
to be in the public domain. Email addresses are removed from the archive but
other items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at
//www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list
website at //www.freelists.org/webpage/frgeek-michiana


--

--
Richard "Goose" Zimmerman, ke4rit
Mishawaka, IN
** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to 
be in the public domain. Email addresses are removed from the archive but other 
items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to 
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list website 
at //www.freelists.org/webpage/frgeek-michiana

N+--ryv-vȭ+b{,ʋ_ފ{az'fIVvޚu^j!-鬊w%ם{ayʇj)pY[zht+"^ןkyzC"r(m
y&bjv見-azXǧu隊[h~zIryb(y܆+ފƯj)ZnWmyb(ڭb?~zIrآ칻&ޢ!jxʋ+-mx,o((x-^+-ךmyb(yZ߮h&
** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to 
be in the public domain. Email addresses are removed from the archive but other 
items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to 
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list website 
at //www.freelists.org/webpage/frgeek-michiana


** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to 
be in the public domain. Email addresses are removed from the archive but other 
items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to 
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list website 
at //www.freelists.org/webpage/frgeek-michiana


** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to 
be in the public domain. Email addresses are removed from the archive but other 
items included in the body of the email will be shown.

To visit the main website for Free Geek Michiana go to 
http://www.freegeekmichiana.org

To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/

You may unsubscribe or change your list settings by going to the list website 
at //www.freelists.org/webpage/frgeek-michiana

Other related posts: