[frgeek-michiana] Re: Server update gone bad
- From: Mike Cook <mikecook@xxxxxxxxxxxx>
- To: frgeek-michiana@xxxxxxxxxxxxx
- Date: Fri, 6 Jan 2012 01:09:41 -0500 (GMT-05:00)
It sounds like the raid card isn't being used for raid which would seem to
indicate that either no raid is being used for the hard drives or that the
server is running software raid and using the raid card for additional ide
channel(s). I have seen. some ide raid cards (Promise) that use the onboard ide
channels though. I would proceed carefully if we wish to salvage data.
-----Original Message-----
>From: Imre <imreb321@xxxxxxxxx>
>Sent: Jan 5, 2012 12:06 PM
>To: frgeek-michiana@xxxxxxxxxxxxx
>Subject: [frgeek-michiana] Re: Server update gone bad
>
>Then it seems to me that's the problem. The motherboard bios is not
>seeing the optical drive, therefore it cannot boot from it. The optical
>is probably being recognized by the raid card bios only. But the system
>will not be able to boot from the optical. You have to connect the
>optical to the onboard ide.
>
> On Thu, 2012-01-05 at 16:47 -0500, Phil wrote:
>> The hard drives are connected to the motherboard directly, one each on the
>> IDE channels. The optical is connected the RAID (PCI IDE) card.
>>
>> Sent via Android, G2
>>
>> Richard Zimmerman <ke4rit@xxxxxxxxx> wrote:
>>
>> >I looked at the raid setup on the colo server (/proc/mdstat says raid
>> >arrays fine :) )
>> >
>> >here's what it reads out on cat /boot/grub/menu.lst
>> >
>> ><readout>
>> >## ## End Default Options ##
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-2-openvz-686
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-2-openvz-686 root=/dev/md0 ro vga=791
>> >initrd /boot/initrd.img-2.6.26-2-openvz-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-2-openvz-686 (single-user
>> >mode)
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-2-openvz-686 root=/dev/md0 ro single
>> >initrd /boot/initrd.img-2.6.26-2-openvz-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-2-686
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-2-686 root=/dev/md0 ro vga=791
>> >initrd /boot/initrd.img-2.6.26-2-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-2-686 (single-user mode)
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-2-686 root=/dev/md0 ro single
>> >initrd /boot/initrd.img-2.6.26-2-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-1-openvz-686
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-1-openvz-686 root=/dev/md0 ro vga=791
>> >initrd /boot/initrd.img-2.6.26-1-openvz-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-1-openvz-686 (single-user
>> >mode)
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-1-openvz-686 root=/dev/md0 ro single
>> >initrd /boot/initrd.img-2.6.26-1-openvz-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-1-686
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-1-686 root=/dev/md0 ro vga=791
>> >initrd /boot/initrd.img-2.6.26-1-686
>> >
>> >title Debian GNU/Linux, kernel 2.6.26-1-686 (single-user mode)
>> >root (hd0,5)
>> >kernel /boot/vmlinuz-2.6.26-1-686 root=/dev/md0 ro single
>> >initrd /boot/initrd.img-2.6.26-1-686
>> >
>> >### END DEBIAN AUTOMAGIC KERNELS LIST
>> ></readout>
>> >
>> >colo mounts root=/dev/md0
>> >
>> >The error "mount: mounting /dev/hda2 on /root failed: Device or
>> >resource busy" would make sense as it's attached to the array md0...
>> >
>> >If you can get onto the array.... mount /dev/md0 look at the menu.lst
>> >file...
>> >
>> >/dev/md1 should be the swap file IIRC...
>> >
>> >Richard
>> >
>> >
>> >
>> >
>> >On Thu, Jan 5, 2012 at 11:17 AM, Tom Brown <tbrown@xxxxxxxxxxxx> wrote:
>> >> I tried to boot the server off an Ultimate Boot CD (UBCD) Phil carries in
>> >> his kit. The goal was to find, copy and edit the GRUB conf file. btw, the
>> >> contents of the Grub conf file are displayed on the console for about 1-2
>> >> seconds, too fast for me to pause the console output.
>> >>
>> >> The server wouldn't boot off the optical drive even though we set it as
>> >> the
>> >> first boot device. The BIOS continued to boot off the hard drive. Phil
>> >> and I
>> >> disconnected and reconnected cables with no improvement. Assuming the BIOS
>> >> doesn't see the optical drive or the drive is broken, we installed a DVD
>> >> ROM
>> >> from inventory. We must have disconnected the floppy drive because the
>> >> server won't boot and throws the type of error you'd expect from a bad or
>> >> disconnected FD. We ran out of time and will pick it up again tonight,
>> >> starting with reconnecting the FD.
>> >>
>> >> Here is what I transcribed from the end of the console on booting. It
>> >> appears the BIOS is not using UUID. The init script finds hda (hda1, hda2)
>> >> and hdc (hdc1, hdc2) so I’ll quote the init output from there.
>> >>
>> >> <boot log snippet>
>> >>
>> >> Begin: Mounting root files system ...
>> >> Begin: Running /scripts/local-top ...
>> >> Begin: Loading md: raid1 personality registered for level 1
>> >> Success: loaded module raid1.
>> >> done.
>> >> Begin: Assembling all MD arrays ...
>> >> md: md0 stopped.
>> >> md: bind<hdc2>
>> >> md: bind<hda2>
>> >> raid1: raid set md0 active with 2 out of 2 mirrors
>> >> mdadm: /dev/md0 has been started with 2 drives.
>> >> Success: assembled all arrays.
>> >> done.
>> >> Begin: waiting for udev to process events ... done.
>> >> done.
>> >> Begin: Running /scripts/local-premount ... kinit:
>> >> name_to_dev_t(/dev/hda1) =
>> >> hda1(3,1)
>> >> kinit: trying to resume from /dev/hda1
>> >> PM: Starting manual resume from disk
>> >> kinit: No resume image, doing normal boot...
>> >> resume: libgcrypt version: 1.4.1
>> >> done.
>> >> mount: mounting /dev/hda2 on /root failed: Device or resource busy
>> >> Begin: Running /scripts/local-bottom ... done.
>> >> done.
>> >> Begin: Running /scripts/init-bottom ... mount: mounting /dev on /root/dev
>> >> failed: No such file or directory
>> >> done.
>> >> mount: mounting /sys on /root/sys failed: No such file or direcotry
>> >> mount: mounting /proc on /root/proc failed: No such file or directory
>> >> Target filesystem doesn’t have /sbin/init.
>> >> No init found. Try passing init= bootarg.
>> >>
>> >> BusyBox v1.10.2 (Debian 1:1.10.2-2) built-in shell (ash).
>> >> Enter 'help' for a list of commands.
>> >>
>> >> /bin/sh: can’t access tty; job control turned off
>> >> (initramfs) _
>> >>
>> >> </boot log snippet>
>> >>
>> >> Tom --
>> >>
>> >>
>> >> Imre B wrote:
>> >>>
>> >>> # blkid
>> >>> will show disks with their UUID
>> >>>
>> >>> On Fri, Dec 30, 2011 at 1:45 PM, Tony Germano <tony_germano@xxxxxxxxxxx>
>> >>> wrote:
>> >>>
>> >>>>
>> >>>> Is the UUID used anywhere besides fstab and maybe the boot loader? It
>> >>>> makes
>> >>>> sense to use it there when you have multiple drives because it
>> >>>> guarantees
>> >>>> that your mounts will all work even if boot order changes for some
>> >>>> reason.
>> >>>>
>> >>>> Each file system is assigned a UUID upon creation. You can use the basic
>> >>>> tools for that file system to find out what the UUID is. It will never
>> >>>> change, even if you move the drive to a different server. It will also
>> >>>> not
>> >>>> change if you clone the partition.
>> >>>>
>> >>>> If you are able to boot from a cd, I think you can use tune2fs -l
>> >>>> /dev/<hdxx> to see the UUID for the file system on a given partition
>> >>>> (assuming it is ext2/3/4.)
>> >>>>
>> >>>> This page also lists some other ways (I haven't tried them) to get the
>> >>>> UUID
>> >>>> http://liquidat.wordpress.com/2007/10/15/short-tip-get-uuid-of-hard-disks/
>> >>>> The ls -l /dev/disk/by-uuid way looks pretty cool. I didn't know that
>> >>>> directory was there.
>> >>>>
>> >>>> Tony
>> >>>>
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> Date: Thu, 29 Dec 2011 17:46:07 -0500
>> >>>>> From: tbrown@xxxxxxxxxxxx
>> >>>>> To: frgeek-michiana@xxxxxxxxxxxxx
>> >>>>> Subject: [frgeek-michiana] Re: Server update gone bad
>> >>>>>
>> >>>>> Sorry to reply so late. I've been very busy over the holiday and out of
>> >>>>> town for a few days.
>> >>>>>
>> >>>>> I ran 'apt-get update' using the default 'stable' sources, but that
>> >>>>> listed a lot of unverified packages which turned out to be Squeeze
>> >>>>> packages so I never started the upgrade. After some research I realized
>> >>>>> I needed to stay at Lenny by changing the sources from 'stable' to
>> >>>>> 'Lenny' as stable now means 'squeeze'. That seemed to work as the
>> >>>>> number
>> >>>>> of unverified packages were very few.
>> >>>>>
>> >>>>> I don't know UUID. I'll research it at the workshop tonight.
>> >>>>>
>> >>>>> afaik, there is no data backup. I'll try booting with a rescue disk and
>> >>>>> poke around. As a last resort, I will pull the drives and try to read
>> >>>>> them on another system. I don't know yet whether the drives are SCSI,
>> >>>>> SATA, PATA or some combination. I believe the drives were in some sort
>> >>>>> of RAID config.
>> >>>>>
>> >>>>> Tom --
>> >>>>>
>> >>>>> Richard Zimmerman wrote:
>> >>>>>
>> >>>>>>
>> >>>>>> Reminds me since we are on the subject...
>> >>>>>>
>> >>>>>> Colo server IS LOCKED to "Lenny" and I have every intention of leaving
>> >>>>>> it there....
>> >>>>>>
>> >>>>>> If a new server admin takes over, his or her call then...lists
>> >>>>>>
>> >>>>>> BTW, also speaking from experience after I toasted my home server on a
>> >>>>>> blind 'dist-upgrade' option.... (from the Lenny to Squeeze upgrade)
>> >>>>>>
>> >>>>>> ***********************************
>> >>>>>> NEVER EVER USE IT AGAIN
>> >>>>>> ***********************************
>> >>>>>>
>> >>>>>> That BTDT thing I mentioned in the previous email also happened on my
>> >>>>>> home server but I got lucky and recovered it....
>> >>>>>>
>> >>>>>> Now days I **ONLY** use the 'safe-upgrade' option...
>> >>>>>>
>> >>>>>> Lessons learned the hard way about 8 months ago...
>> >>>>>>
>> >>>>>> Goose
>> >>>>>>
>> >>>>>> / Where is the old RedHat 5.2 when you want it? It was simple and
>> >>>>>> 'just
>> >>>>>> worked'
>> >>>>>> // Debian Lenny is the same way. Functional and 'just works'
>> >>>>>> /// Why keep screwing with it...???
>> >>>>>>
>> >>>>>>
>> >>>>>> On Sat, Dec 24, 2011 at 7:14 PM, Richard Zimmerman <ke4rit@xxxxxxxxx>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> If I'm reading it correctly. You pretty well screwed the server....
>> >>>>>>>
>> >>>>>>> The new Debian forces everything to that NOT SO wonderful UUID Crap
>> >>>>>>> for everything. If you then rolled back to Lenny, congrats, you own a
>> >>>>>>> toasted server....
>> >>>>>>>
>> >>>>>>> BTDT, you pretty screwed... That happened to me.... Happen to have a
>> >>>>>>> backup of the server?
>> >>>>>>>
>> >>>>>>> Goose
>> >>>>>>>
>> >>>>>>> / Happened on a server at work. FORTUNATELY, I had grab some dead
>> >>>>>>> hardware out of the back room and had used it so it wasn't a
>> >>>>>>> production server I toasted...
>> >>>>>>>
>> >>>>>>> // Left all work servers @ Lenny and have been researching finding a
>> >>>>>>> distro that DOES NOT use UUID. That crap is for the birds....
>> >>>>>>>
>> >>>>>>> /// Maybe force it UUID in Debian for the win!
>> >>>>>>>
>> >>>>>>> //// If I can't find a distro that doesn't use UUID, guess I'll be
>> >>>>>>> retiring as a server admin....
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Sat, Dec 24, 2011 at 12:41 PM, Tom Brown <tbrown@xxxxxxxxxxxx>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>>
>> >>>>>>>> Mike started a hard drive wipe too late to complete by closing time
>> >>>>>>>> last
>> >>>>>>>> Thursday. I started a warehouse server update which also didn't
>> >>>>>>>> complete
>> >>>>>>>> before we departed so Mike and I left the systems running with the
>> >>>>>>>> intention
>> >>>>>>>> I would stop by the warehouse today and turn off the computers.
>> >>>>>>>>
>> >>>>>>>> Mike's DBAN wipe completed so I turned off the PC. The server update
>> >>>>>>>> completed so I rebooted to test the update.
>> >>>>>>>>
>> >>>>>>>> When the server first tried to boot it died trying to load drivers
>> >>>>>>>> for
>> >>>>>>>> hdc
>> >>>>>>>> or the next item after that; I couldn't tell which. I turned the
>> >>>>>>>> server off,
>> >>>>>>>> booted and got the following errors.
>> >>>>>>>>
>> >>>>>>>> ----------
>> >>>>>>>>
>> >>>>>>>> Running /scripts/inti-bottom ... mount: mounting /dev on /root/dev
>> >>>>>>>> failed:
>> >>>>>>>> No such file or directory
>> >>>>>>>> done
>> >>>>>>>> mount: mounting /sys on /root/sys failed: No such file or directory
>> >>>>>>>> mount: mounting /proc on /root/proc failed: No such file or
>> >>>>>>>> directory
>> >>>>>>>> Target filesystem doesn't have /sbin/init.
>> >>>>>>>> No init found. Try passing init=bootarg.
>> >>>>>>>>
>> >>>>>>>> BusyBox v1.10.2 (Debian 1:1.10.2-2) built-in shell (ash).
>> >>>>>>>> Enter 'help' for a list of commands.
>> >>>>>>>>
>> >>>>>>>> /bin/sh can't access tty; job control turned off
>> >>>>>>>> (initramfs) __
>> >>>>>>>>
>> >>>>>>>> ----------
>> >>>>>>>>
>> >>>>>>>> A Google search didn't find anything in the first two pages of
>> >>>>>>>> results
>> >>>>>>>> which
>> >>>>>>>> exactly applies to the problem and update back story.
>> >>>>>>>>
>> >>>>>>>> The source lists for apt-get were set to "stable" repositories.
>> >>>>>>>> Since
>> >>>>>>>> Debian advanced to v6.x 'Squeeze', stable was resulting in v6
>> >>>>>>>> packages
>> >>>>>>>> which
>> >>>>>>>> wouldn't verify. So I changed 'stable' to 'Lenny' (Debian v5.x) in
>> >>>>>>>> the
>> >>>>>>>> sources list, and the update/upgrade sequence went much better
>> >>>>>>>> (until
>> >>>>>>>> I
>> >>>>>>>> rebooted).
>> >>>>>>>>
>> >>>>>>>> I'm not sure if there was a kernel update or not.
>> >>>>>>>>
>> >>>>>>>> The server is running RAID and possibly LVM. I seldom use either so
>> >>>>>>>> I'm not
>> >>>>>>>> well versed on either. Also, Debian uses Grub (possibly several
>> >>>>>>>> versions of
>> >>>>>>>> Grub). Grub runs on my Ubuntu 10.04 LTS netbook install, but I never
>> >>>>>>>> touch
>> >>>>>>>> it.
>> >>>>>>>>
>> >>>>>>>> Any ideas?
>> >>>>>>>>
>> >>>>>>>> Tom --
>> >>>>>>>> ** This list is PUBLICLY archived. **
>> >>>>>>>> PLEASE don't post personal or sensitive information unless you wish
>> >>>>>>>> for it
>> >>>>>>>> to be in the public domain. Email addresses are removed from the
>> >>>>>>>> archive but
>> >>>>>>>> other items included in the body of the email will be shown.
>> >>>>>>>>
>> >>>>>>>> To visit the main website for Free Geek Michiana go to
>> >>>>>>>> http://www.freegeekmichiana.org
>> >>>>>>>>
>> >>>>>>>> To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
>> >>>>>>>> The archive is available at
>> >>>>>>>> //www.freelists.org/archives/frgeek-michiana/
>> >>>>>>>>
>> >>>>>>>> You may unsubscribe or change your list settings by going to the
>> >>>>>>>> list
>> >>>>>>>> website at //www.freelists.org/webpage/frgeek-michiana
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>>
>> >>>>>>> --
>> >>>>>>> Richard "Goose" Zimmerman, ke4rit
>> >>>>>>> Mishawaka, IN
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> ** This list is PUBLICLY archived. **
>> >>>>> PLEASE don't post personal or sensitive information unless you wish for
>> >>>>> it
>> >>>>> to be in the public domain. Email addresses are removed from the
>> >>>>> archive
>> >>>>> but
>> >>>>> other items included in the body of the email will be shown.
>> >>>>>
>> >>>>> To visit the main website for Free Geek Michiana go to
>> >>>>> http://www.freegeekmichiana.org
>> >>>>>
>> >>>>> To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
>> >>>>> The archive is available at
>> >>>>> //www.freelists.org/archives/frgeek-michiana/
>> >>>>>
>> >>>>> You may unsubscribe or change your list settings by going to the list
>> >>>>> website at //www.freelists.org/webpage/frgeek-michiana
>> >>>>>
>> >>>>>
>> >>>
>> >>> ** This list is PUBLICLY archived. **
>> >>> PLEASE don't post personal or sensitive information unless you wish for
>> >>> it
>> >>> to be in the public domain. Email addresses are removed from the archive
>> >>> but
>> >>> other items included in the body of the email will be shown.
>> >>>
>> >>> To visit the main website for Free Geek Michiana go to
>> >>> http://www.freegeekmichiana.org
>> >>>
>> >>> To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
>> >>> The archive is available at
>> >>> //www.freelists.org/archives/frgeek-michiana/
>> >>>
>> >>> You may unsubscribe or change your list settings by going to the list
>> >>> website at //www.freelists.org/webpage/frgeek-michiana
>> >>>
>> >>>
>> >>>
>> >>
>> >> ** This list is PUBLICLY archived. **
>> >> PLEASE don't post personal or sensitive information unless you wish for it
>> >> to be in the public domain. Email addresses are removed from the archive
>> >> but
>> >> other items included in the body of the email will be shown.
>> >>
>> >> To visit the main website for Free Geek Michiana go to
>> >> http://www.freegeekmichiana.org
>> >>
>> >> To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
>> >> The archive is available at
>> >> //www.freelists.org/archives/frgeek-michiana/
>> >>
>> >> You may unsubscribe or change your list settings by going to the list
>> >> website at //www.freelists.org/webpage/frgeek-michiana
>> >>
>> >
>> >
>> >
>> >--
>> >
>> >--
>> >Richard "Goose" Zimmerman, ke4rit
>> >Mishawaka, IN
>> >** This list is PUBLICLY archived. **
>> >PLEASE don't post personal or sensitive information unless you wish for it
>> >to be in the public domain. Email addresses are removed from the archive
>> >but other items included in the body of the email will be shown.
>> >
>> >To visit the main website for Free Geek Michiana go to
>> >http://www.freegeekmichiana.org
>> >
>> >To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
>> >The archive is available at
>> >//www.freelists.org/archives/frgeek-michiana/
>> >
>> >You may unsubscribe or change your list settings by going to the list
>> >website at //www.freelists.org/webpage/frgeek-michiana
>> >
>> N+--ryv-v ȭ+b{,ʋ_ފ{az' fIVvޚu^j!-鬊w%ם{ay ʇj)pY[zht+"^ןkyzC"r(m
>> y&bjv見-azXǧu隊[h~ zIryb(y܆+ފƯj)ZnWmyb(ڭb?~ zIrآ칻 &ޢ!jx ʋ+-mx,o((x-^+-ךmyb(yZ߮h&
>
>
>** This list is PUBLICLY archived. **
>PLEASE don't post personal or sensitive information unless you wish for it to
>be in the public domain. Email addresses are removed from the archive but
>other items included in the body of the email will be shown.
>
>To visit the main website for Free Geek Michiana go to
>http://www.freegeekmichiana.org
>
>To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
>The archive is available at //www.freelists.org/archives/frgeek-michiana/
>
>You may unsubscribe or change your list settings by going to the list website
>at //www.freelists.org/webpage/frgeek-michiana
>
** This list is PUBLICLY archived. **
PLEASE don't post personal or sensitive information unless you wish for it to
be in the public domain. Email addresses are removed from the archive but other
items included in the body of the email will be shown.
To visit the main website for Free Geek Michiana go to
http://www.freegeekmichiana.org
To post to the list send email to frgeek-michiana@xxxxxxxxxxxxxx
The archive is available at //www.freelists.org/archives/frgeek-michiana/
You may unsubscribe or change your list settings by going to the list website
at //www.freelists.org/webpage/frgeek-michiana
Other related posts:
