Re: High Availability Options
- From: David Robillard <david.robillard@xxxxxxxxx>
- To: Yury Velikanov <j.velikanovs@xxxxxxxxx>, Rui Amaral <Rui.Amaral@xxxxxxxxxxxxxxxx>, Jeremy Schneider <jeremy.schneider@xxxxxxxxxxxxxx>, Michael Dinh <mdinh@xxxxxxxxx>, John Thompson <jhthomp@xxxxxxxxx>
- Date: Thu, 28 Oct 2010 16:43:23 -0400
>> Jeremy wrote:
> ...I was saying that I'd wager Oracle will include asmlib in their particular
> distribution of the linux kernel.
That's an interesting idea. It would remove the version mismatch
problem with ASMLib. In fact, if we push this idea a little further, I
wouldn't be too suprised if Oracle forced us to use their own kernel
in the next database version... Time will tell. It will be interesting
to see how the OEL vs Solaris story unfolds as well.
> I was involved in the engineering effort at a very large org and we decided
> to use udev.
> (In fact, I was one of the main proponents of this approach.) It has in fact
> fallen by the wayside since I left,
> because noone else can figure out how to write those cryptic udev rules.
> One of the major problems was that for every new device that came along, we
> had to figure out new udev rules to detect this device.
> (New HBA vendor, new multipath driver, new SSD card, etc.) As long as I was
> around to reverse engineer what kinds of kernel
> rules we needed to write, it was fine. But noone else could really figure it
> out.
I guess that brings us back to the documentation issue. If your site
uses udev, then one needs to document how to configure it and how to
add a new LUN. On the other end, if your site uses ASMLib, then one
has to write proper procedures to upgrade the kernel with ASMLib at
the same time. I personally think there's less risk in writing proper
documentation that explains how to add a new LUN then to risk having
no ASM disks after a kernel upgrade, but that's just me.
> Personally, I love the flexibility and control of udev. But I really do
> think it's a bear to learn - you pretty much have to understand
> basic linux kernel internals to get it... there are some people out there
> who can do that, but not everyone can.
> I think that (for better or worse) pretty much anyone can use Asmlib.
Quite true.
So we see that udev is easy to keep up to date (since it's part of the
OS) and it's flexible. But it's more complex to setup and it's harder
to add a LUN to ASM. While ASMLib is easier to install and use. But
since it's not part of the OS, it's more error prone when you need to
update the kernel. And both of them must be properly documented.
Then I guess one has to check which one comes more often: a kernel
update or a new ASM LUN? If your site's storage needs grow very fast,
then you're going to be adding a LUN more often then you're going to
upgade the kernel. That means ASMLib would be better suited then udev.
On the contrary, if you don't add a new LUN more often then you update
the kernel, then maybe udev would be a better fit.
One has to check the cluster size too. On small clusters (i.e. two to
four nodes), it's easy to go around all nodes and update both ASMLib
and the kernel. But on large clusters, I'd think that udev is easier
to maintain, since all you have to do once the rule is configured is
to send a copy of the rules to all cluster nodes. Using cfengine [1]
or Puppet [2] makes that task very easy. I'd be curious to know what
do the administrators of large clusters use? udev or ASMLib? Do they
use cfengine or Puppet or an equivalent?
[1] http://www.cfengine.org/
[2] http://www.puppetlabs.com/puppet/introduction/
Some of us don't use neither ASMLib nor udev. Indeed, that's what both
Yury Velikanov and Rui Amaral wrote (see below). It's an interesting
approach.
>> Yury wrote:
> +1 IMHO: both udev or ASM Lib just useless. Why we should introduse
> any additional layers when we can avoid those.
> Keep it simple. Use devices directly (ASM will read headers for you,
> just set the discovery string right) + rc.local will fix privileges
> esely.
>> Rui wrote:
> Personally, I would suggest staying away from udev.
>
> My reason for saying this - udev has issues with multipath devices. I spent a
> couple of weeks some
> 18 months ago trying to get this implemented and failed. Digging through
> various linux forums dug up
> a bug with udev and mpio (maybe that's why the rawdevices was still available
> in rhel5 but that's just a guess).
>
> In terms of asmlib - I too would stay away from it. I used it in many places
> until I had my problem
> (which I mentioned in a previous thread). My asm outage was using asmlib. I
> had asm on raw and none
> of those had problems (all large installations) beyond silly stuff like
> packet collisions on shared i/o ports
> on the san and such. I have had issues with asmlib in other places all
> relating to asm header corruption.
> None on any of my raw devices based asm installs.
>
> Just my 2 cents
But I wonder, how do you manage block device persistent naming then?
In other words, how can you make sure that a LUN is always mapped to
the same ASM disk? Unless you configure persistent naming, both the FC
and iSCSI stacks change LUN mapping between reboots. LUN 1 can be
mapped to /dev/sdd after one reboot and be mapped to /dev/sdg after
another.
I should probably RTFM here, but I'll ask anyway :) If you set the ASM
discovery string to /dev/sd, then ASM will read the LUN header and
know in which ASM diskgroup to place each LUN? Will it know not to use
disks which have no ASM headers (such as the OS disks for example)?
Many thanks,
David
--
http://ca.linkedin.com/in/davidrobillard
--
//www.freelists.org/webpage/oracle-l
Other related posts:
