RE: OCR / VD external vs. normal redundancy using NFS.
- From: D'Hooge Freek <Freek.DHooge@xxxxxxxxx>
- To: David Robillard <david.robillard@xxxxxxxxx>
- Date: Fri, 9 Jul 2010 18:20:49 +0200
David,
DirectNFS can't be used for cluster files (ocr or voting disks).
When using AMS with external redundancy you only have 1 vd and 1 ocr device.
I only have setup such a configuration once. There I used 3 luns to create a
separate diskgroup that only holds my voting disks and ocr devices.
The other diskgroups I created where set to external redundancy.
Note that when you create the diskgroup for the ocr / voting disks during the
rac installation, normal redundancy means 3 way mirror and high redundancy
means 5 way mirroring.
regards,
Freek D'Hooge
Uptime
Oracle Database Administrator
email: freek.dhooge@xxxxxxxxx
tel +32(0)3 451 23 82
http://www.uptime.be
disclaimer: www.uptime.be/disclaimer
________________________________________
From: David Robillard [david.robillard@xxxxxxxxx]
Sent: 09 July 2010 17:26
To: D'Hooge Freek
Cc: oracle-l mailing list; LS Cheng
Subject: Re: OCR / VD external vs. normal redundancy using NFS.
Hello Freek,
On Thu, Jul 8, 2010 at 2:26 AM, D'Hooge Freek <Freek.DHooge@xxxxxxxxx> wrote:
> Yes and no, when using a single ocr device then the cluster would hang
> waiting on an io response instead of failing.
> But I don't think many people would care for the difference when encountering
> such a situation.
Indeed, I totally agree with you.
Do you know if this bug is also present with Oracle Direct NFS ?
> The chance that you run into this bug depends on your storage configuration.
> So, you would best test what happens in you configuration.
Well, you sort of convinced me to swith from NFSv3 to ASM with iSCSI
over GbE. I've watched the webinar "ASM: advantages and disadvantages"
presented by Frits Hoogland [1]. Besides the very important
problematic behavior you thankfully pointed out to me, I like the fact
that ASM can help you migrate from one stoage array to another without
taking the cluster down.
The way I see it, ASM takes the file management "problem" out, but
introduces a new set of complexities in managing LUNs and iSCSI
configurations. NFS is still a lot less complicated to manage.
Anyway, I'll switch to ASM now and see how it goes.
> I like having multiple ocr and voting devices, even when I only have a single
> storage array as it
> protects you against file corruptions and against some "administrator oepses"
> (which are these days more command than hardware failures).
I also agree with you here. However, I plan to use External Redundancy
with ASM. But in "Metalink note 810394.1 - "RAC Assurance Support
Team: RAC and Oracle Clusterware Starter Kit and Best Practices
(Generic)" [2], you found this:
<quote>
For Oracle 11gR2 it is a best practice to store the OCR and Voting
Disk within ASM and to maintain the ASM best practice of having no
more than 2 diskgroups (Flash Recovery Area and Database Area). This
means that the OCR and Voting disk will be stored along with the
database related files. If you are utilizing external redundancy for
your disk groups this means you will have 1 Voting Disk and 1 OCR.
</quote>
How then do you create the OCR mirror and multiple Voting Disks with
ASM using External redundancy?
And do you create a disk group to store only the OCR + voting disks or
you really just drop them into the DATA and FRA diskgroups as stated
above?
Many thanks,
David
[1] http://www.oracleracsig.org/pls/apex/f?p=105:200:3736449467233452::NO:::
[2]
https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx?id=810394.1
> Regards,
>
>
> Freek D'Hooge
> Uptime
> Oracle Database Administrator
> email: freek.dhooge@xxxxxxxxx
> tel +32(0)3 451 23 82
> http://www.uptime.be
> disclaimer: www.uptime.be/disclaimer
> -----Original Message-----
> From: David Robillard [mailto:david.robillard@xxxxxxxxx]
> Sent: maandag 5 juli 2010 22:29
> To: D'Hooge Freek
> Cc: oracle-l mailing list; LS Cheng
> Subject: Re: OCR / VD external vs. normal redundancy using NFS.
>
> On Thu, Jul 1, 2010 at 1:31 AM, D'Hooge Freek <Freek.DHooge@xxxxxxxxx> wrote:
>> I'm usually using the normal redundancy, putting each voting disk / cluster
>> registry in its own volume.
>> You might want to read
>> http://freekdhooge.wordpress.com/2009/11/03/two-oracle-rac-bugs-on-the-wall-two-oracle-bugs-take-one-down/.
>> Because of this "bug" I needed to switch to iscsi for the cluster registry
>> volumes. It is probably still valid with 11gR2
>
> Interesting. Thanks for the heads-up.
>
> I've read your post and the ones from Geert de Paep. What I understand
> is that problem seems to exist when you have several OCR disks. But if
> you choose "External redundancy" instead of "Normal redundancy" when
> you install the Grid Infrastructure software, you end up with a single
> OCR disk. Hence this bug would not happen. Right?
>
> But is using "External redundancy" for OCR a bad idea when you have a
> single storage array?
>
> Because if that single storage array suffers a catastrophic failure,
> you end up loosing not only the OCR disk, but the voting disk and all
> database files as well. The entire cluster would be down even with
> several OCR disks.
>
> So even if I switch to iSCSI, I would still have the same "single
> storage array" problem. Since all LUNs would be stored on that storage
> array.
>
> Unless there are other ways for an OCR disk to become corrupted?
>
> Thanks,
>
> David
>
>> Regards,
>>
>> Freek D'Hooge
>> Uptime
>> Oracle Database Administrator
>> email: freek.dhooge@xxxxxxxxx
>> tel +32(0)3 451 23 82
>> http://www.uptime.be
>> disclaimer: www.uptime.be/disclaimer
>>
>> -----Original Message-----
>> From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx]
>> On Behalf Of David Robillard
>> Sent: woensdag 16 juni 2010 16:44
>> To: oracle-l mailing list
>> Subject: OCR / VD external vs. normal redundancy using NFS.
>>
>> Hello everyone,
>>
>> I'd like to know how does each of us configure redundancy for both the
>> Grid Infrastructure's Oracle Cluster Registry (OCR) and Voting disks
>> (VD) when they're stored over NFSv3 on an enterprise grade storage
>> array with RAID disks. Do you use external or normal redundancy for
>> OCR and VD?
>>
>> I'm looking to install Grid Infrastructure 11gR2 on RedHat Enterprise
>> Linux 5 x86_64 over NFSv3 on a clustered Sun/Oracle Unified Storage
>> 7410. The storage array has built-in redundancy on every components.
>> Each cluster node has two quad-ethernet network interface cards (NIC).
>> All network interfaces are built with two ports, one on each NIC, to
>> create a private (bond0), public (bond1) and storage (bond2)
>> interface. There are two redundant GbE switches dedicated for a
>> storage-only non-routable subnet.
>>
>> Quick note, I'm not using ASM over iSCSI for various reasons. Mostly
>> because every paper I read from NetApp/Sun/Oracle/EMC says the
>> performance of ASM with software initiator iSCSI is not as good as
>> NFS. Also because it's easier to manage the storage array's total disk
>> space when using NFS instead of iSCSI. I'm also not using Oracle's
>> dNFS feature simply because I haven't had the time to look at it and
>> I've been working with NFS for over 10 years. Plus I'm also lucky (?)
>> enough to be the UNIX sysadmin, the storage array administrator and
>> the DBA, so I don't need to configure everything from the Oracle
>> stand-point (i.e. I can't do finger pointing in case things go wrong,
>> I'm the only one to blame).
>>
>> With that in mind, which option would you choose and why?
>>
>> Option A)
>>
>> Create an NFS share for OCR and another one for VD then use external
>> redundancy. That would generate the following mount points:
>>
>> OCR = /u01/ocr/cluster.registry
>> VD = /u01/vd/voting.disk
>>
>> -or-
>>
>> Option B)
>>
>> Create three different NFS shares for OCR and three other shares for
>> VD then use normal redundancy. That would create the following mount
>> points:
>>
>> OCR 1 = /u01/ocr/ocr1/cluster.registry
>> OCR 2 = /u02/ocr/ocr2/cluster.registry
>> OCR 3 = /u03/ocr/ocr3/cluster.registry
>> VD 1 = /u01/vd/vd1/voting.disk
>> VD 2 = /u02/vd/vd2/voting.disk
>> VD 3 = /u03/vd/vd3/voting.disk
>>
>> Of course there are other options and variations. I welcome all
>> comments and critics on that setup :)
>>
>> Many thanks,
>>
>> David
>> --
>> David Robillard, UNIX team leader and Oracle DBA
>> CISSP, RHCE, SCSA & SCSECA
>> --
>> //www.freelists.org/webpage/oracle-l
>>
>>
>>
>
--
//www.freelists.org/webpage/oracle-l
Other related posts:
