RE: OCR / VD external vs. normal redundancy using NFS.

David,

See following oracle documentation part: 
http://download.oracle.com/docs/cd/E11882_01/install.112/e10812/storage.htm#sthref536

When using normal redundancy, Oracle will create 3 voting disks and 1 ocr 
device, which will be mirrored.
When using high redundancy, Oracle will create 5 voting disks and 1 ocr device, 
which will be mirrored 2 times.

The output below comes from a setup where I have a separate normal redundancy 
diskgroup to hold my cluster files. I needed 3 different luns in this diskgroup 
before I could create it.

Also note that you can't see the voting disks as "files" within the diskgroup.
My guess is that Oracle uses a header within each asm disk to store the voting 
disk.

[grid@arsvorats1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7855e73b3d2a4fb3bf901beecad62093 (/dev/oracle/asm_clfilests1p1) 
[DG_CLUSTER]
 2. ONLINE   213259f135934f27bf86b91559a44df5 (/dev/oracle/asm_clfilests2p1) 
[DG_CLUSTER]
 3. ONLINE   1b12d9dc6d2b4fb8bfe63b365e2db555 (/dev/oracle/asm_clfilests3p1) 
[DG_CLUSTER]
Located 3 voting disk(s).


[grid@arsvorats1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2800
         Available space (kbytes) :     259320
         ID                       : 1074808153
         Device/File Name         : +DG_CLUSTER
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user


[grid@arsvorats1 ~]$ export ORACLE_SID=+ASM1
[grid@arsvorats1 ~]$ asmcmd
ASMCMD> ls
DG_CLUSTER/
DG_DWHTS_DATA/
DG_FRA/
ASMCMD> cd DG_CLUSTER
ASMCMD> ls
arclorats/
ASMCMD> cd arclorats
ASMCMD> ls
ASMPARAMETERFILE/
OCRFILE/
ASMCMD> cd OCRFILE
ASMCMD> ls -l
Type     Redund  Striped  Time             Sys  Name
OCRFILE  MIRROR  COARSE   JUL 10 00:00:00  Y    REGISTRY.255.721844181


SQL> set linesize 120
SQL> column full_alias_path format a70
SQL> column file_type format a15
SQL> select concat('+'||gname, sys_connect_by_path(aname, '/')) full_alias_path,
  2         system_created, alias_directory, file_type
  3  from ( select b.name gname, a.parent_index pindex, a.name aname,
  4                a.reference_index rindex , a.system_created, 
a.alias_directory,
  5                c.type file_type
  6         from v$asm_alias a, v$asm_diskgroup b, v$asm_file c
  7         where a.group_number = b.group_number
  8               and a.group_number = c.group_number(+)
  9               and a.file_number = c.file_number(+)
 10               and a.file_incarnation = c.incarnation(+)
 11       )
 12  where alias_directory = 'N'
 13        and system_created = 'Y'
 14        and file_type like 'OCR%'
 15  start with (mod(pindex, power(2, 24))) = 0
 16              and rindex in
 17                  ( select a.reference_index
 18                    from v$asm_alias a, v$asm_diskgroup b
 19                    where a.group_number = b.group_number
 20                          and (mod(a.parent_index, power(2, 24))) = 0
 21                  )
 22  connect by prior rindex = pindex;

FULL_ALIAS_PATH                                                        S A 
FILE_TYPE
---------------------------------------------------------------------- - - 
---------------
+DG_CLUSTER/arclorats/OCRFILE/REGISTRY.255.721844181                   Y N 
OCRFILE

 
Freek D'Hooge
Uptime
Oracle Database Administrator
email: freek.dhooge@xxxxxxxxx
tel +32(0)3 451 23 82
http://www.uptime.be
disclaimer: www.uptime.be/disclaimer
-----Original Message-----
From: David Robillard [mailto:david.robillard@xxxxxxxxx] 
Sent: vrijdag 9 juli 2010 18:36
To: D'Hooge Freek
Cc: oracle-l mailing list; LS Cheng
Subject: Re: OCR / VD external vs. normal redundancy using NFS.

Hi Freek,

On Fri, Jul 9, 2010 at 12:20 PM, D'Hooge Freek <Freek.DHooge@xxxxxxxxx> wrote:
> DirectNFS can't be used for cluster files (ocr or voting disks).
> When using AMS with external redundancy you only have 1 vd and 1 ocr device.
> I only have setup such a configuration once. There I used 3 luns to create a 
> separate diskgroup that only holds my voting disks and ocr devices.
> The other diskgroups I created where set to external redundancy.

Well, I found the answer to my question...

> "How then do you create the OCR mirror and multiple Voting Disks with ASM 
> using External redundancy?"

...by RTFM. Chapter 2 of "Oracle Clusterware Administration and
Deployment Guide 11g Release 2 (11.2)" states that:

<quote>
If OCR is stored in an Oracle ASM disk group with external redundancy,
then Oracle recommends that you add another OCR location to another
disk group to avoid the loss of OCR, if a disk fails in the disk
group.
</quote>

There you have it! I think I'll create a very small ASM disk group
called +GRID in which I'll store one OCR disk and one Voting Disk.
I'll use the +DATA disk group to store another Voting Disk and the
+FRA disk group will have the second OCR disk and third Voting Disk.

I guess that should have me covered.


> Note that when you create the diskgroup for the ocr / voting disks during the 
> rac installation, normal redundancy means 3 way mirror and high redundancy 
> means 5 way mirroring.

Ah, not exactly :) If you read chapter 1 of "Oracle Database Storage
Administrator's Guide 11g Release 2 (11.2)" [2], you see:

<quote>
    *      Normal for 2-way mirroring
    *      High for 3-way mirroring
    *      External to not use Oracle ASM mirroring, such as when you
configure hardware RAID for redundancy
</quote>

Many thanks for your help, I appreciated it.

David

[1] 
http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/votocr.htm#CHDJIIJH

[2] 
http://download.oracle.com/docs/cd/E11882_01/server.112/e10500/asmcon.htm#BABGDGAA



> regards,
>
> Freek D'Hooge
> Uptime
> Oracle Database Administrator
> email: freek.dhooge@xxxxxxxxx
> tel +32(0)3 451 23 82
> http://www.uptime.be
> disclaimer: www.uptime.be/disclaimer
> ________________________________________
> From: David Robillard [david.robillard@xxxxxxxxx]
> Sent: 09 July 2010 17:26
> To: D'Hooge Freek
> Cc: oracle-l mailing list; LS Cheng
> Subject: Re: OCR / VD external vs. normal redundancy using NFS.
>
> Hello Freek,
>
> On Thu, Jul 8, 2010 at 2:26 AM, D'Hooge Freek <Freek.DHooge@xxxxxxxxx> wrote:
>> Yes and no, when using a single ocr device then the cluster would hang 
>> waiting on an io response instead of failing.
>> But I don't think many people would care for the difference when 
>> encountering such a situation.
>
> Indeed, I totally agree with you.
>
> Do you know if this bug is also present with Oracle Direct NFS ?
>
>> The chance that you run into this bug depends on your storage configuration.
>> So, you would best test what happens in you configuration.
>
> Well, you sort of convinced me to swith from NFSv3 to ASM with iSCSI
> over GbE. I've watched the webinar "ASM: advantages and disadvantages"
> presented by Frits Hoogland [1]. Besides the very important
> problematic behavior you thankfully pointed out to me, I like the fact
> that ASM can help you migrate from one stoage array to another without
> taking the cluster down.
>
> The way I see it, ASM takes the file management "problem" out, but
> introduces a new set of complexities in managing LUNs and iSCSI
> configurations. NFS is still a lot less complicated to manage.
>
> Anyway, I'll switch to ASM now and see how it goes.
>
>> I like having multiple ocr and voting devices, even when I only have a 
>> single storage array as it
>> protects you against file corruptions and against some "administrator oepses"
>> (which are these days more command than hardware failures).
>
> I also agree with you here. However, I plan to use External Redundancy
> with ASM. But in "Metalink note 810394.1 - "RAC Assurance Support
> Team: RAC and Oracle Clusterware Starter Kit and Best Practices
> (Generic)" [2], you found this:
>
> <quote>
> For Oracle 11gR2 it is a best practice to store the OCR and Voting
> Disk within ASM and to maintain the ASM best practice of having no
> more than 2 diskgroups (Flash Recovery Area and Database Area). This
> means that the OCR and Voting disk will be stored along with the
> database related files. If you are utilizing external redundancy for
> your disk groups this means you will have 1 Voting Disk and 1 OCR.
> </quote>
>
> How then do you create the OCR mirror and multiple Voting Disks with
> ASM using External redundancy?
>
> And do you create a disk group to store only the OCR + voting disks or
> you really just drop them into the DATA and FRA diskgroups as stated
> above?
>
> Many thanks,
>
> David
>
> [1] http://www.oracleracsig.org/pls/apex/f?p=105:200:3736449467233452::NO:::
>
> [2] 
> https://supporthtml.oracle.com/ep/faces/secure/km/DocumentDisplay.jspx?id=810394.1
>
>
>> Regards,
>>
>>
>> Freek D'Hooge
>> Uptime
>> Oracle Database Administrator
>> email: freek.dhooge@xxxxxxxxx
>> tel +32(0)3 451 23 82
>> http://www.uptime.be
>> disclaimer: www.uptime.be/disclaimer
>> -----Original Message-----
>> From: David Robillard [mailto:david.robillard@xxxxxxxxx]
>> Sent: maandag 5 juli 2010 22:29
>> To: D'Hooge Freek
>> Cc: oracle-l mailing list; LS Cheng
>> Subject: Re: OCR / VD external vs. normal redundancy using NFS.
>>
>> On Thu, Jul 1, 2010 at 1:31 AM, D'Hooge Freek <Freek.DHooge@xxxxxxxxx> wrote:
>>> I'm usually using the normal redundancy, putting each voting disk / cluster 
>>> registry in its own volume.
>>> You might want to read 
>>> http://freekdhooge.wordpress.com/2009/11/03/two-oracle-rac-bugs-on-the-wall-two-oracle-bugs-take-one-down/.
>>>  Because of this "bug" I needed to switch to iscsi for the cluster registry 
>>> volumes. It is probably still valid with 11gR2
>>
>> Interesting. Thanks for the heads-up.
>>
>> I've read your post and the ones from Geert de Paep. What I understand
>> is that problem seems to exist when you have several OCR disks. But if
>> you choose "External redundancy" instead of "Normal redundancy" when
>> you install the Grid Infrastructure software, you end up with a single
>> OCR disk. Hence this bug would not happen. Right?
>>
>> But is using "External redundancy" for OCR a bad idea when you have a
>> single storage array?
>>
>> Because if that single storage array suffers a catastrophic failure,
>> you end up loosing not only the OCR disk, but the voting disk and all
>> database files as well. The entire cluster would be down even with
>> several OCR disks.
>>
>> So even if I switch to iSCSI, I would still have the same "single
>> storage array" problem. Since all LUNs would be stored on that storage
>> array.
>>
>> Unless there are other ways for an OCR disk to become corrupted?
>>
>> Thanks,
>>
>> David
>>
>>> Regards,
>>>
>>> Freek D'Hooge
>>> Uptime
>>> Oracle Database Administrator
>>> email: freek.dhooge@xxxxxxxxx
>>> tel +32(0)3 451 23 82
>>> http://www.uptime.be
>>> disclaimer: www.uptime.be/disclaimer
>>>
>>> -----Original Message-----
>>> From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] 
>>> On Behalf Of David Robillard
>>> Sent: woensdag 16 juni 2010 16:44
>>> To: oracle-l mailing list
>>> Subject: OCR / VD external vs. normal redundancy using NFS.
>>>
>>> Hello everyone,
>>>
>>> I'd like to know how does each of us configure redundancy for both the
>>> Grid Infrastructure's Oracle Cluster Registry (OCR) and Voting disks
>>> (VD) when they're stored over NFSv3 on an enterprise grade storage
>>> array with RAID disks. Do you use external or normal redundancy for
>>> OCR and VD?
>>>
>>> I'm looking to install Grid Infrastructure 11gR2 on RedHat Enterprise
>>> Linux 5 x86_64 over NFSv3 on a clustered Sun/Oracle Unified Storage
>>> 7410. The storage array has built-in redundancy on every components.
>>> Each cluster node has two quad-ethernet network interface cards (NIC).
>>> All network interfaces are built with two ports, one on each NIC, to
>>> create a private (bond0), public (bond1) and storage (bond2)
>>> interface. There are two redundant GbE switches dedicated for a
>>> storage-only non-routable subnet.
>>>
>>> Quick note, I'm not using ASM over iSCSI for various reasons. Mostly
>>> because every paper I read from NetApp/Sun/Oracle/EMC says the
>>> performance of ASM with software initiator iSCSI is not as good as
>>> NFS. Also because it's easier to manage the storage array's total disk
>>> space when using NFS instead of iSCSI. I'm also not using Oracle's
>>> dNFS feature simply because I haven't had the time to look at it and
>>> I've been working with NFS for over 10 years. Plus I'm also lucky (?)
>>> enough to be the UNIX sysadmin, the storage array administrator and
>>> the DBA, so I don't need to configure everything from the Oracle
>>> stand-point (i.e. I can't do finger pointing in case things go wrong,
>>> I'm the only one to blame).
>>>
>>> With that in mind, which option would you choose and why?
>>>
>>> Option A)
>>>
>>> Create an NFS share for OCR and another one for VD then use external
>>> redundancy. That would generate the following mount points:
>>>
>>> OCR = /u01/ocr/cluster.registry
>>> VD = /u01/vd/voting.disk
>>>
>>> -or-
>>>
>>> Option B)
>>>
>>> Create three different NFS shares for OCR and three other shares for
>>> VD then use normal redundancy. That would create the following mount
>>> points:
>>>
>>> OCR 1 = /u01/ocr/ocr1/cluster.registry
>>> OCR 2 = /u02/ocr/ocr2/cluster.registry
>>> OCR 3 = /u03/ocr/ocr3/cluster.registry
>>> VD 1 = /u01/vd/vd1/voting.disk
>>> VD 2 = /u02/vd/vd2/voting.disk
>>> VD 3 = /u03/vd/vd3/voting.disk
>>>
>>> Of course there are other options and variations. I welcome all
>>> comments and critics on that setup :)
>>>
>>> Many thanks,
>>>
>>> David
>>> --
>>> David Robillard, UNIX team leader and Oracle DBA
>>> CISSP, RHCE, SCSA & SCSECA
>>> --
>>> http://www.freelists.org/webpage/oracle-l
>>>
>>>
>>>
>>
>
--
http://www.freelists.org/webpage/oracle-l


Other related posts: