Martin, may I recommend the following to mull over-my customer had the same problem after he changed the asm_diskstring from '/dev/oracleasm/disks/' to ''. The customer used ASMLib, but hasn't configured it correctly, which has been visible in v$asm_disk.library before the change. BTW This is 11.2.0.2 with voting disks and OCR in ASM: diskgroup "+OCR" (Linux x86-64) Upon restart CSS discovered all voting files only to discard them :( The solution was to - prevent crs from starting automatically (crsctl disable crs) - reboot - start clusterware in -excl mode (this started the ASM instance) - change ASM diskstring to '' - use crsctl replace <DG_NAME> Then reboot again, start stack again and voila-it worked. It doesn't mean that it works in your environment though, please test this first before applying to production etc and run it past Oracle Support. Kind regards, (another) Martin http://martincarstenbach.wordpress.com http://www.linkedin.com/in/martincarstenbach > Martin, > X-archive-position: 37362 > X-ecartis-version: Ecartis v1.0.0 > Sender: oracle-l-bounce@xxxxxxxxxxxxx > Errors-to: oracle-l-bounce@xxxxxxxxxxxxx > X-original-sender: Freek.DHooge@xxxxxxxxx > Precedence: normal > Reply-To: Freek.DHooge@xxxxxxxxx > List-help: <mailto:ecartis@xxxxxxxxxxxxx?Subject=help> > List-unsubscribe: <oracle-l-request@xxxxxxxxxxxxx?Subject=unsubscribe> > List-software: Ecartis version 1.0.0 > List-Id: oracle-l <oracle-l.freelists.org> > X-List-ID: oracle-l <oracle-l.freelists.org> > List-subscribe: <oracle-l-request@xxxxxxxxxxxxx?Subject=subscribe> > List-owner: <mailto:steve.adams@xxxxxxxxxxxx> > List-post: <mailto:oracle-l@xxxxxxxxxxxxx> > List-archive: <//www.freelists.org/archives/oracle-l> > X-list: oracle-l > > Is this 11.2 with voting files in ASM? > If so, following can help (but note this is not supported) > > The asm diskstring is stored in the profile.xmlf file in the grid > infrastructure $ORACLE_HOME/ gpnp/profiles/peer/ directory. > > This file is a signed xml file, but can be unsigned / signed using the > gpnptool > Best is to stop the entire cluster before making modifications to this > file > > Use gpnptool -? to get the syntax on how to unsign /sign the file > > To get a formatted outout: > > [grid@beantorarac03 peer]$ gpnptool c14n -fmt -fmtonly > Warning: some command line parameters were defaulted. Resulting command > line: > /opt/grid/11.2.0.2/bin/gpnptool.bin c14n -fmt=0,2 -fmtonly > -p=profile.xml -o- > > <?xml version="1.0" encoding="UTF-8"?> > <gpnp:GPnP-Profile Version="1.0" > xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile"; > xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"; > xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile > gpnp-profile.xsd" ProfileSequence="4" > ClusterUId="548a704f469c5fceff540a328dc5eaf3" ClusterName="xxxx" > PALocation=""> > <gpnp:Network-Profile> > <gpnp:HostNetwork id="gen" HostName="*"> > <gpnp:Network id="net1" IP="xxxx" Adapter="bond0" Use="public"/> > <gpnp:Network id="net2" IP="xxxx" Adapter="bond1" > Use="cluster_interconnect"/> > </gpnp:HostNetwork> > </gpnp:Network-Profile> > <orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/> > <orcl:ASM-Profile id="asm" DiscoveryString="/dev/oracle/*asm*p1" > SPFile="+DG_CLUSTER/beantorarac/asmparameterfile/registry.253.730433213"/> > <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#";> > <ds:SignedInfo> > <ds:CanonicalizationMethod > Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> > <ds:SignatureMethod > Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/> > <ds:Reference URI=""> > <ds:Transforms> > <ds:Transform > Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> > <ds:Transform > Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#";> > <InclusiveNamespaces > xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"; > PrefixList="gpnp orcl xsi"/> > </ds:Transform> > </ds:Transforms> > <ds:DigestMethod > Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/> > <ds:DigestValue>5a1vWvd/aWUneNgBBj/k6vYeSus=</ds:DigestValue> > </ds:Reference> > </ds:SignedInfo> > > <ds:SignatureValue>Q/PJMAMSq6qNxWUiwqNUHCakFJJWlcZW/zA94sDdqcVfc8oUiphDqHwg2TBMaSvnDVfCl+HCq2upiag91XFl55/5O5UGd7ZklkvCPWr3M1nUPf+HdrifgtmuvQ7wUczZ04vUS9U9jVW2YA6ae6Ek/bQARZmozuw/HXg113C4Tq8=</ds:SignatureValue> > </ds:Signature> > </gpnp:GPnP-Profile> > > Success. > > > The string you are looking for is: > > <orcl:ASM-Profile id="asm" DiscoveryString="/dev/oracle/*asm*p1" > > Modify the discoverystring to the correct value > > > Again, editing the profile.xml file is not supported by Oracle. > The supported way is to reconfigure the cluster using the > > $GRID_HOME/crs/install/rootcrs.pl -deconfig -force > $GRID_HOME/crs/install/rootcrs.pl -deconfig -force -lastnode > And then rerun the root scripts on all nodes. > > However, this will erase all the asm diskgroups on which an ocr file is > located. > (this is the reason why you should have a separate asm diskgroup just for > your voting / ocr files) > > Regards, > > Freek D'Hooge > Uptime > Oracle Database Administrator > email: freek.dhooge@xxxxxxxxx > tel +32(0)3 451 23 82 > http://www.uptime.be > disclaimer: www.uptime.be/disclaimer > > -----Original Message----- > From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] > On Behalf Of Martin Klier > Sent: dinsdag 12 juli 2011 13:45 > To: Oracle L > Subject: ASM Grid Infrastructure voting files not found any more > > Dear list, > > I messed up my ASM diskstring, after this and a reboot my cssd's do not > start, because they can't find the voting files any more. Now one node > is fencing, the other one just attempts to restart the CSSD frequently. > > How can I reconfigure the device name mask for the cssd (or the ASM disk > string - if it helps), to allow it to find its voting files in the right > way? > > > Thank you very much, any help is greatly appreciated! > > Martin Klier > > > > Here the corresponding part of the ocssd log file. I think the crucial > part is: > > 2011-07-11 09:36:45.976: [ CSSD][3997759248]clssnmvDiskCreate: Found > a duplicate voting file > /dev/disk/by-id/scsi-360080e50001c169e000002334dfebb44 > in the discovery queue which appears to be the same physical device as > the newly discovered disk /dev/disk/by-id/ASMOCW2. Rejecting both these > files > > It does repeat that for all three disks bearing the voting file copies. > => bad > > ======================================================= > > 2011-07-11 09:36:45.975: [ CSSD][3997759248]clssnmvDiskVerify: > discovered a potential voting file > 2011-07-11 09:36:45.975: [ SKGFD][3997759248]Handle 0x88e450 from lib > :UFS:: for disk :/dev/disk/by-id/ASMOCW2: > > 2011-07-11 09:36:45.976: [ CSSD][3997759248]clssnmFindVF: found VF by > vdin in the discovered queue > 2011-07-11 09:36:45.976: [ CSSD][3997759248]clssnmFindVF: Duplicate > voting file found in the queue of previously discovered disks > queued(/dev/disk/by-id/scsi-360080e5000 > 1c169e000002334dfebb44|[7903ab4c-10b84f92-bfec4e33-5fb310fb]), > found(/dev/disk/by-id/ASMOCW2|[7903ab4c-10b84f92-bfec4e33-5fb310fb]) > 2011-07-11 09:36:45.976: [ CSSD][3997759248]clssnmvDiskCreate: Found > a duplicate voting file > /dev/disk/by-id/scsi-360080e50001c169e000002334dfebb44 in the discovery > queu > e which appears to be the same physical device as the newly discovered > disk /dev/disk/by-id/ASMOCW2. Rejecting both these files > 2011-07-11 09:36:45.976: [ CSSD][3997759248]clssnmvDiskDestroy: > removing the voting disk /dev/disk/by-id/ASMOCW2 > 2011-07-11 09:36:45.976: [ SKGFD][3997759248]Lib :UFS:: closing handle > 0x88e450 for disk :/dev/disk/by-id/ASMOCW2: > > 2011-07-11 09:36:45.976: [ CSSD][3997759248]clssnmvDiskVerify: > discovered a potential voting file > 2011-07-11 09:36:45.976: [ SKGFD][3997759248]Handle 0x88dab0 from lib > :UFS:: for disk :/dev/disk/by-id/ASMOCW3: > > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmFindVF: found VF by > vdin in the discovered queue > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmFindVF: Duplicate > voting file found in the queue of previously discovered disks > queued(/dev/disk/by-id/scsi-360080e5000 > 1c1726000003374e01867c|[70ccc4d5-e1d94f7d-bface149-7b3d8e5f]), > found(/dev/disk/by-id/ASMOCW3|[70ccc4d5-e1d94f7d-bface149-7b3d8e5f]) > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskCreate: Found > a duplicate voting file > /dev/disk/by-id/scsi-360080e50001c1726000003374e01867c in the discovery > queu > e which appears to be the same physical device as the newly discovered > disk /dev/disk/by-id/ASMOCW3. Rejecting both these files > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskDestroy: > removing the voting disk /dev/disk/by-id/ASMOCW3 > 2011-07-11 09:36:45.977: [ SKGFD][3997759248]Lib :UFS:: closing handle > 0x88dab0 for disk :/dev/disk/by-id/ASMOCW3: > > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskVerify: > discovered a potential voting file > 2011-07-11 09:36:45.977: [ SKGFD][3997759248]Handle 0x88dab0 from lib > :UFS:: for disk :/dev/disk/by-id/ASMOCW1: > > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmFindVF: found VF by > vdin in the discovered queue > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmFindVF: Duplicate > voting file found in the queue of previously discovered disks > queued(/dev/disk/by-id/scsi-360080e5000 > 1c1726000002a54dfebb58|[416b1bc2-1f764f0f-bfc1430b-8e447d30]), > found(/dev/disk/by-id/ASMOCW1|[416b1bc2-1f764f0f-bfc1430b-8e447d30]) > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskCreate: Found > a duplicate voting file > /dev/disk/by-id/scsi-360080e50001c1726000002a54dfebb58 in the discovery > queu > e which appears to be the same physical device as the newly discovered > disk /dev/disk/by-id/ASMOCW1. Rejecting both these files > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskDestroy: > removing the voting disk /dev/disk/by-id/ASMOCW1 > 2011-07-11 09:36:45.977: [ SKGFD][3997759248]Lib :UFS:: closing handle > 0x88dab0 for disk :/dev/disk/by-id/ASMOCW1: > > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskVerify: > Successful discovery of 3 disks > 2011-07-11 09:36:45.977: [ > CSSD][3997759248]clssnmCompleteInitVFDiscovery: Completing initial > voting file discovery > 2011-07-11 09:36:45.977: [ > CSSD][3997759248]clssnmCompleteVFDiscovery: Completing voting file > discovery > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskStateChange: > state from discovered to deconfigured disk > /dev/disk/by-id/scsi-360080e50001c1726000002a54dfebb58 > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskStateChange: > state from discovered to deconfigured disk > /dev/disk/by-id/scsi-360080e50001c1726000003374e01867c > 2011-07-11 09:36:45.977: [ CSSD][3997759248]clssnmvDiskStateChange: > state from discovered to deconfigured disk > /dev/disk/by-id/scsi-360080e50001c169e000002334dfebb44 > 2011-07-11 09:36:45.977: [ > CSSD][3997759248]clssnmvVerifyCommittedConfigVFs: Insufficient voting > files found, found 0 of 3 configured, needed 2 voting files > 2011-07-11 09:36:45.977: [ > CSSD][3997759248](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting > file 0, id 416b1bc2-1f764f0f-bfc1430b-8e447d30 not found > 2011-07-11 09:36:45.977: [ > CSSD][3997759248](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting > file 1, id 7903ab4c-10b84f92-bfec4e33-5fb310fb not found > 2011-07-11 09:36:45.977: [ > CSSD][3997759248](:CSSNM00020:)clssnmvVerifyCommittedConfigVFs: voting > file 2, id 70ccc4d5-e1d94f7d-bface149-7b3d8e5f not found > 2011-07-11 09:36:45.977: [ CSSD][3997759248]ASSERT clssnm1.c 3141 > 2011-07-11 09:36:45.977: [ > CSSD][3997759248](:CSSNM00021:)clssnmCompleteVFDiscovery: Found 0 voting > files, but 2 are required. Terminating due to insufficient configure > d voting files > > > -- > Usn's IT Blog for Linux, Oracle, Asterisk > http://www.usn-it.de > > -- > //www.freelists.org/webpage/oracle-l > > > -- > //www.freelists.org/webpage/oracle-l > > > -- //www.freelists.org/webpage/oracle-l