Re: strange error on switchover to standby

  • From: Mladen Gogala <gogala.mladen@xxxxxxxxx>
  • To: oracle-l@xxxxxxxxxxxxx
  • Date: Fri, 20 Jan 2017 20:55:04 -0500

Hi Andrew,
Why not trap the error:
ALTER SYSTEM SET EVENTS='48189 TRACE NAME ERRORSTACK FOREVER, LEVEL 16';
When the error occurs, it will emit a trace file which will, with any luck, contain the information you need.
Regards

On 01/20/2017 05:50 PM, Andrew Kerber wrote:

So here is where I am. On the primary side, lets call it testp, switchover works fine using dgmgrl. (switchover to testd).

I go over to the testd side, and run a switchover command in testd (switchover to testp), I get the error message:

Error: ORA-48189: OS command to create directory failed
Error: ORA-16625: cannot reach database "TESTD"

However, if I go back over to the original primary side, and run the same switchover command there, it works fine.


I am sure its some sort of privilege problem, but even with the trace level set to support, it wont tell me what command is failing. And unfortunately, I cant even tell from the error message whether its on the testp or testd side that it is trying to create a file, though it always seem to be when its communicating to testd.

There is no command in the drc*.log or the alert log. Is there anywhere else I could look?




On Fri, Jan 20, 2017 at 3:38 PM, Andrew Kerber <andrew.kerber@xxxxxxxxx <mailto:andrew.kerber@xxxxxxxxx>> wrote:

    I tried that one already.  It simply wont show the command that is
    failing:
    Properties:
        FastStartFailoverThreshold   = '30'
        OperationTimeout   = '30'
        TraceLevel   = 'SUPPORT'
        FastStartFailoverLagLimit   = '30'
        CommunicationTimeout   = '180'
        ObserverReconnect   = '0'
        FastStartFailoverAutoReinstate  = 'TRUE'
        FastStartFailoverPmyShutdown   = 'TRUE'
        BystandersFollowRoleChange   = 'ALL'
        ObserverOverride   = 'FALSE'
        ExternalDestination1   = ''
        ExternalDestination2   = ''
        PrimaryLostWriteAction   = 'CONTINUE'


    On Fri, Jan 20, 2017 at 2:47 PM, Bernhard de Cock Buning - GRID-IT
    <bdcbuning@xxxxxxxxxx <mailto:bdcbuning@xxxxxxxxxx>> wrote:

        Hi Andrew,

        Would expect to see more details in the drc logfile.

        If you find out it is related to one server(s) cluster, It
        starts to look like it is related to the log/trace directories.
        Have a look at the ADR on that host and the permissions, maybe
        it is related to the diag_dest parameter on the standby
        location where it defines an incorrect location. But I would
        in that case expect you would already have issues for some
        time and not only when you perform a switchover.

        But if you need more tracing info you can execute in dgmgrll
         (assume you use 11.2.0.3 or higher)
        Admin is the default tracelevel, change it to support.

        edit configuration set property tracelevel=support


        Hope this helps

        Greetings,
        Bernhard

        Op 20 jan. 2017, om 21:32 heeft Andrew Kerber
        <andrew.kerber@xxxxxxxxx <mailto:andrew.kerber@xxxxxxxxx>>
        het volgende geschreven:

        interestingly, if I moved over to the original primary
        servers to run the command, the switchover succeeds.  It only
        fails with the error on one cluster, which means I at least
        know for sure what servers the error is coming from. Now if I
        could just figure out what command.

        On Fri, Jan 20, 2017 at 2:10 PM, Andrew Kerber
        <andrew.kerber@xxxxxxxxx <mailto:andrew.kerber@xxxxxxxxx>> wrote:

            I was trailing that file, and it does not contain the
            command that failed. Here is what is in the broker log file:

            Failed to connect to remote database TEST. Error is ORA-48189
            Failed to send message to site TEST. Error code is ORA-48189.
            database TEST unable to contact primary database for
            version check; status ORA-48189
                  completing bootstrap of this database
            Notifying Oracle Clusterware to buildup


            On Fri, Jan 20, 2017 at 2:05 PM, Bernhard de Cock Buning
            - GRID-IT <bdcbuning@xxxxxxxxxx
            <mailto:bdcbuning@xxxxxxxxxx>> wrote:

                Hi Andrew,

                Check out the broker logfile, when correct this will
                explain where it goes wrong.
                You can find this logfile drc*.log  in the same
                location as the alert.log of the database.
                Be sure you check both location (the primary and standby)

                You can also enable additional tracing in the broker
                but as mentioned I believe the drc log will contain
                the error.

                Greetings,
                Bernhard

                Op 20 jan. 2017, om 20:56 heeft Andrew Kerber
                <andrew.kerber@xxxxxxxxx
                <mailto:andrew.kerber@xxxxxxxxx>> het volgende
                geschreven:

                I am getting a strange error when I am trying to
                switchover to my standby database. The first
                switchover worked fine, so I am switching over to my
                original primary, and I am getting this message:

                DGMGRL> switchover to 'TEST';
                Performing switchover NOW, please wait...
                Operation requires a connection to instance "TEST3"
                on database "TEST"
                Connecting to instance "TEST3"...
                Connected as SYSDBA.
                Error: ORA-48189: OS command to create directory failed
                Error: ORA-16625: cannot reach database "TEST"


                It looks like it is trying to create a directory,
                but I cannot figure out what directory it is trying
                to creat. I expect its a simple ownership problem,
                but I cannot figure out what directory I need to
                work on.  I tried setting the trace level to support
                in dgmgrl, but it didnt show the information I
                needed.  Any ideas?


-- Andrew W. Kerber

                'If at first you dont succeed, dont take up skydiving.'




-- Andrew W. Kerber

            'If at first you dont succeed, dont take up skydiving.'




-- Andrew W. Kerber

        'If at first you dont succeed, dont take up skydiving.'




-- Andrew W. Kerber

    'If at first you dont succeed, dont take up skydiving.'




--
Andrew W. Kerber

'If at first you dont succeed, dont take up skydiving.'


--
Mladen Gogala
Oracle DBA
Tel: (347) 321-1217

Other related posts: