Re: strange error on switchover to standby

  • From: Mladen Gogala <gogala.mladen@xxxxxxxxx>
  • To: Andrew Kerber <andrew.kerber@xxxxxxxxx>
  • Date: Fri, 20 Jan 2017 21:07:04 -0500

You should set the trap in sqlplus, not dgmgrl. The trace file will be written by Oracle kernel, when the error happens.
Regards

On 01/20/2017 09:04 PM, Andrew Kerber wrote:

this works in dgmgrl?

Sent from my iPhone

On Jan 20, 2017, at 7:55 PM, Mladen Gogala <gogala.mladen@xxxxxxxxx <mailto:gogala.mladen@xxxxxxxxx>> wrote:

Hi Andrew,
Why not trap the error:
ALTER SYSTEM SET EVENTS='48189 TRACE NAME ERRORSTACK FOREVER, LEVEL 16';
When the error occurs, it will emit a trace file which will, with any luck, contain the information you need.
Regards

On 01/20/2017 05:50 PM, Andrew Kerber wrote:
So here is where I am. On the primary side, lets call it testp, switchover works fine using dgmgrl. (switchover to testd).

I go over to the testd side, and run a switchover command in testd (switchover to testp), I get the error message:

Error: ORA-48189: OS command to create directory failed
Error: ORA-16625: cannot reach database "TESTD"

However, if I go back over to the original primary side, and run the same switchover command there, it works fine.


I am sure its some sort of privilege problem, but even with the trace level set to support, it wont tell me what command is failing. And unfortunately, I cant even tell from the error message whether its on the testp or testd side that it is trying to create a file, though it always seem to be when its communicating to testd.

There is no command in the drc*.log or the alert log. Is there anywhere else I could look?




On Fri, Jan 20, 2017 at 3:38 PM, Andrew Kerber <andrew.kerber@xxxxxxxxx <mailto:andrew.kerber@xxxxxxxxx>> wrote:

    I tried that one already.  It simply wont show the command that
    is failing:
    Properties:
        FastStartFailoverThreshold   = '30'
        OperationTimeout   = '30'
        TraceLevel   = 'SUPPORT'
        FastStartFailoverLagLimit   = '30'
        CommunicationTimeout   = '180'
        ObserverReconnect   = '0'
        FastStartFailoverAutoReinstate  = 'TRUE'
        FastStartFailoverPmyShutdown   = 'TRUE'
        BystandersFollowRoleChange   = 'ALL'
        ObserverOverride   = 'FALSE'
        ExternalDestination1   = ''
        ExternalDestination2   = ''
        PrimaryLostWriteAction   = 'CONTINUE'


    On Fri, Jan 20, 2017 at 2:47 PM, Bernhard de Cock Buning -
    GRID-IT <bdcbuning@xxxxxxxxxx <mailto:bdcbuning@xxxxxxxxxx>> wrote:

        Hi Andrew,

        Would expect to see more details in the drc logfile.

        If you find out it is related to one server(s) cluster, It
        starts to look like it is related to the log/trace directories.
        Have a look at the ADR on that host and the permissions,
        maybe it is related to the diag_dest parameter on the
        standby location where it defines an incorrect location. But
        I would in that case expect you would already have issues
        for some time and not only when you perform a switchover.

        But if you need more tracing info you can execute in dgmgrll
         (assume you use 11.2.0.3 or higher)
        Admin is the default tracelevel, change it to support.

        edit configuration set property tracelevel=support


        Hope this helps

        Greetings,
        Bernhard

        Op 20 jan. 2017, om 21:32 heeft Andrew Kerber
        <andrew.kerber@xxxxxxxxx> het volgende geschreven:

        interestingly, if I moved over to the original primary
        servers to run the command, the switchover succeeds.  It
        only fails with the error on one cluster, which means I at
least know for sure what servers the error is coming from. Now if I could just figure out what command.

        On Fri, Jan 20, 2017 at 2:10 PM, Andrew Kerber
        <andrew.kerber@xxxxxxxxx> wrote:

            I was trailing that file, and it does not contain the
            command that failed. Here is what is in the broker log
            file:

            Failed to connect to remote database TEST. Error is
            ORA-48189
            Failed to send message to site TEST. Error code is
            ORA-48189.
            database TEST unable to contact primary database for
            version check; status ORA-48189
                  completing bootstrap of this database
            Notifying Oracle Clusterware to buildup


            On Fri, Jan 20, 2017 at 2:05 PM, Bernhard de Cock
            Buning - GRID-IT <bdcbuning@xxxxxxxxxx> wrote:

                Hi Andrew,

                Check out the broker logfile, when correct this
                will explain where it goes wrong.
                You can find this logfile drc*.log  in the same
                location as the alert.log of the database.
                Be sure you check both location (the primary and
                standby)

                You can also enable additional tracing in the
                broker but as mentioned I believe the drc log will
                contain the error.

                Greetings,
                Bernhard

                Op 20 jan. 2017, om 20:56 heeft Andrew Kerber
                <andrew.kerber@xxxxxxxxx> het volgende geschreven:

                I am getting a strange error when I am trying to
                switchover to my standby database. The first
                switchover worked fine, so I am switching over to
                my original primary, and I am getting this message:

                DGMGRL> switchover to 'TEST';
                Performing switchover NOW, please wait...
                Operation requires a connection to instance
                "TEST3" on database "TEST"
                Connecting to instance "TEST3"...
                Connected as SYSDBA.
                Error: ORA-48189: OS command to create directory
                failed
                Error: ORA-16625: cannot reach database "TEST"


                It looks like it is trying to create a directory,
                but I cannot figure out what directory it is
                trying to creat. I expect its a simple ownership
                problem, but I cannot figure out what directory I
                need to work on.  I tried setting the trace level
                to support in dgmgrl, but it didnt show the
                information I needed.  Any ideas?


-- Andrew W. Kerber

                'If at first you dont succeed, dont take up
                skydiving.'




-- Andrew W. Kerber

            'If at first you dont succeed, dont take up skydiving.'




-- Andrew W. Kerber

        'If at first you dont succeed, dont take up skydiving.'




-- Andrew W. Kerber

    'If at first you dont succeed, dont take up skydiving.'




--
Andrew W. Kerber

'If at first you dont succeed, dont take up skydiving.'


--
Mladen Gogala
Oracle DBA
Tel: (347) 321-1217


--
Mladen Gogala
Oracle DBA
Tel: (347) 321-1217

Other related posts: