Oh yeah please do upgrade Checking in our bug database, It was fixed in
20.4.1
-kris
On Tue, Oct 19, 2021 at 8:12 PM Ruan Linehan <ruandav@xxxxxxxxx> wrote:
Hi Kris,
We are using Oracle Rest Data Services version 20.3.0.
I've scanned through the Release Notes for subsequent versions, but I've
not yet been able to identify an obvious match to what you describe below
within the issues fixed or changes.
Regards,
Ruan
On Tue, Oct 19, 2021 at 3:43 PM kris rice <kris.rice@xxxxxxxx> wrote:
Ruan,
There was an issue a while back that if not available at startup it was
flagged and never revisited. That was changed but I can not recall what
version. There were basically 2 issues that got addressed. 1) what you see
in that a down db is bad listed and 2) connecting to every defined db at
startup. What version are you using?
-kris
On Mon, Oct 18, 2021 at 7:24 PM Ruan Linehan <ruandav@xxxxxxxxx> wrote:
Hi Kris/Tim,
After quite a few various scenario based attempts at "breaking" the
connectivity between ORDS and the DB endpoints, I think I have observed
what is really going on. Trying to recreate my original suggested symptom
by creating a service availability disconnect, even for long periods of
time, was unsuccessful; insofar as ORDS always then successfully resumed
work once the DB connection was re-established (Which is good). So
apologies for mis-representing the issue in my original mail.
I investigated some of the older historical APEX logs. What is more
likely happening in our situation, is that restarts of the ORDS services
(i.e. Which we do frequently to introduce new endpoint configs) on the load
balanced VMs is sometimes crossing over with periodic maintenance window
periods of individual pluggable databases.
Therefore, sometimes it will happen that for 1 in 100 endpoints, there
will be an ORDS complaint with respect to the initial startup validation of
the pool config. e.g. "WARNING: The pool named: |apex|pu| is missing and
will be ignored: The database service named: |apex|pu| does not exist."
These startup errors were not being properly trapped / flagged as part
of our processes, so I can easily fix that.
Testing of this scenario with an ORDS startup, whilst a PDB database
service is currently down, does result in the connection never establishing
once the PDB service is eventually started. So I believe this is actually
what is occurring for us; Not a disconnect of the database from ORDS, but
it is the initial startup verification which if unsuccessful for a
particular endpoint, means that the pool entry is literally "ignored" from
then on.
I assume this is the expected behaviour and if so, is there any work
around beyond a restart of ORDS at that point?
Kind regards,
Ruan
On Tue, Oct 12, 2021 at 9:58 PM Ruan Linehan <ruandav@xxxxxxxxx> wrote:
*"and it never requires a restart. The pools should reestablish
themselves as needed"*
Thanks Tim and Kris for taking the time to reply.
Well, that is quite puzzling to me that our 'broken' connection issue
seems unexpected to you both; but it also makes me hopeful that this is
some mis-config or error in implementation on our part. I'm certainly not
proficient in configuring ORDS as I'm usually investigating from the other
side of the (database) fence. I'm intrigued now, as you mentioned Tim, that
maybe we have some sequence of events or triggers leading to our particular
issue. Yes, we have multiple ORDS installs on separate VMs behind a
load-balancer.
I've just "broken" a non-production environment ORDS web services pool
connection in the last hour, by stopping the PDB services and killing
existing ORDS sessions to test. I'm going to leave this down now for a few
different periods of hours to see what happens and will reply back here
with some specifics.
Kind regards
Ruan
On Tue, Oct 12, 2021 at 1:57 PM kris rice <kris.rice@xxxxxxxx> wrote:
I'd suggest, as always, upgrade. We have ords nodes on some databases
that go up/down, active/readonlny,... and it never requires a restart. The
pools should reestablish themselves as needed. For example, I have to
manage all the ords nodes in Autonomous DB and those things come and go on
the whim of customers kicking tires or shutting down to save money when
not
in use. These ords nodes run somewhere around 2k connection pools each
and
never need a restart, even when the pdb is relocated out from under us to
another CDB.
Happy to jump on a zoom or medium of choice and chat more if you'd
like.
-kris
On Tue, Oct 12, 2021 at 7:43 AM Tim Hall <tim@xxxxxxxxxxxxxxx> wrote:
Hi.
That's interesting.I can't ever remember having to restart ORDS as a
result of a database outage. Even prolonged ones. We install in the PDB,
not the CDB and have one or more ORDS instances in Docker containers for
each PDB. As a result, a problem with one instance doesn't affect
everything else. Even so, these are basic connection issues you are
having,
so I don't think the topology differences can be that relevant. I think
this may be a job for the ORDS team. They could certainly tell you what
the
expected behaviour is.
Do you have a non-prod/test setup where you can test some failure
scenarios? I wonder if there are specific patterns that cause the issue,
rather than a general overarching issue.
I guess in the interim I would consider a mitigation. I assume you
have multiple ORDS installations behind a load balancer to support this.
If
so, you could script a restart of all ORDS instances (one at a time of
course), and call that at the end of every piece of scheduled
maintenance.
It would minimise the apparent outage.
I'll see if I can get someone from the ORDS team to look at this
thread.
Cheers
Tim...
On Tue, Oct 12, 2021 at 9:22 AM Ruan Linehan <ruandav@xxxxxxxxx>
wrote:
Hi all,
I've researched elsewhere but not been able to identify a suitable
solution, so I'm asking here in the hopes that an ORDS aficionado might
provide some direction.
My issue is around the perception of a restart of ORDS being a
requirement to re-establishing a connection to an endpoint which may
have
been unavailable for a period of time.
We run ORDS v20 on a Linux VM as part of a solution accompanying an
Exadata multitenant environment. ORDS is made available to all PDBs,
installed in the CDB. Within the 'conf' directory of ORDS - we stage all
the associated apex_aa.xml, apex_ab.xml, apex_ac.xml etc configuration
mapping files. Periodically, one of the PDB environments may be made
unavailable (i.e Closed or else RAC services stopped, or someone
inadvertently locks the ORDS_PUBLIC_USER account etc) for maintenance,
for
a day or weekend etc. When this takes place, the pluggables
ORDS_PUBLIC_USER database sessions are terminated and the ORDS
connection
cannot be re-established for a period of time to that PDB. So far so
good.
Once the maintenance is complete, and the PDB is re-opened once
again, RAC services restarted, ORDS does not automatically re-establish
a
database connection to that same PDB.
If I need to get the ORDS_PUBLIC_USER connections re-established
once more for that specific PDB, then I need to stop ORDS processes for
all
clients and restart.
i.e. This reads the url mapping xml and validates the associated
apex_aa.xml files etc., and eventually successfully re-establishes ALL
the
database connection and all is good.
The difficulty is though, that we have literally hundreds of these
PDBs in a CDB, and literally hundreds of accompanying ORDS endpoints.
So,
if one of these environments is impacted by a "maintenance" of some
sort,
and the database connection is severed for a time, then it requires a
full
restart of ORDS for ALL to get it back, which is rather painful.
There must be something I am missing right? I understand XML config
changes require a restart of ORDS to be picked up, but I find it
troubling
that a full restart is also required when just one client endpoint
connection out of a hundred is impacted?
Is there any way I can force ORDS to 'reinit' and re-read the conf
files to re-establish a single broken connection with restarting?
Kind regards,
Ruan
