Hi all,
In my testing, when a client is connected to a multi-az RDS instance and I
force failover, that client doesn't 'see' it. If it makes any DB request after
or during the failover it ends up timing out - eventually. Unfortunately this
timeout is controlled by the tcp keepalive setting which defaults to 2 hours.
Not very helpful when the actual failover can be complete in a couple of
minutes.
I'm wondering what other RDS users are doing to handle this scenario.
I've tried tinkering with sqlnet.ora params but couldn't find any that would
allow a connected client to detect the failover.
We may have many connected clients - both on-prem and from AWS - including our
on-prem DB servers using DB links. I'm reluctant to muck with OS-level tcp
keepalive params, and in some cases that may not even be possible (e.g. from
another RDS instance).
My current solution involves using socat (http://www.dest-unreach.org/socat/)
as a proxy. I can easily adjust the tcp keepalive parameters with this and
depending on the values I set for those parameters I can detect failover almost
immediately.
However it means either running socat on every client, or having a dedicated
containter/ec2 running socat - which I then have to monitor. If that
container/ec2 fails but the DB doesn't all in-flight connections are lost.
I'm thinking there has to be a better way. I've contacted AWS support but they
suggested mucking with the tcp keepalive settings on all clients. Or
alternatively using SNS and Lambdas to notify/kill connected clients. The
latter wasn't ideal because the Lambda didn't get fired until well after the
failover (8-10 mins), and I have a mixture of AWS and on-prem clients, so the
notification part is messy.
Thanks for any suggestions.
Steve
------------------------------------------------------------------
This email is intended solely for the use of the
addressee and may contain information that is
confidential, proprietary, or both. If you receive
this email in error please immediately notify
the sender and delete the email.
------------------------------------------------------------------
