Hi Doug!
Thanks for sharing. This reminds me of the famous Torvalds vs Tanenbaum
debate. Andrew Tanenbaum, a famous university professor and a designer
of the still popular Minix operating system admonished Linux Torvalds,
even more famous designer of the Linux OS, for not using micro-kernel
architecture. While debate itself was extremely technical, as can be
expected from the two top notch OS experts, the winning argument by
Linus Torvalds was - performance. Micro kernel architecture requires a
lot of inter-process communication and the monolithic architecture
simply performs better. Here is the Wikipedia entry about the debate:
https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_debate
Now, what does this famous debate has to do with ever more popular
microservices architecture? Microservices architecture essentially
requires re-architecting the application as a bunch of micro-services
which have to talk to one another, in order for the whole system to
function. The advantages are twofold: easier debugging because each
micro-service is a separate unit and easier adjustment to the container
technologies like Docker or Kubernetes which are usually advertised as
"lightweight virtualization". Well, the opposing principles are the
same: a bunch of chatty services which have to constantly talk to one
another in order to check that everything is working as it should vs.
large application which is harder to debug and harder to put into a
container, but doesn't have radical communication requirements. The
solution is relatively simple: the microservices will have to separate
status data from the user data and use something else for status
exchange, something like MySQL. That would not make the microservices
cheaper, it will just make them consume the cheaper resource. That,
however, requires another round of programming which might not be so cheap.
Regards
On 8/13/20 12:34 AM, DOUG KUSHNER wrote:
--
Our dev team recently rolled out an application using an F5 load-balanced microservices architecture. There are several miscroservices, each load balanced on up to 4 servers each, and each with a health-check api that hits the database. While this may have looked good on paper, just the overhead of the health-checks with no work being processed has resulted in roughly 7 connection attempts per second to the database. This results in a version check query about 40K times per hour. The database is on an Exadata (2-node RAC) with several other production databases.
Of course the Exadata has been handling it, so unless you are looking for anomalies (which I always am), this will fly under the radar until it doesn't. :)
I'm wondering if anyone knows how to determine the theoretical max connections/sec that a listener can handle based on the number of cores licensed in the system?
Also wondering if anyone here has encountered this scenario before and how they dealt with it. I'm also looking for a good reference on the subject.
My immediate focus will be on determining why these health check connections do not appear to be utilizing the services' connection pools, while the dev team determines whether they can relax the frequency of these health checks.
Regards,
Doug