Fantastic, thanks a lot for your input. Indeed, it looks like setting
Power Regulator to 'OS Control mode' stabilizes the system and *_raw_spin_lock
*and *acpi_os_write_port* overhead disappears . Unfortunately I also don't
have access to redhat support to further explore details about this issue.
I have the feeling now that the way cpu frequency is modulated is variable
and I have some performance penalty with a single core activity. I tested
this by running a query in the DB and comparing it with time/wait events on
a non-upgraded node. By running the query with parallel degree it looks
like execution time was similar.
Thanks again for your help.
În joi, 19 aug. 2021 la 13:58, Andre Maasikas <amaasikas@xxxxxxxxx> a scris:
Looks very much like OS and platform are constantly trying/fighting
over cpu frequency, smth like
https://access.redhat.com/solutions/3393161
although i have no access to the rest of that doc.
Since I have not seen the HW details nor the kernel log I cannot say
exactly.
I'd suggest if you are on HP turn Power Regulator to 'OS Control mode'
and ion advanced options disable 'Collaborative Power management' or smth
or investigate how to get rid/replace of pcc-cpufreq driver which seems to
be the problem
Andre
On Thu, Aug 19, 2021 at 12:57 PM Laurentiu Oprea
<laurentiu.oprea06@xxxxxxxxx> wrote:
15-20% in %usr space and 35-40% in %sys space.
Thanks a lot Tanel for the valuable information.
mpstat will show me 0 load for %irq , on average each core is around
_raw_spin_lock and and acpi_os_write_port.
By looking into a section of FG it looks like kworker is the culprit for
this is OL6 . Any ideas on what I can do as a workaround?
I`m somehow at a dead end as I`m not able to request a bug fix for it as
scris:
În mie., 18 aug. 2021 la 10:48, Tanel Poder <tanel@xxxxxxxxxxxxxx> a
threads :-)
Sorry, been swamped and haven't had time to lurk around oracle-l
stuff). So, when using libaio, you'd see kworkers involved in I/Os and
The kworker threads do asynch I/O stuff in Linux kernel (among other
apparently asmlib also ends up using kworkers under the hood (I never
looked into asmlib in detail, but the psn/kstack output indicates kworker
use too).
hardware changes, like this one:
I'd say it's some asmlib/kernel bug, especially if you didn't do any
Comsumes High CPU Under Low System Load by kworker and ksoftirqd (Doc ID
Oracle Linux : OL6 Running On Exadata Observes A Particular CPU
2645898.1) - "Known issue with kernel version 4.1.12-124.24.3.el6uek.x86_64
and lower."
behavior - mpstat would show how much (CPU) time you spend on
Sometimes various interrupt storms & issues can also cause this weird
hardware/software interrupts.
left-side of the FG_UEK4_all_30seconds.jpg screenshot, what do you see?
If you click on some of the stack sections on the very busy looking
when you use perf -a, as that option reports all CPU time of all CPUs
Note that the intel_idle and related stack frames always show up anyway
(including CPU idle time, which is not the same as a process/thread idle
time, essentially it's CPU's time spent in
no-work/low-power-suspend/poweroff states). This is different from perf
record -p behavior as -p would only show on-CPU samples of a process/thread
and not show off-CPU samples of a process/thread.
a spinlock in that function), look up (with the FlameGraph search box, like
As you see lots of _raw_spin_lock contention (CPU time spent to acquire
I did in my blog entry), who calls that function? That would tell you which
type of spinlock has that contention.
As so frequently interacting with hardware via I/O ports (instead of DMA)
Also the fact that you see lots of acpi_os_write_port calls is weird.
so that this acpi_os_write_port function shows up in the top CPU users,
also indicates some sort of a kernel<->hardware communication issue... see
what functions are its parents, grandparents, it may provide extra insight.
laurentiu.oprea06@xxxxxxxxx> wrote:
--
Tanel Poder
https://tanelpoder.com/events
On Fri, Aug 13, 2021 at 9:54 AM Laurentiu Oprea <
laurentiu.oprea06@xxxxxxxxx> a scris:
quite suspicious is that running :
perf record -F 99 -g a sleep 30
and looking in FG looks like the word "idle" is a big chunk of it.
În vin., 13 aug. 2021 la 11:25, Laurentiu Oprea <
cpu core / booting with acpi=ht I`ll not have the single cpu issue but my
Thanks a lot for all the answers. Taking in order of what I tried:
-> booting with acpi=off will apparently make the system see just 1
load/slowness issue remains there -> not sure what I can explore more on
this subject
observed that starting just 8 instances on the upgrade node will not
-> On this particular rac cluster I have around 32 databases, I
generate a significant CPU load, starting 12 or more will make the
situation far worse (the other 2 nodes with UEK3 are running perfectly well
with all instances up) . So I started just 8 instances and I wrote a test
query (simple select with group by, all significant ash time being spent
on CPU) that executed on the upgraded node will run in 68 seconds while on
non-upgraded node will run in 38. I sampled with psn the spid of the
session while running the statement and also I sampled with perf and
generated a flamegraph.
distribution perspective is just that each call takes longer time .
From FG I might conclude that all the calls are identical from a
(-) or retin_carefull(), is just that on nodes running UEK3 is runningFrom PSN Looks like the majority of samples are on CPU with no kstack
faster so less samples are collected.
running UEK4 compared with UEK3 (and maybe seeing those kworker process is
I`m starting to think that there is actually a general slowness
just a side effect) While the system is slow the kstack for kworkers is
just: ret_from_fork() ->kthread -> worker_thread() )
scris:
Basically I`m out of ideas, so any suggestions are very welcomed :)
Thank you.
În joi, 12 aug. 2021 la 20:29, Tanel Poder <tanel@xxxxxxxxxxxxxx> a
exists there, but you can still use the techniques from my blog entry to
Oh, UEK 4 is Linux kernel 4.1.x, I don't recall if that bug/problem
drill down deeper. One of the main questions is - is your Linux system load
high due to on-CPU threads or threads in D state. perf record -g would help
you drill down more into the CPU usage and from the full stack traces you'd
see which kernel functions are trying to get that spinlock you already saw
contention on, but if it's mostly D state sleeping threads, then
/proc/PID/stack or psnapper would tell you more.
wrote:
On Thu, Aug 12, 2021 at 1:26 PM Tanel Poder <tanel@xxxxxxxxxxxxxx>
Oracle & kworker processes (that are in D state) and check if you have
Hi,
Run psnapper or just sudo cat /proc/PID/stack for some of the
wbt_wait in the stack. UEK 4 (kernel 4.14) has a Linux kernel write
throttling bug/problem and lots of kworkers showing up is one of the
symptoms. I've written about drilling down into high Linux system load in
general here:
https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/
upcoming "troubleshooting very complex Oracle performance problems" virtual
Incidentally, this is one of the problem scenarios I cover at the
"conference" (where all speakers are me - https://tanelpoder.com/events
;-)
laurentiu.oprea06@xxxxxxxxx> wrote:
--
Tanel Poder
On Thu, Aug 12, 2021 at 2:41 AM Laurentiu Oprea <
similar issue.
Hello everyone,
I know this is more of a linux question but maybe someone had a
servers (rac 2/3 nodes) running OL6. I had no issue upgrading the kernel on
I started to upgrade the kernel version from UEK3 to UEK4 on
virtual machines.
after starting the oracle instances the cpu load went crazy, the whole
When I upgraded the first rac node located on physical hardware,
system was far slower. For example the cpu load on the other rac nodes with
UEK3 was 4/3/3/ while the upgraded node load was 107/104/77 with only half
the instances started, this workload being created even if the instances
had no significant workload (they were just started)
start there is quite a big number of kworker processes. Perf top will show
I observed that with the new kernel version when oracle instances
me as overhead
17.24% [kernel] _raw_spin_lock
4.50% [kernel] acpi_os_write_port
Does anyone have any idea on what could be the issue?
THanks a lot.
