[procps] top: NUMA node CPU utilization support
- From: Jim Warner <james.warner@xxxxxxxxxxx>
- To: procps@xxxxxxxxxxxxx
- Date: Tue, 9 Apr 2013 08:05:03 -0500
On Apr 8, 2013, at 6:58 PM, Lance Shelton <LShelton@xxxxxxxxxxxx> wrote:
> The libnuma/numactl packages needed for the this support are not always
> installed by default in RHEL and SLES. Jim added a compile-out feature to
> ensure compatibility, but for these NUMA features for top to get compiled
> in, will these distributions need to include these packages even in their
> minimal distribution?
Hello Craig & Jaromir,
When a distro prepares their version procps, if it's compiled in the presence
of libnuma and numa.h then top will have NUMA support built-in. But if libnuma
is not then part of that distro's basic system, top would break with a loader
error.
This dependency issue is way beyond me but I assured Lance you guys would know.
Anyway, I thought you might be interested in some actual top usage examples.
Lance was kind enough to share the attached captures as we were finalizing the
NUMA support and he has authorized me to pass them along.
Regards,
Jim
4/8/13, from: Lance Shelton <LShelton@xxxxxxxxxxxx>
First, my application without any CPU pinning. Notice that most nodes
have some CPU usage:
lt-top - 07:17:10 up 3 days, 16:46, 4 users, load average: 7.77, 7.16, 4.13
Tasks: 1344 total, 6 running, 1338 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 3.0 sy, 0.0 ni, 94.2 id, 0.2 wa, 0.0 hi, 2.1 si, 0.0 st
%Node0 : 0.3 us, 6.8 sy, 0.0 ni, 86.9 id, 1.1 wa, 0.0 hi, 4.9 si, 0.0 st
%Node1 : 0.3 us, 4.7 sy, 0.0 ni, 91.5 id, 0.1 wa, 0.0 hi, 3.5 si, 0.0 st
%Node2 : 0.0 us, 1.2 sy, 0.0 ni, 97.8 id, 0.1 wa, 0.0 hi, 0.9 si, 0.0 st
%Node3 : 0.1 us, 0.2 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node4 : 0.0 us, 0.3 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st
%Node5 : 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node6 : 0.2 us, 4.1 sy, 0.0 ni, 92.4 id, 0.4 wa, 0.0 hi, 2.9 si, 0.0 st
%Node7 : 3.0 us, 6.5 sy, 0.0 ni, 85.5 id, 0.2 wa, 0.0 hi, 4.8 si, 0.0 st
KiB Mem: 26399888+total, 11982297+used, 14417590+free, 309928 buffers
KiB Swap: 20971512 total, 0 used, 20971512 free, 4081096 cached
For the rest of these captures I have added pinning. The interface cards
I am sending I/O to are located near nodes 0-3, so here is my application
with threads pinned to cores at nodes 0 and 2 (visible in that nodes 0 and
2 are the least idle):
lt-top - 07:15:26 up 3 days, 16:45, 4 users, load average: 10.32, 6.95, 3.70
Tasks: 1344 total, 6 running, 1338 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.5 us, 2.8 sy, 0.0 ni, 94.1 id, 1.3 wa, 0.0 hi, 1.3 si, 0.0 st
%Node0 : 0.7 us, 12.2 sy, 0.0 ni, 77.3 id, 4.5 wa, 0.0 hi, 5.2 si, 0.0 st
%Node1 : 0.1 us, 0.0 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node2 : 0.6 us, 10.6 sy, 0.0 ni, 77.4 id, 5.9 wa, 0.0 hi, 5.5 si, 0.0 st
%Node3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node4 : 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node5 : 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node6 : 0.0 us, 0.1 sy, 0.0 ni, 99.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Node7 : 2.6 us, 0.4 sy, 0.0 ni, 97.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 26399888+total, 11980486+used, 14419403+free, 309928 buffers
KiB Swap: 20971512 total, 0 used, 20971512 free, 4080688 cached
Now I can look at the CPUs in node 0 to see how CPU usage is distributed
between the cores in these nodes:
lt-top - 07:15:12 up 3 days, 16:45, 4 users, load average: 10.34, 6.77, 3.59
Tasks: 1344 total, 9 running, 1335 sleeping, 0 stopped, 0 zombie
%Node0 : 0.7 us, 12.0 sy, 0.0 ni, 77.5 id, 4.5 wa, 0.0 hi, 5.3 si, 0.0 st
%Cpu0 : 3.7 us, 48.3 sy, 0.0 ni, 2.2 id, 0.0 wa, 0.0 hi, 45.8 si, 0.0 st
%Cpu1 : 1.4 us, 30.2 sy, 0.0 ni, 48.6 id, 10.8 wa, 0.0 hi, 9.0 si, 0.0 st
%Cpu2 : 0.7 us, 22.8 sy, 0.0 ni, 58.1 id, 11.2 wa, 0.0 hi, 7.1 si, 0.0 st
%Cpu3 : 1.1 us, 19.8 sy, 0.0 ni, 61.5 id, 11.4 wa, 0.0 hi, 6.2 si, 0.0 st
%Cpu4 : 1.4 us, 22.3 sy, 0.0 ni, 59.0 id, 10.4 wa, 0.0 hi, 6.8 si, 0.0 st
%Cpu5 : 1.4 us, 21.9 sy, 0.0 ni, 60.2 id, 10.0 wa, 0.0 hi, 6.5 si, 0.0 st
%Cpu6 : 1.1 us, 21.3 sy, 0.0 ni, 58.8 id, 10.8 wa, 0.0 hi, 7.9 si, 0.0 st
%Cpu7 : 1.1 us, 20.4 sy, 0.0 ni, 60.7 id, 10.5 wa, 0.0 hi, 7.3 si, 0.0 st
%Cpu8 : 1.1 us, 20.1 sy, 0.0 ni, 62.7 id, 10.4 wa, 0.0 hi, 5.7 si, 0.0 st
%Cpu9 : 1.4 us, 21.2 sy, 0.0 ni, 60.1 id, 10.1 wa, 0.0 hi, 7.2 si, 0.0 st
%Cpu80 : 0.3 us, 4.6 sy, 0.0 ni, 92.5 id, 0.3 wa, 0.0 hi, 2.3 si, 0.0 st
%Cpu81 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu82 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu83 : 0.0 us, 1.0 sy, 0.0 ni, 98.4 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu84 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu85 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu86 : 0.0 us, 0.7 sy, 0.0 ni, 98.7 id, 0.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu87 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu88 : 0.0 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu89 : 0.0 us, 0.3 sy, 0.0 ni, 99.3 id, 0.3 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 26399888+total, 11981024+used, 14418864+free, 309924 buffers
KiB Swap: 20971512 total, 0 used, 20971512 free, 4080684 cached
Finally, I can verify that not much is going on with the nodes that I do
not want to use for this application:
lt-top - 07:16:08 up 3 days, 16:45, 4 users, load average: 10.12, 7.33, 3.97
Tasks: 1344 total, 3 running, 1341 sleeping, 0 stopped, 0 zombie
%Node6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu60 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu61 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu62 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu63 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu64 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu65 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu66 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu67 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu68 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu69 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu140: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu141: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu142: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu143: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu144: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu145: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu146: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu147: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu148: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu149: 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 26399888+total, 11980511+used, 14419377+free, 309928 buffers
KiB Swap: 20971512 total, 0 used, 20971512 free, 4080896 cached
Other related posts:
