Well, one final(?) piece of information. On Friday, one of the slow boxes
was rebooted. After it came back up I ran turbostat and the. J.Lewis cpu
heavy SQL. The turbostat output had PKG_% (RAPL throttling) at 0 and the
SQL ran at the fast server speed (~20sec intead of 50).
So a reboot seemed to flip something back. Not a really satisfying answer
as I don't know what started this to begin with and have no way to tell if
it will happen again.
Thanks for all your help.
Henry
On Sep 5, 2017 09:32, "Henry Poras" <henry.poras@xxxxxxxxx> wrote:
Stefen,
If I didn't post the kernel, I should have
$uname -r
2.6.32-696.1.1.e16.x86_64
All I know about perf is that the fiber cards had driver issues with newer
kernels which is what is preventing the installation of perf.
I'll take a look at some of your links.
Thanks for the pointers.
Henry
On Sep 5, 2017 5:15 AM, "Stefan Koehler" <contact@xxxxxxxx> wrote:
Hello Henry,
1. Our sysadm said that there are kernel issues and "perf stat" cannotrun on these systems
I really would like to hear about these kernel issues and his concerns -
never seen an issue with these.
Also RAPL throttling is on in the slow server. I am guessing this couldaccount for the throttled cpu speed. I am starting to google search to see
where this can be set from. Anyone out there know any good resources on
RAPL throttling or how to disable it?
Unfortunately you have not mentioned your linux kernel version yet (or i
just missed it) but there are several ways to drill down into RAPL.
1) PowerCap interface: https://www.kernel.org/doc/Doc
umentation/power/powercap/powercap.txt
2) Perf
3) MSR registers (/dev/msr)
However if you can do the fancy stuff at all depends on BIOS (and
vendor). If you run dmesg you may see some messages about "intel_rapl" to
get an idea if it works anyway with your hardware.
Best Regards
Stefan Koehler
Independent Oracle performance consultant and researcher
Website: http://www.soocs.de
Twitter: @OracleSK
Henry Poras <henry.poras@xxxxxxxxx> hat am 5. September 2017 um 06:39geschrieben:
run on these systems
Figured it was time to send the list an update.
1. Our sysadm said that there are kernel issues and "perf stat" cannot
2. I installed a slob instance on both a fast and slow server. Therewas no way to stop our workload from running, but I ran slob multiple
times, trying to keep each run short (~3 minutes) so the underlying
workload would not vary much. Also tried running when the load was
relatively low. All of the run results were pretty consistent with one
another. Configuring SLOB to run strictly LIO, I saw ~4M lio/sec vs. 1.5M
lio/sec
3. Our sysadm did get turbostat installed. What I take as the mostimportant output from that was:
slow: TSC_MHz - 2600 Bzy_MHz - 1200 PKG_% - 100%busy are very different. Also RAPL throttling is on in the slow server. I
fast: TSC_MHz - 2606 Bzy_MHz - 3110 PKG_% - 0%
So the clock speeds are nearly identical, ,but the running speeds when
am guessing this could account for the throttled cpu speed. I am starting
to google search to see where this can be set from. Anyone out there know
any good resources on RAPL throttling or how to disable it?
Thanks.
Henry
