I think I finally got to the bottom of this - running more tests to
confirm. It seems like, due to parallel_degree_limit=CPU,
parallel_degree_policy=MANUAL, and the default DOP being so low (2), the
kernel decided the CPU was too loaded to grant the requested DOP, so
instead calcuated that a DOP of 2 would be easier on the CPU.
Unfortunately, while this is indeed true, it kills performance.
It seems like using parallel_degree_policy=AUTO and
parallel_degree_limit=100 (or some other high number, maybe even "IO") will
avoid the kernel freaking out because the CPU has a little load on it, and
the added advantage is that AUTO will defer until more slaves are
available. Going to bump cpu_count=256 for another test, as well.
Hat tip to Anju Garg's blog Oracle In Action
<http://oracleinaction.com/tag/parallel_adaptive_multi_user/> (has several
posts on parallelism). Thanks again to Jonathan Lewis and Stefan Koehler
for helping me get started on the science and diagnostics.
--
Charles Schultz
