Extended trace analysis and perf. problem.
- From: Brian Wisniewski <brian_wisniewski@xxxxxxxxx>
- To: oracle-l@xxxxxxxxxxxxx
- Date: Fri, 13 May 2005 05:49:46 -0700 (PDT)
We are in process of moving from a 2-node (4 CPU per node Xeon 1.6GHz) RAC to a
3-node (2 CPU per node 3.0GHz) RAC. The 2 CPU machines are hyper-threaded so
it appears as 4-CPU's. We're splitting databases that currently sit on the
2-node into seperate RAC's. Going from RHAS 2.1 to RHAS 3.0. Also going from
a mix of raw devices and OCFS in the existing environment to complete OCFS and
from a dedicated switch to a VLAN interconnect. Is it possible to change
anything else? :-)
This is a 600gb database that stores a lot of blobs. I copied a backup over to
the new server, restored it and applied some archive logs from production and
then opened it for testing.
Doing performance testing on the new machine and of course it is abysmal - it
would have been too good to be true for it to actually work! Loads are about
1/2 as fast. I've extended trace'd loads in the current production and in the
new production and that where I'm lost. All of the wait time averages for the
new setup are right in line with current production. If this was database
related I'd expect to see some wait-time averages way off the chart somewhere -
wouldn't you? Am I missing something if this is database related - shouldn't
it show in the extended traces if it was?
I don't know what this is going to come across like but I'll give it a shot.
This is my table holding the wait stats of the loads. The 1st column is the
avg wait time (microseconds), the 2nd column is the number of events it pulled
out of the trace file, 3rd is the wait event and 4th is the name of the run.
RAC 'C' is the new cluster - all the others are current production runs.
Any help is appreciated or if my assumptions are incorrect I'd be happy to know.
Thanks - Brian
AVG COUNT(*) WAIT_EVENT LOAD_TXT
---------- ---------- ----------------------------------------
------------------------------
27.50 2 KJC Wait for msg sends to complete Prod run 25464
05/09
152.00 12 KJC Wait for msg sends to complete Prod run 8317
172.61 33 KJC Wait for msg sends to complete Prod run 14892
201.77 13 KJC Wait for msg sends to complete Prod run amcc
359.33 6 KJC Wait for msg sends to complete RAC C 3node async
1407.40 5 buffer busy global CR Prod run 14892
3833.00 9 buffer busy global CR Prod run 25464
05/09
16060.80 5 buffer busy global cache Prod run 14892
50.00 1 buffer busy waits Prod run 14892
72.40 15 buffer busy waits Prod run 25464
05/09
1753.50 8 buffer busy waits Prod run 8317
3218.38 23297 db file sequential read Prod run 25464
05/09
5741.80 117792 db file sequential read Prod run 14892
6326.64 85276 db file sequential read Prod run 8317
7326.37 60987 db file sequential read RAC C 3node
async cert client
7626.50 57413 db file sequential read Prod run amcc
8002.31 46649 db file sequential read RAC C 3node async
8243.14 57986 db file sequential read RAC C 1node async
8907.66 20778 db file sequential read RAC C 3node
noasync initial
5.50 12 direct path write (lob) RAC C 3node async
6.17 12 direct path write (lob) RAC C 1node async
6.50 14 direct path write (lob) RAC C 3node
async cert client
10.09 5998 direct path write (lob) Prod run amcc
10.59 13329 direct path write (lob) Prod run 14892
10.80 6893 direct path write (lob) Prod run 8317
15.00 4 direct path write (lob) Prod run 25464
05/09
123.00 2 enqueue RAC C 3node
async cert client
133.00 44 enqueue Prod run amcc
181.73 143 enqueue RAC C 3node
noasync initial
225.75 215 enqueue RAC C 3node async
232.69 55 enqueue Prod run 8317
233.93 151 enqueue Prod run 25464
05/09
1429.65 142 enqueue Prod run 14892
336.00 2 global cache busy Prod run 14892
14982.00 2 global cache busy Prod run 25464
05/09
111.22 58269 global cache cr request Prod run amcc
134.23 123907 global cache cr request Prod run 14892
135.20 84376 global cache cr request Prod run 8317
184.61 25831 global cache cr request Prod run 25464
05/09
239.21 99254 global cache cr request RAC C 3node
async cert client
251.68 62661 global cache cr request RAC C 3node async
281.58 27544 global cache cr request RAC C 3node
noasync initial
152.00 2 global cache null to s Prod run amcc
160.50 2 global cache null to s Prod run 8317
176.04 28 global cache null to s Prod run 25464
05/09
135.64 28 global cache null to x Prod run amcc
147.58 12 global cache null to x Prod run 8317
6870.91 1154 global cache null to x Prod run 25464
05/09
12296.99 467 global cache null to x Prod run 14892
129.13 8 global cache open s Prod run 8317
172.40 10 global cache open s RAC C 3node async
183.60 10 global cache open s Prod run 14892
216.66 376 global cache open s RAC C 3node
async cert client
608.91 43 global cache open s Prod run 25464
05/09
92.74 1007 global cache open x RAC C 3node async
119.83 192 global cache open x Prod run amcc
121.28 648 global cache open x RAC C 3node
noasync initial
125.09 733 global cache open x Prod run 8317
298.99 5641 global cache open x RAC C 3node
async cert client
548.07 1363 global cache open x Prod run 14892
553.81 1238 global cache open x Prod run 25464
05/09
109.86 3145 global cache s to x Prod run amcc
132.11 8585 global cache s to x Prod run 14892
133.69 5958 global cache s to x Prod run 8317
137.21 1286 global cache s to x Prod run 25464
05/09
219.80 2205 global cache s to x RAC C 3node
async cert client
245.66 2344 global cache s to x RAC C 3node async
277.20 1236 global cache s to x RAC C 3node
noasync initial
36.67 3 latch free Prod run amcc
80.46 13 latch free Prod run 25464
05/09
200.56 16 latch free Prod run 8317
2074.43 23 latch free Prod run 14892
125.80 246 library cache lock Prod run amcc
181.92 637 library cache lock Prod run 14892
189.28 276 library cache lock Prod run 8317
208.26 61 library cache lock Prod run 25464
05/09
213.79 66 library cache lock RAC C 3node
async cert client
230.29 7 library cache lock RAC C 3node async
244.17 18 library cache lock RAC C 3node
noasync initial
80.00 1 library cache pin RAC C 1node async
499.00 1 library cache pin RAC C 3node async
866.00 1 library cache pin RAC C 3node
async cert client
81776.00 2 log file switch completion Prod run 14892
106827.50 2 log file switch completion Prod run 8317
6249.67 12 log file sync Prod run 14892
6257.25 4 log file sync RAC C 1node async
6409.80 5 log file sync Prod run amcc
11107.67 3 log file sync RAC C 3node
async cert client
11291.33 3 log file sync Prod run 25464
05/09
11633.60 10 log file sync RAC C 3node async
12709.00 7 log file sync Prod run 8317
13767.50 2 log file sync RAC C 3node
noasync initial
83.00 1 rdbms ipc reply RAC C 1node async
358.00 1 rdbms ipc reply RAC C 3node async
449.00 1 rdbms ipc reply RAC C 3node
async cert client
92.50 8 row cache lock RAC C 1node async
96.00 1 row cache lock RAC C 3node
noasync initial
307.29 7 row cache lock RAC C 3node async
320.17 6 row cache lock Prod run amcc
343.45 11 row cache lock Prod run 8317
419.29 7 row cache lock Prod run 25464
05/09
616.82 11 row cache lock RAC C 3node
async cert client
1155.74 19 row cache lock Prod run 14892
---------------------------------
Do you Yahoo!?
Yahoo! Mail - Helps protect you from nasty viruses.
--
//www.freelists.org/webpage/oracle-l
Other related posts:
