Re: 11i RAC question

  • From: Harish Kalra <hkalra27@xxxxxxxxxxx>
  • To: racdba@xxxxxxxxxxxxx
  • Date: Wed, 31 Aug 2005 06:51:10 +0100 (BST)

Hi Amir,
 
Hope this will help you.
To evaluate the performance of your network with Oracle Application, there is 
also a standard test provided by Oracle, which can be used to compare the 
results of two different locations.
You can navigate to Application menu item under System Administrator 
responsibility and you will find Netowrk Test there.
 
The Network Test consists of a latency test and a bandwidth test. Latency is 
the time it takes for a single packet to make a round trip from your client 
side application to the server. The bandwidth test examines the data rate to 
see how many bytes per second your network can transfer from the server to the 
client.
 
Thanks
-Harish Kalra


Anand Rao <panandrao@xxxxxxxxx> wrote:
Amir,

I am probably missing something very fundamental.

You said that there is no delay when you establish a connection to the
RAC node from the middle-tier. I would expect the middle-tier to be
another server running Forms/Reports Server, Web Server or some other
something like that....correct?

The connection hang is being noticed from the PC...with WinRunner....
am i right?

what does the ping or tnsping from the PC tell you? if an sqlplus
connect from the middle-tier works fine, it doesn't have to be that
things will be fine from the PC to the RAC node.

You need to check end-to-end connection speed, latency, etc... so, you
need to check the ping and tnsping from PC to the RAC Node...and just
for the heck of it, check it from PC to the middle-tier servers
also...

what is the listener/tnsnames configuration? have you configured
local_listener, remote_listener parameters *correctly* in init.ora or
spfile? what about service names? check this over and over....you
would be surprised how many people make mistakes in the Net config...

i think there are many Metalink docs to help you with that bit.

The private Gigabit interconnect has nothing to do with this unless
your Cluster configuration has gone horribly wrong...

anand

On 31/08/05, Hameed, Amir wrote:
> Anand,
> The connection latency occurs in RAC configuration but not in the
> single-instance when the very same test is run. The problem is not with
> the network; it is a private gigabit network layer. Also, if the problem
> was with the network then I should also have experienced the same
> latency with my SQL*Plus connections that I was establishing from the
> middle-tier to the backend while my forms connections were hanging. I
> will let you know with the Oracle support's analysis.
> 
> Thanks
> Amir
> -----Original Message-----
> From: racdba-bounce@xxxxxxxxxxxxx [mailto:racdba-bounce@xxxxxxxxxxxxx]
> On Behalf Of Anand Rao
> Sent: Tuesday, August 30, 2005 8:47 PM
> To: racdba@xxxxxxxxxxxxx
> Subject: Re: 11i RAC question
> 
> Hi Amir,
> 
> I would first look at the connection problem.
> 
> If connecting to E-Business Suite 11i from IE takes so much time, then
> there is something else wrong. You need to check your network layer,
> the network hops, the time taken to ping and tnsping to the RAC
> machines?
> 
> check with different packet sizes. Talk to your network administrator
> and get his help for diagnosis.
> 
> need to check the tcp parameters like timeout, keepalive, etc. on both
> client PCs and RAC Servers. You can set TCP.NODELAY=YES in sqlnet.ora
> (on all RAC nodes and PCs) and see the result.
> 
> 2) If you are already using freelist setting that is fine. But, you
> can switch to Locally Managed Tablespaces. It will avoid a lot of data
> dictionary lookups and updates. This should be a priority i
> guess...unless you want to stay with Dictionary Managed tablespaces.
> as i said, if you want to stay with freelists, it is fine but try and
> switch to LMTs.
> 
> 3) Sequence caching depends on whether you want the values ordered or
> not. If you cannot lose a single sequence number, then caching it can
> *potentially* cause loss of a sequence number or many numbers if your
> instance(s) crash.
> 
> talk to you application team, figure out which sequences can afford
> gaps in the sequence. caching can be done depending on how frequently
> they are used in the application. There is no generic number for this.
> 200 is a good starting point.
> 
> 4) Logical IO - can be looked into once your connection problems are
> solved.
> 
> cheers
> anand
> 
> On 31/08/05, Hameed, Amir wrote:
> > Hi Anand,
> > Here are answers to your questions:
> >
> > 1. you need to give us more info on the machine config, environment
> > details, etc.
> > ==>From my first mail note, this is the information that I had
> provided:
> > * Hardware Configuration:
> > ** Servers:
> > RAC Node#1: Sun 15K domain with 12 CPUs, 24 GB RAM
> > RAC Node#2: Sun 15K domain with 12 CPUs, 24 GB RAM
> > These are dedicated nodes for this RAC environment and do not run
> any
> > other instance/application
> >
> > * Interconnect:
> > Three private gigabit interconnects between the two servers
> connected
> > via a switch
> >
> > * Network:
> > The network interface between the middle-tier and the backend tier
> > goes over a gigabit private network.
> >
> > * Storage:
> > EMC DMX 3000
> >
> > * Software configuration
> > Veritas DBE/AC for 9iRAC 3.5/MP3
> > Oracle applications 11.5.9
> > Oracle RDBMS 9.2.0.6 (64-bit) configured with ODM
> > Shared Pool size: 1GB
> > Buffer cache size: 6GB
> > Total SGA size: ~ 7.5 GB
> >
> > 2. The MRBC is set to 8
> >
> > 3. What is the value of 'Ratio of current block defers' and 'global
> > cache defers'?
> > ==> On instance#1:
> > Ratio of current block defers=> 0.8
> > global cache defers=> Total=6147, Per second=20.6, Per
> > trans=122.9
> >
> > On instance#2:
> > Ratio of current block defers=> 0.8
> > global cache defers=> Total=5552, Per second=18.5, Per
> > trans=63.1
> >
> > 4) Are you using ASSM? if not, use it now.
> > ==> I am not using ASSM. All the segments in the database have
> freelist
> > groups=4 and freelists=16. I had asked Steve Adams last year that if I
> > already have multiple freelist groups and freelists defined, do I need
> > to use ASSM and his reply was that there is no need to use ASSM in
> that
> > case. Let me know if you think otherwise.
> >
> > 5) have you checked the statspack reports for logical io problems?
> what
> > objects do you see there? workflow tables, index branch blocks and
> > leaves are most common. What modules are you using? fnd tables will
> have
> > a lot of logical io.
> > Instance#2 has the following objects with high LIO (sorry for the bad
> > formatting):
> >
> > ==>
> > Subobject Obj.
> Logical
> > Owner Tablespace Object Name Name Type
> Reads
> > %Total
> > ---------- ---------- -------------------- ---------- -----
> ------------
> > -------
> > APPLSYS APPLSYSD SYS_IOT_TOP_1343779 INDEX
> 82,992
> > 73.71
> > APPLSYS APPLSYSD SYS_IOT_TOP_1343777 INDEX
> 6,208
> > 5.51
> > PERFSTAT TOOLS STATS$LATCH_CHILDREN INDEX
> 6,160
> > 5.47
> > JTF JTFD JTF_FM_STATUS_ALL TABLE
> 3,040
> > 2.70
> > APPLSYS APPLSYSX FND_PROFILE_OPTIONS_ INDEX
> 2,096
> > 1.86
> >
> -------------------------------------------------------------
> > The first IOT belongs to the AQ$WF_CONTROL queue and that is the one
> > that showed up with all the cache layer waits in the 10046/level 12
> > trace files. Instance#1 reports the following:
> >
> > Subobject Obj.
> Logical
> > Owner Tablespace Object Name Name Type
> Reads
> > %Total
> > ---------- ---------- -------------------- ---------- -----
> ------------
> > -------
> > SYS SYSTEM TYPE$ TABLE
> 1,181,984
> > 25.20
> > SYS SYSTEM USER$ TABLE
> 1,073,904
> > 22.89
> > SYS SYSTEM OBJ$ TABLE
> 735,808
> > 15.68
> > SYS SYSTEM I_OBJ3 INDEX
> 545,648
> > 11.63
> > SYS SYSTEM I_TYPE2 INDEX
> 354,112
> > 7.55
> >
> -------------------------------------------------------------
> >
> > 6) Are you caching sequences?
> > ==> Most of them. I need to check you sequence caching again. What is
> a
> > good caching number (100, 200, etc)?
> >
> > 7) let us know what exactly is your 'performance' issue.
> > ==> The forms-based connections, run by WinRunner, were getting timed
> > out and loosing connections. While this was happening, I tried to open
> a
> > forms-based connection to the core apps and it took me over a minutes
> to
> > finally log into the apps(it normally takes around five to ten
> seconds).
> > At the same time I ran a script from the middle-tier to loop and
> > repeatedly make SQL*Plus connections to the RAC database tier using
> > SQL*Net and the connections were quick and I did not see any latency
> > issue. This issue was not observed during the single-instance testing
> > and we used the same process and scripts in RAC environment that we
> used
> > in non-RAC. I can reproduce this issue any time.
> >
> > Please let me know if you need Statspack reports, 10046 trace files
> from
> > RAC instances.
> >
> > Thank you
> > Amir
> >
> > -----Original Message-----
> > From: racdba-bounce@xxxxxxxxxxxxx [mailto:racdba-bounce@xxxxxxxxxxxxx]
> > On Behalf Of Anand Rao
> > Sent: Tuesday, August 30, 2005 7:40 PM
> > To: racdba@xxxxxxxxxxxxx
> > Subject: Re: 11i RAC question
> >
> > Hi Amir,
> >
> > some more questions..
> >
> > 1) Are you using ASSM? if not, use it now.
> >
> > 2) have you checked the statspack reports for logical io problems?
> > what objects do you see there?
> >
> > workflow tables, index branch blocks and leaves are most common. what
> > modules are you using? fnd tables will have a lot of logical io.
> >
> > 3) Are you caching sequences?
> >
> > there are many serious design issues in Apps 11i which has caused a
> > lot of trouble, so let us know what exactly is your 'performance'
> > issue.
> >
> > just increasing CPU priority will take care of the issue for some
> > time, then you could be back to square one once you run out of CPU
> > cycles. You cannot just keep adding CPUs for that reason alone.
> >
> > anand
> >
> > On 31/08/05, Anand Rao 
wrote:
> > > Increase CPU Priority for LMS. This is definitely the way to go. no
> > > doubts at all.
> > >
> > > you need to give us more info on the machine config, enviornment
> > details, etc.
> > >
> > > I have seen quite a bit of improvement on many RAC systems, mainly
> 11i
> > > with RAC on large E15K clusters and HP clusters when we increase LMS
> > > priority.
> > >
> > > Don't forget to increase LGWR priority. I would always suggest that
> > > all Oracle background processes (especially LGWR, SMON, DBWR) should
> > > run on a higher priority and should not be fighting for CPU with
> other
> > > processes. On RAC, LMON, LMD and LMS priority should be higher than
> > > any other Oracle process. On Solaris, place them on Real Time
> > > priority. Talk to your system admin if you don't understand the
> > > various priorities.
> > >
> > > Don't use a value more than 8 for db_file_multi_block_read_count
> > > UNLESS you are a total datawarehouse or something like that. I have
> > > learnt it the hard way that 8 gives the best respone. 4 is also good
> > > but that depends on your application. I am running my current 4-node
> > > AIX RAC cluster with a value of 4 for MBRC.
> > >
> > > In your statspack,
> > >
> > > What is the value of 'Ratio of current block defers' and 'global
> cache
> > defers'?
> > >
> > > by setting _gc_defer_time you are treating the symptom and not the
> > cause...
> > >
> > > let me know
> > > cheers
> > > anand
> > >
> > >
> > > On 30/08/05, Ravi_Kulkarni@xxxxxxxx wrote:
> > > >
> > > > Hamid,
> > > >
> > > > Curious - what are the values your max_commit_propagation_delay &
> > > > _fairness_threshold ?
> > > > Also, can you pass on the TAR# (mail me offline, if you wish)
> > > >
> > > > Thanks,
> > > > Ravi.
> > > >
> > > > ________________________________
> > > > From: racdba-bounce@xxxxxxxxxxxxx
> > [mailto:racdba-bounce@xxxxxxxxxxxxx] On
> > > > Behalf Of Hameed, Amir
> > > > Sent: Tuesday, August 30, 2005 8:37 AM
> > > > To: racdba@xxxxxxxxxxxxx
> > > > Subject: RE: 11i RAC question
> > > >
> > > >
> > > >
> > > > 9.2.0.6 (64-bit) on Solaris. This question is in continuation of
> the
> > problem
> > > > that I had sent a note out to this DL last Friday. I finally
> opened
> > a TAR
> > > > last night with Oracle as it seemed like an Oracle issue to me.
> > > >
> > > > ________________________________
> > > > From: racdba-bounce@xxxxxxxxxxxxx
> > [mailto:racdba-bounce@xxxxxxxxxxxxx] On
> > > > Behalf Of Sudhi
> > > > Sent: Tuesday, August 30, 2005 9:29 AM
> > > > To: racdba@xxxxxxxxxxxxx
> > > > Subject: RE: 11i RAC question
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Increasing the priority of LMS/LMD processes is a known thing for
> > > > performance. I'm not sure about the "_gc_defer_time". KG or Anand
> > should be
> > > > aware of it.
> > > >
> > > >
> > > >
> > > > BTW what version of RAC ?
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > -Sudhi.
> > > >
> > > >
> > > >
> > > > ________________________________
> > > >
> > > >
> > > > From: racdba-bounce@xxxxxxxxxxxxx
> > [mailto:racdba-bounce@xxxxxxxxxxxxx] On
> > > > Behalf Of Hameed, Amir
> > > > Sent: Tuesday, August 30, 2005 6:55 PM
> > > > To: racdba@xxxxxxxxxxxxx
> > > > Subject: 11i RAC question
> > > >
> > > >
> > > >
> > > > Is there anyone in this DL running 11i with RAC? If yes then can
> you
> > please
> > > > tell me if you had to make the following settings to address some
> > cache
> > > > fusion issues:
> > > >
> > > > + alter system set "_gc_defer_time"=0;
> > > > + Increase the priority of LMS process ( renice -n -p 
> > process> )
> > > >
> > > > I have a TAR opened with Oracle and they are suggesting this
> > solution. I
> > > > want to make sure that the solution is not custom and is a generic
> > one and
> > > > has just not documented. Any feedback will be appreciated.
> > > >
> > > > Thanks
> > > > Amir
> > >
> >
> >
> 
> 
>



                
---------------------------------
 Free antispam, antivirus and 1GB to save all your messages
 Only in Yahoo! Mail: http://in.mail.yahoo.com

Other related posts: