Re: Swap Leak on Solaris 9 and a posting from last year by Grzegorz Goryszewski

  • From: "Mark Strickland" <strickland.mark@xxxxxxxxx>
  • To: "Hameed, Amir" <Amir.Hameed@xxxxxxxxx>
  • Date: Tue, 13 Feb 2007 15:05:00 -0800

Solved!  This seems to be a Perfect Storm of Stupidness.  On further
investigation, we discovered that ocssd.bin was running from a
10.1.0.3ORACLE_HOME instead of our current
10.1.0.5 ORACLE_HOME.  When we applied the 10.1.0.5 patchset last  October,
we actually did a fresh install into a new ORACLE_HOME.  We failed to
correct /etc/init.d/init.cssd with the new ORACLE_HOME.  The two standby
servers are not RACified and we don't use ASM so we don't actually even need
to run ocssd.bin.  At any rate, running it from the wrong ORACLE_HOME hadn't
caused any problems until yesterday when a Qualys port scan was run on both
servers.  The combination of the port scan and running ocssd.bin from the
wrong ORACLE_HOME somehow caused ocssd.bin to grab 12-GB of swap on one
server and 6-GB on the other server and it was gradually sucking up more
until we ran out of swap on one of the servers this morning and would have
run out of swap on the other server soon.  It didn't show up as used swap,
it actually reduced the total swap size on the servers.  I fixed the
init.cssd on both servers and re-started cssd.bin and we ran another port
scan.  No problem.  The size of the cssd.bin process stayed very low and
total swap size stayed where it should be.  We have no clue what the
interactions really are that would cause this.  If anyone thinks they know,
I'm all eyes.

We had scheduled a Qualys scan to run on the RAC cluster Thursday night, but
it probably would not have caused a problem because we applied the
10.1.0.5patchset in place on those servers and didn't change the
ORACLE_HOME.

Mark


On 2/13/07, Mark Strickland <strickland.mark@xxxxxxxxx> wrote:


New information.  On the logical standby server, the total swap size
plunged yesterday morning from 14-GB to 1.8-GB when  the network admin ran
a Qualys port scan.  On the physical standby server, I don't collect OS
stats, but total swap size (not swap available...TOTAL swap size) was 450-MB
a few minutes ago and the size of the ocssd process was 6-GB.  I just killed
ocssd and total swap size jumped back up to 6.6-GB and seems to be rising
steadily and is now at 6.9-GB.  The network admin just ran another port
scan on the logical standby server and total swap size plunged from 14-GB to
2-GB and seems to be steadily dropping.  At the same time, the size of the
ocssd process jumped from 30-MB to 12-GB.  So, we need to figure out the
relationship between the port scan, swap, and the ocssd daemon.

Thanks for responding.

Mark


On 2/13/07, Hameed, Amir <Amir.Hameed@xxxxxxxxx> wrote:
>
>  Mark,
> We are running Solaris 9 and the OS is patched up with DST patches. We
> have some large enterprise level servers, E20K, E6900 as well as small
> servers, like v490, etc. running Oracle9i and Oracle 10g databases but I
> have not experienced this issue on our servers. I can monitor some servers
> to see if we are leaking swap space.
>
>  ------------------------------
> *From:* oracle-l-bounce@xxxxxxxxxxxxx [mailto:
> oracle-l-bounce@xxxxxxxxxxxxx] *On Behalf Of *Mark Strickland
> *Sent:* Tuesday, February 13, 2007 3:34 PM
> *To:* oracle-l@xxxxxxxxxxxxx
> *Subject:* Swap Leak on Solaris 9 and a posting from last year by
> Grzegorz Goryszewski
>
> This past weekend, we patched our Production Sun servers with Solaris
> patches and the Oracle DST patch.  On two of those servers which host
> standby databases (one server for physical, one server for logical), we are
> seeing swap space gradually drop down to almost nothing from 14-GB.  Just
> had to re-boot one of the servers and the other one is not far behind.  A
> Google search came up with a posting on this list from last year from
> Grzegorz Goryszewski who was experiencing the same thing.  Grzegorz, are you
> monitoring this list still?  Did you find a solution?  Has anyone else seen
> this behavior?
>
> Regards,
> Mark Strickland
> Seattle, WA
>
>

Other related posts:

  1. Home
  2. oracle-l
  3. Archive
  4. 02-2007