Re: [foxboro] /var full, but none of the usual offenders

I'm vague on this, as it was years ago, but I did use du -sk * /var,
then work downwards to identify the system "hugelogfile" as mentioned
earlier. The correction process was to stop cron (required), then cat
/dev/null > "hugelogfile", then restart cron. A reboot was not required
using this method, but the cron stop/start was critical.

I also remember that this logging of cron was enabled by a file in
/etc/defaults. Changing the setting to cron log setting to NO stopped
the growth of this huge file forevermore.

Your situation may not have involved this common problem with the cron
log file (don't remember exact name). If so, disregard this.

Jack Easley
Sr. I&C Technician
Luminant Power, Martin Lake Plant
Phone 903.836.6241
jack.easley@xxxxxxxxxxxx
-----Original Message-----
From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
On Behalf Of Kevin Fitzgerrell
Sent: Wednesday, November 04, 2009 7:44 PM
To: foxboro@xxxxxxxxxxxxx
Subject: Re: [foxboro] /var full, but none of the usual offenders

Reboot fixed the problem.  Thanks Frits, it was likely a process holding
a
file open by inode that we had cleaned at some point.
Regards,

Kevin

On Thu, Nov 5, 2009 at 10:27 AM, Fitzgerrell, Kevin <
Kevin_Fitzgerrell@xxxxxxx> wrote:

> Thanks Frits.
>
> It does not appear to be the online.log file this time, however we
will try
> a reboot and see if Informix (or another process) is keeping something
open.
>
> AW301F# du -sk *
> 82      adm
> 1       audit
> 2       crash
> 11      cron
> 16      log
> 1       lost+found
> 4       lp
> 2       mail
> 1       news
> 1       nis
> 1       opt
> 1       preserve
> 1869    sadm
> 2       saf
> 50      spool
> 4       statmon
> 49      tmp
> 10      uucp
> 4       yp
>
> Regards,
>
> Kevin
>
> -----Original Message-----
> From: foxboro-bounce@xxxxxxxxxxxxx
[mailto:foxboro-bounce@xxxxxxxxxxxxx]
> On Behalf Of Schouten, Frits JF
> Sent: Thursday, November 05, 2009 10:17 AM
> To: 'foxboro@xxxxxxxxxxxxx'
> Subject: Re: [foxboro] /var full, but none of the usual offenders
>
> Hi Kevin,
> have you tried du -sk * in /var?
>
> btw. the most likely contender for filling up /var is
/var/tmp/online.log.
> Deleting it will not clear it. Informix is keeping track of it by
inode.
> A reboot or stop/start Informix is required to clear the space used up
by
> online.log.
>
> Cheers,
> Frits.
>
> -----Original Message-----
> From: foxboro-bounce@xxxxxxxxxxxxx
[mailto:foxboro-bounce@xxxxxxxxxxxxx]
> On Behalf Of Kevin Fitzgerrell
> Sent: Thursday, 5 November 2009 1:58 PM
> To: foxboro@xxxxxxxxxxxxx
> Subject: [foxboro] /var full, but none of the usual offenders
>
> Hi all,
> I've got an odd problem on one of my E box AWs.
>
> My /var partition is full (nothing unusual there), but we've gone
through
> and can't find any of the usual offenders.
/var/adm/utmp,utmpx,wtmp,wtmpx
> are all small.  The cron and mail logs are also small, and there
aren't any
> print jobs in the spool or lp directories.  Nothing unusual in
/var/tmp.
>
> Checking for large files gets me very little:
>
> AW301F# df -kFufs
>
> Filesystem            kbytes    used   avail capacity  Mounted on
> /dev/md/dsk/d0         68571   27293   34428    45%    /
> /dev/md/dsk/d4        953015  398414  459301    47%    /usr
> /dev/md/dsk/d2         42655   42128       0   100%    /var
> /dev/md/dsk/d5       15281341 2297913 11455298    17%    /opt
> AW301F#cd /var
> AW301F# find . -fstype urfs -prune -o -type f -a -size +20 -ls
>
>  555 1416 -rw-r--r--  1 root     other     1436513 Nov 13  2001
> ./sadm/install/contents
> 18060   18 -rw-r--r--  1 root     other       18376 Jul 31  2001
> ./sadm/pkg/SUNWcsu/save/105256-01/undo.Z
>  243   21 -rw-r--r--  1 root     root        20571 Nov 13  2001
> ./sadm/pkg/SUNWcsu/pkginfo
>  470   11 -rw-r--r--  1 root     root        10421 Nov 13  2001
> ./sadm/pkg/SUNWcsr/pkginfo
>  362   29 -rw-r--r--  1 root     other       29403 Sep 30  1997
> ./sadm/pkg/SUNWidecr/install/postinstall
>  363   29 -rw-r--r--  1 root     other       29403 Sep 30  1997
> ./sadm/pkg/SUNWidecr/install/preremove
>  437   19 -rw-r--r--  1 root     root        19024 Jul 31  2001
> ./sadm/system/admin/.clustertoc
> 18058   11 -rw-r--r--  1 root     root        11013 Jul 31  2001
> ./sadm/system/logs/install_log
> 18080   13 -rw-r--r--  1 root     root        13252 Jul 31  2001
> ./sadm/system/logs/finish.log
>  353   24 -rw-r--r--  1 root     other       23623 Nov 13  2001
> ./sadm/patch/.patchDB
> 18271   11 -rw-r--r--  1 root     other       10897 Oct 23 14:51
> ./adm/messages.1
> 17665   40 -rw-r--r--  1 root     other       40543 Oct  8 18:54
> ./adm/messages.3
>  382   55 -rw-r--r--  1 root     sys         55353 Oct 23 14:54
> ./saf/zsmon/log
> 18006   19 -rw-rw-rw-  1 root     root        18972 Oct 23 14:54
./saf/_log
>  617   11 -rw-rw----  1 root     informix    10623 Nov  4 23:18
> ./tmp/sysinfx.lOg
>
> I moved the biggest file (./sadm/install/contents) off to /opt and no
> change
> in the /var/ partition size.
>
> Any thoughts?
>
> I'm assuming I've likely got a file system problem.
>
> Regards,
>
> Kevin
>
>
>
>
>
_______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys
Process
> Systems (formerly The Foxboro Company). Use the info you obtain here
at
> your own risks. Read
http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:
http://www.freelists.org/list/foxboro
> to subscribe:
mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:
mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>
>
>
> NOTICE - This message and any attached files may contain information
that
> is confidential, legally privileged or proprietary. It is intended
only for
> use by the intended recipient. If you are not the intended recipient
or the
> person responsible for delivering the message to the intended
recipient, be
> advised that you have received this message in error. Any
dissemination,
> copying, use or re-transmission of this message or attachment, or the
> disclosure of any information therein, is strictly forbidden.
BlueScope
> Steel Limited does not represent or guarantee that this message or
> attachment is free of errors, virus or interference.
>
> If you have received this message in error please notify the sender
> immediately and delete the message. Any views expressed in this email
are
> not necessarily the views of BlueScope Steel Limited.
>
>
>
_______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys
Process
> Systems (formerly The Foxboro Company). Use the info you obtain here
at
> your own risks. Read
http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:
http://www.freelists.org/list/foxboro
> to subscribe:
mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:
mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>
>
>
_______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys
Process
> Systems (formerly The Foxboro Company). Use the info you obtain here
at
> your own risks. Read
http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:
http://www.freelists.org/list/foxboro
> to subscribe:
mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:
mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>


 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             http://www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
 

Confidentiality Notice: This email message, including any attachments, 
contains or may contain confidential information intended only for the 
addressee. If you are not an intended recipient of this message, be 
advised that any reading, dissemination, forwarding, printing, copying
or other use of this message or its attachments is strictly prohibited. If
you have received this message in error, please notify the sender 
immediately by reply message and delete this email message and any
attachments from your system. 
 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             http://www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave

Other related posts: