Re: [foxboro] /var full, but none of the usual offenders

Thanks Jack.
We trim some log files automatically, but don't restart the processes.  I
suspect we'd had the AW running quite a while and that cron, Informix or the
service that manages the wtmp/wtmpx/utmp/utmpx files was keeping the space
blocked even though we'd trimmed the file.

Cheers,

Kevin

On Thu, Nov 5, 2009 at 11:12 PM, <Jack.Easley@xxxxxxxxxxxx> wrote:

> I'm vague on this, as it was years ago, but I did use du -sk * /var,
> then work downwards to identify the system "hugelogfile" as mentioned
> earlier. The correction process was to stop cron (required), then cat
> /dev/null > "hugelogfile", then restart cron. A reboot was not required
> using this method, but the cron stop/start was critical.
>
> I also remember that this logging of cron was enabled by a file in
> /etc/defaults. Changing the setting to cron log setting to NO stopped
> the growth of this huge file forevermore.
>
> Your situation may not have involved this common problem with the cron
> log file (don't remember exact name). If so, disregard this.
>
> Jack Easley
> Sr. I&C Technician
> Luminant Power, Martin Lake Plant
> Phone 903.836.6241
> jack.easley@xxxxxxxxxxxx
> -----Original Message-----
> From: foxboro-bounce@xxxxxxxxxxxxx [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> On Behalf Of Kevin Fitzgerrell
> Sent: Wednesday, November 04, 2009 7:44 PM
> To: foxboro@xxxxxxxxxxxxx
> Subject: Re: [foxboro] /var full, but none of the usual offenders
>
> Reboot fixed the problem.  Thanks Frits, it was likely a process holding
> a
> file open by inode that we had cleaned at some point.
> Regards,
>
> Kevin
>
> On Thu, Nov 5, 2009 at 10:27 AM, Fitzgerrell, Kevin <
> Kevin_Fitzgerrell@xxxxxxx> wrote:
>
> > Thanks Frits.
> >
> > It does not appear to be the online.log file this time, however we
> will try
> > a reboot and see if Informix (or another process) is keeping something
> open.
> >
> > AW301F# du -sk *
> > 82      adm
> > 1       audit
> > 2       crash
> > 11      cron
> > 16      log
> > 1       lost+found
> > 4       lp
> > 2       mail
> > 1       news
> > 1       nis
> > 1       opt
> > 1       preserve
> > 1869    sadm
> > 2       saf
> > 50      spool
> > 4       statmon
> > 49      tmp
> > 10      uucp
> > 4       yp
> >
> > Regards,
> >
> > Kevin
> >
> > -----Original Message-----
> > From: foxboro-bounce@xxxxxxxxxxxxx
> [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> > On Behalf Of Schouten, Frits JF
> > Sent: Thursday, November 05, 2009 10:17 AM
> > To: 'foxboro@xxxxxxxxxxxxx'
> > Subject: Re: [foxboro] /var full, but none of the usual offenders
> >
> > Hi Kevin,
> > have you tried du -sk * in /var?
> >
> > btw. the most likely contender for filling up /var is
> /var/tmp/online.log.
> > Deleting it will not clear it. Informix is keeping track of it by
> inode.
> > A reboot or stop/start Informix is required to clear the space used up
> by
> > online.log.
> >
> > Cheers,
> > Frits.
> >
> > -----Original Message-----
> > From: foxboro-bounce@xxxxxxxxxxxxx
> [mailto:foxboro-bounce@xxxxxxxxxxxxx]
> > On Behalf Of Kevin Fitzgerrell
> > Sent: Thursday, 5 November 2009 1:58 PM
> > To: foxboro@xxxxxxxxxxxxx
> > Subject: [foxboro] /var full, but none of the usual offenders
> >
> > Hi all,
> > I've got an odd problem on one of my E box AWs.
> >
> > My /var partition is full (nothing unusual there), but we've gone
> through
> > and can't find any of the usual offenders.
> /var/adm/utmp,utmpx,wtmp,wtmpx
> > are all small.  The cron and mail logs are also small, and there
> aren't any
> > print jobs in the spool or lp directories.  Nothing unusual in
> /var/tmp.
> >
> > Checking for large files gets me very little:
> >
> > AW301F# df -kFufs
> >
> > Filesystem            kbytes    used   avail capacity  Mounted on
> > /dev/md/dsk/d0         68571   27293   34428    45%    /
> > /dev/md/dsk/d4        953015  398414  459301    47%    /usr
> > /dev/md/dsk/d2         42655   42128       0   100%    /var
> > /dev/md/dsk/d5       15281341 2297913 11455298    17%    /opt
> > AW301F#cd /var
> > AW301F# find . -fstype urfs -prune -o -type f -a -size +20 -ls
> >
> >  555 1416 -rw-r--r--  1 root     other     1436513 Nov 13  2001
> > ./sadm/install/contents
> > 18060   18 -rw-r--r--  1 root     other       18376 Jul 31  2001
> > ./sadm/pkg/SUNWcsu/save/105256-01/undo.Z
> >  243   21 -rw-r--r--  1 root     root        20571 Nov 13  2001
> > ./sadm/pkg/SUNWcsu/pkginfo
> >  470   11 -rw-r--r--  1 root     root        10421 Nov 13  2001
> > ./sadm/pkg/SUNWcsr/pkginfo
> >  362   29 -rw-r--r--  1 root     other       29403 Sep 30  1997
> > ./sadm/pkg/SUNWidecr/install/postinstall
> >  363   29 -rw-r--r--  1 root     other       29403 Sep 30  1997
> > ./sadm/pkg/SUNWidecr/install/preremove
> >  437   19 -rw-r--r--  1 root     root        19024 Jul 31  2001
> > ./sadm/system/admin/.clustertoc
> > 18058   11 -rw-r--r--  1 root     root        11013 Jul 31  2001
> > ./sadm/system/logs/install_log
> > 18080   13 -rw-r--r--  1 root     root        13252 Jul 31  2001
> > ./sadm/system/logs/finish.log
> >  353   24 -rw-r--r--  1 root     other       23623 Nov 13  2001
> > ./sadm/patch/.patchDB
> > 18271   11 -rw-r--r--  1 root     other       10897 Oct 23 14:51
> > ./adm/messages.1
> > 17665   40 -rw-r--r--  1 root     other       40543 Oct  8 18:54
> > ./adm/messages.3
> >  382   55 -rw-r--r--  1 root     sys         55353 Oct 23 14:54
> > ./saf/zsmon/log
> > 18006   19 -rw-rw-rw-  1 root     root        18972 Oct 23 14:54
> ./saf/_log
> >  617   11 -rw-rw----  1 root     informix    10623 Nov  4 23:18
> > ./tmp/sysinfx.lOg
> >
> > I moved the biggest file (./sadm/install/contents) off to /opt and no
> > change
> > in the /var/ partition size.
> >
> > Any thoughts?
> >
> > I'm assuming I've likely got a file system problem.
> >
> > Regards,
> >
> > Kevin
> >
> >
> >
> >
> >
> _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys
> Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here
> at
> > your own risks. Read
> http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:
> http://www.freelists.org/list/foxboro
> > to subscribe:
> mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:
> mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
> >
> >
> > NOTICE - This message and any attached files may contain information
> that
> > is confidential, legally privileged or proprietary. It is intended
> only for
> > use by the intended recipient. If you are not the intended recipient
> or the
> > person responsible for delivering the message to the intended
> recipient, be
> > advised that you have received this message in error. Any
> dissemination,
> > copying, use or re-transmission of this message or attachment, or the
> > disclosure of any information therein, is strictly forbidden.
> BlueScope
> > Steel Limited does not represent or guarantee that this message or
> > attachment is free of errors, virus or interference.
> >
> > If you have received this message in error please notify the sender
> > immediately and delete the message. Any views expressed in this email
> are
> > not necessarily the views of BlueScope Steel Limited.
> >
> >
> >
> _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys
> Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here
> at
> > your own risks. Read
> http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:
> http://www.freelists.org/list/foxboro
> > to subscribe:
> mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:
> mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
> >
> >
> _______________________________________________________________________
> > This mailing list is neither sponsored nor endorsed by Invensys
> Process
> > Systems (formerly The Foxboro Company). Use the info you obtain here
> at
> > your own risks. Read
> http://www.thecassandraproject.org/disclaimer.html
> >
> > foxboro mailing list:
> http://www.freelists.org/list/foxboro
> > to subscribe:
> mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> > to unsubscribe:
> mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
> >
> >
>
>
>
>
> _______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys Process
> Systems (formerly The Foxboro Company). Use the info you obtain here at
> your own risks. Read http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:             http://www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>
> Confidentiality Notice: This email message, including any attachments,
> contains or may contain confidential information intended only for the
> addressee. If you are not an intended recipient of this message, be
> advised that any reading, dissemination, forwarding, printing, copying
> or other use of this message or its attachments is strictly prohibited. If
> you have received this message in error, please notify the sender
> immediately by reply message and delete this email message and any
> attachments from your system.
>
>
> _______________________________________________________________________
> This mailing list is neither sponsored nor endorsed by Invensys Process
> Systems (formerly The Foxboro Company). Use the info you obtain here at
> your own risks. Read http://www.thecassandraproject.org/disclaimer.html
>
> foxboro mailing list:             http://www.freelists.org/list/foxboro
> to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
> to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave
>
>


 
 
_______________________________________________________________________
This mailing list is neither sponsored nor endorsed by Invensys Process
Systems (formerly The Foxboro Company). Use the info you obtain here at
your own risks. Read http://www.thecassandraproject.org/disclaimer.html
 
foxboro mailing list:             http://www.freelists.org/list/foxboro
to subscribe:         mailto:foxboro-request@xxxxxxxxxxxxx?subject=join
to unsubscribe:      mailto:foxboro-request@xxxxxxxxxxxxx?subject=leave

Other related posts: