>>According to Oracle Support, we were "unlucky".
Wow, that is very scientific ;-)
Riyaj "Re-yas" Shamsudeen Certified Oracle DBA (ver 7.0 - 9i) Allocation & Assortment planning systems JCPenney
-----Original Message----- From: Binley Lim [mailto:Binley.Lim@xxxxxxxxxx]
Care to explain why running out of disk space crashes a database, requiring recovery?
Good question. I don't really know the answer. According to Oracle Support, we were "unlucky". I've had filesystems fill up before and never had this problem, and this same filesystem even filled up again later in the day on Sunday after I finished recovery (yes, I'm an idiot for not taking preventative measures after I got the database back up, but I was exhausted by that point and not thinking clearly) but did not have corruption the 2nd time. Here are the errors from the logs:
When the file system first filled up:
Sat Apr 15 06:32:13 2006 ARC1: Beginning to archive log 1 thread 1 sequence 11021 Creating archive destination LOG_ARCHIVE_DEST_1: '/baan4/oraarc/log_-1917883320_11021_1.arc' ARC1: I/O error 19502 archiving log 1 to '/baan4/oraarc/log_-1917883320_11021_1.arc' Sat Apr 15 06:32:18 2006 Errors in file /baan4/admin/bdump/baan4_arc1_8405102.trc: ORA-19502: write error on file "/baan4/oraarc/log_-1917883320_11021_1.arc", blockno 192513 (blocksize=512) ORA-27063: skgfospo: number of bytes read/written is incorrect IBM AIX RISC System/6000 Error: 28: No space left on device
Then the errors when the instance was terminated by LGWR a few minutes later:
Sat Apr 15 06:35:03 2006 Errors in file /baan4/admin/bdump/baan4_lgwr_1671218.trc: ORA-00340: IO error processing online log 2 of thread 1 ORA-00345: redo log write error block 158327 count 323 ORA-00312: online log 2 thread 1: '/baan4/oralog/redo02/redo02b.log' ORA-27063: skgfospo: number of bytes read/written is incorrect IBM AIX RISC System/6000 Error: 28: No space left on device Additional information: -1 Additional information: 165376 ORA-00345: redo log write error block 158327 count 323 ORA-00312: online log 2 thread 1: '/baan4/oralog/redo02/redo02a.log' ORA-27063: skgfospo: number of bytes read/written is incorrect IBM AIX RISC System/6000 Error: 28: No space left on device Additional information: -1 Additional information: 165376 Sat Apr 15 06:35:03 2006 LGWR: terminating instance due to error 340 Instance terminated by LGWR, pid = 1671218
And, then the errors that occurred after restarting the database:
Sat Apr 15 07:51:11 2006 Errors in file /baan4/admin/bdump/baan4_smon_7880942.trc: ORA-00604: error occurred at recursive SQL level 1 ORA-00607: Internal error occurred while making a change to a data block ORA-00600: internal error code, arguments: , , , , , , , 
According to Metalink # 39282.1, the ORA-600  error means "A mismatch has been detected between Redo records and Rollback (Undo) records.", and the solution is point-in-time recovery to before the error occurred.
Why this happened? I don't know. Any ideas?
Privileged/Confidential Information may be contained in this message or attachments hereto. Please advise immediately if you or your employer do not consent to Internet email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of this company shall be understood as neither given nor endorsed by it.
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If the reader of this message is not the intended recipient, you are hereby notified that your access is unauthorized, and any review, dissemination, distribution or copying of this message including any attachments is strictly prohibited. If you are not the intended recipient, please contact the sender and delete the material from any computer.