Hi John, On 2013-04-27 at 01:17:42 [+0200], John Scipione <jscipione@xxxxxxxxx> wrote: [ ... ] > > Thanks for the update, the above sounds like a decent explanation of > the problem analysis and solution. > > The critical factor here seems to be monitoring. It sounds like we got > lucky that you noticed there was a problem in time, although I assume > that if worse-came-to-worst we would have been able to restore from a > backup. Yes. > Still, it would be nice to setup some sort of email notification on > the server such as Nagios that could warn us (you?) when there is a > problem in the future. At my workplace we use Nagios to monitor > servers and it emails us when a disk is failing, if we are low on hard > drive space or low on memory, or if the network become saturated. I > assume that the server is running some sort of Linux (Suse IIRC) so > Nagios should be a snap to setup, I can lend my expertise here if need > be. We are using something similar to Nagios, actually: collectd. That's where we get the info from that's shown on the server-stats page. Additionally, smartd would send out mails when it encounters disk problems. The problem this time was just that smartd wasn't running anymore. The latter has been fixed by telling systemd to restart it automatically, so next time around we should get warned as soon as the disks themselves notice something fishy. > We can set it up to email a @haiku-os.org email account such as > monitoring@xxxxxxxxxxxx and then we can forward that address to you or > setup a Google account or something. The point of using a > @haiku-os.org email address is just so that we can easily forward the > mails to multiple recipients and if you don't want to get the emails > anymore we can just stop forwarding them. We are already sending out those kind of mails to the mailing list haiku-sysadmin at freelists.org. If you're interested in the admin-related work, please subscribe there (or just look at the archives) and you'll get a better idea about what's going on. cheers, Oliver