[haiku-development] [NOTICE] Unexpected forum downtime - Resolved

  • From: "Alex" <kallisti5@xxxxxxxxxxx>
  • To: haiku@xxxxxxxxxxxxx, haiku-sysadmin@xxxxxxxxxxxxx, haiku-development@xxxxxxxxxxxxx
  • Date: Tue, 15 Mar 2022 00:45:55 +0000

Good evening,

A normal upgrade of discourse failed today. The root cause was:

* A surprise database permissions issue
  * Fixed. Pretty simple
* A well placed surprise Digital Ocean outage accessing logs for k8s containers
  * self-resolved after ~45 minutes.  Digital Ocean support didn't respond
    within 3 hours and I closed it.
* Discourse wasn't serving pages properly due to our discourse container missing
  a new undocumented requirement called "terser" needed to generate assets.

On the bright side, I restored a database backup successfully.  It was luckily
unneeded however.

tldr; discourse didn't have any data loss, and https://discuss.haiku-os.org
is back online and stable after a 3 hour outage.

Sorry for the noise.

Thanks!

 -- Alex

Other related posts:

  • » [haiku-development] [NOTICE] Unexpected forum downtime - Resolved - Alex