[haiku-development] Re: [Notice] iSCSI at online.net is down
- From: "Alexander von Gluck IV" <kallisti5@xxxxxxxxxxx>
- To: haiku-development@xxxxxxxxxxxxx
- Date: Thu, 07 Mar 2019 18:10:42 +0000
March 7, 2019 1:03 AM, "Adrien Destugues" <pulkomandy@xxxxxxxxx> wrote:
Le 7 mars 2019 03:53:26 GMT+01:00, waddlesplash <waddlesplash@xxxxxxxxx> a
écrit :
All services have now been restored.
Seems that we should probably jump ship though; likely we'll try out
Ikoula. Or anyone with reasonable prices and better service than
Online.net, anyway...
-waddlesplash
Hi,
First of all, I find statements on our status page and some twitter comments
needlessly agressive
("support has still been awesomely unhelpful" on status.haiku-os.org). I wish
we could avoid that
and keep calm in such situations. It's not like we are a company losing
millions of dollar with
every hour of downtime, right?
Fair... however, I think we have the right to be upset with their services to
some degree.
12/18 - We see huge slowdowns of online.net managed iSCSI at certain times and
open a ticket.
12/18 - They ask to reboot our production server into rescue mode.
12/18 - Us: no, it's production.
12/18 - Them: hands are tied, we gotta reboot to troubleshoot.
12/18 - we provide tons of data and spend a lot of time giving them metrics
that it's them
12/18 - They acknowledge the issue and "escalate it to their senior team"
12/18 - 03/19 "They're working on it"
03/06/19 10:20am Major outage of their iSCSI services
03/06/19 10:40am We notice, raise a support case with details
03/06/19 10:45am Them: Did you read our FAQ on how to set it up?
03/06/19 10:50am Its been working, it's down as of 10:20am
03/06/19 10:55am Them: Investigating.
03/06/19 11:20am Us: it's been down for an hour... any updates?
03/06/19 12:20pm Us: Guys? Two hours? Any ETA
03/06/19 01:00pm Them: There's a problem with our service. Working on it via
online.net Slack chat.
03/06/19 01:13pm Them: "We're still waiting", Please do not send us messages
every 2 hours, it's not going to help.
https://twitter.com/kallisti5/status/1103493983538499584
03/06/19 ~02:00pm - 06:00pm Them: "working on it"
03/06/19 ~8:00pm Them: It's fixed
So. We were 100% down for 10 hours with:
* No ETA.
* No confirmation if our data even still existed.
* No information or updates on what was happening.
* Bitchy replies from online.net about me wanting to know basic updates.
* No troubleshooting.
* No solutions.
As someone who worked for support at a hosting company, this is pretty insane.
I'm not expecting white glove service. If you can't provide a basic update in
10 hours during a major outage, you pretty much just don't care. We told them
about
the problem, and it took them 4 hours to even admit there was a problem.
On the status page you used CST time, making it look like these problems
occured mid-day. Local
time, the incident was notified at half past 5, and it was solved at 1AM on
scaleway side as far as
I can tell. So that's about 8 hours of downtime. Not great, but you get what
you pay for.
10 hours. 10:20pm - 8pm CST.
Datacenters are staffed 24x7. Hell.. i'm on-call right now. It's the
business. In the US i've been
through multiple "super cheap" hosting providers. It may take 4 hours for a
response, but at least
you get a response other than "we're working on it".
Are we going to try all the cheap hosting providers and jump ships whenever
there is a problem? As
long aswe are hosting on a single server with no redundancy, we are going to
get hit by hardware
failures. So, either we get multiple servers and make sure we can run with
one down, or we get some
kind of cloud/virtualized hosting so that we don't have to care for it.
Either way, it won't be
cheap.
I actually agree here. I've recommended configurations like this, however money
was always a limiting
factor per the board. We were paying bottom of the barrel for Hetzner, and were
running into
issues like this. We "upgraded" to online.net which was sold to us as more
premium.
We actually pay a premium to online.net for "premium support". With the
experience above i'm not
sure I really want to know what the "non-premium" experience is :-)
I would say 8 hours of downtime is ok if this does not happen again. I don't
know if it is worth
spending money on something more reliable.
Did they at least solve our daily slowness issue at the same time?
Not that we're aware of. We still don't have an update in the tickets if the
outage was solved,
they told me via Slack. If they couldn't solve the speed over 3 months, i'm not
expecting much.
I've documented the history of our infrastructure here at the direction of Ryan:
https://github.com/haiku/infrastructure/blob/master/HISTORY.md
Sorry to be grumpy, but I (as well as waddlesplash) spent 10 hours yesterday
dealing with this crap.
-- Alex
Other related posts:
