September 3, 2019 1:54 PM, "Alexander von Gluck IV" <kallisti5@xxxxxxxxxxx
(mailto:kallisti5@xxxxxxxxxxx?to=%22Alexander%20von%20Gluck%20IV%22%20<kallisti5@xxxxxxxxxxx>)>
wrote:
Good afternoon!
Just a heads up we will be attempting our server switch this evening. (4-5
Hours away CDT time)
Pretty much everything will be impacted in some way.
The following changes are happening:
1) Docker compose -> Docker swarm (single node for now) (more flexible
long-term and we get secret storage)
2) online.net -> DigitalOcean
(solve the horrible randomly 1 MiB/s - 130 MiB/s Managed iSCSI service that
kill our services)
block storage is 300MiB/s+ throughout all testing.
3) France -> Amsterdam (Still within the EU, woot!)
4) nightly images Our server -> Wasabi s3 (cheaper storage)
This is going to be one of the more risky changes we've done.
The rollback plan is switching back to online.net
Wish us luck! I'll keep everyone updated in this thread.
-- Alex
Well! I sent this yesterday, but freelists has been down over the last 24 hours.
The server migration is complete. We saw the core services down for roughly
15 minutes, and various outages and breakages for ~ 2 hours while I fixed the
rough edges. (we also moved to docker swarm from docker compose)
Everything should be functional with the following known issues:
* Nightly images behind. I'm moving these to https://wasabi.com ;
(https://wasabi.com/) to save on costs... should be synched up soon.
* Builders need updated to point new images wasabi
We still have various build system issues that need worked through, but things
are
running *MUCH* better at DigitalOcean vs online.net.
Keep me posted if anyone sees any other major issues.
-- Alex
