[haiku-sysadmin] Re: [NOTICE] Server change *tonight*

  • From: A L <mail@xxxxxxxxxxxxxx>
  • To: haiku-sysadmin@xxxxxxxxxxxxx
  • Date: Sun, 22 Sep 2019 12:58:57 +0200

Hello,

The rsync.haiku-os.org mirror seems to have goon offline since the server move. At least I can no longer sync from it.
Seems to be a CNAME chain too, not illegal, but discouraged since it uses more DNS resources.

     # drill rsync.haiku-os.org
   ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 1434
   ;; flags: qr rd ra ; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
   ;; QUESTION SECTION:
   ;; rsync.haiku-os.org.  IN      A

   ;; ANSWER SECTION:
   rsync.haiku-os.org.     3556    IN CNAME   ingress.haiku-os.org.
   ingress.haiku-os.org.   3556    IN CNAME   limerick.ams3.haiku-os.org.
   limerick.ams3.haiku-os.org.     3556 IN      A       104.248.198.131


Doing a telnet to port 12000 gives no rsync banner as usually either.

//Anders

On 2019-09-04 15:27, Alexander von Gluck IV wrote:

September 3, 2019 1:54 PM, "Alexander von Gluck IV" <kallisti5@xxxxxxxxxxx <mailto:kallisti5@xxxxxxxxxxx?to=%22Alexander%20von%20Gluck%20IV%22%20<kallisti5@xxxxxxxxxxx>>> wrote:

    Good afternoon!

    Just a heads up we will be attempting our server switch this
    evening. (4-5 Hours away CDT time)

    Pretty much everything will be impacted in some way.

    The following changes are happening:

    1) Docker compose -> Docker swarm (single node for now) (more
    flexible long-term and we get secret storage)
    2) online.net -> DigitalOcean
    (solve the horrible randomly 1 MiB/s - 130 MiB/s Managed iSCSI
    service that kill our services)
    block storage is 300MiB/s+ throughout all testing.
    3) France -> Amsterdam (Still within the EU, woot!)
    4) nightly images Our server -> Wasabi s3 (cheaper storage)

    This is going to be one of the more risky changes we've done.
    The rollback plan is switching back to online.net

    Wish us luck! I'll keep everyone updated in this thread.

    -- Alex


Well! I sent this yesterday, but freelists has been down over the last 24 hours.

The server migration is complete. We saw the core services down for roughly
15 minutes, and various outages and breakages for ~ 2 hours while I fixed the
rough edges. (we also moved to docker swarm from docker compose)

Everything should be functional with the following known issues:

* Nightly images behind. I'm moving these to https://wasabi.com <https://wasabi.com/> to save on costs... should be synched up soon.
* Builders need updated to point new images wasabi

We still have various build system issues that need worked through, but things are
running *MUCH* better at DigitalOcean vs online.net.

Keep me posted if anyone sees any other major issues.

-- Alex


Other related posts: