Good evening,
This morning it was noticed that our git repo (and several services leveraging
it)
were operating oddly with an array of errors. (gerrit, cgit, etc)
I did some investigation (and some git fsck's) on our haiku.git repo and found
a corrupted ~300MiB git .pack file. We were able to recover the repository's
functionality within 30 minutes or so with PulkoMandy's git expertise.
We are not 100% sure of the cause at this time, but my "gut" feeling is we have
recently heard some rumblings of users seeing occasional git disconnects from
Gerrit. There is a java error in Gerrit paired with the event. This could be
related.
As an initial step to help mitigate this, I've updated us to the latest Gerrit
version
and will continue to monitor the situation. (running git fsck's to confirm repo
sanity)
If you're curious about what steps were needed to repair our git repos, I've
documented them here:
https://github.com/haiku/infrastructure/blob/master/docs/git-pack-corruption.md ;
(https://github.com/haiku/infrastructure/blob/master/docs/git-pack-corruption.md)
It was pointed out to me this evening that someone attempting to perform a push
was receiving a permission denied error. This was a mistake on my part.
(during the repair, I wrote files as root to our bare repo and never fixed the
permissions)
I also did some git housekeeping and garbage collection.
Checkouts feel a bit faster now.
Let me know if anyone sees any additional problems. (or even if you see any
"odd"
behavior)
Thanks!
-- Alex
