RE: Regular restore tests
- From: "Herring, David" <HerringD@xxxxxxx>
- To: "schneider@xxxxxxxxxxxxxx" <schneider@xxxxxxxxxxxxxx>, John Hallas <john.hallas@xxxxxxxxxxxxxxxxxx>
- Date: Wed, 26 Apr 2017 15:20:12 +0000
Jeremy, that's plain awesome! I'm sure all that effort also reduced human
stress. If they know how to do those procedures, those procedures are well
documented and reliable, then when the time comes to actually perform any of
them in critical situations DBAs should be less afraid of the situation.
From: oracle-l-bounce@xxxxxxxxxxxxx [mailto:oracle-l-bounce@xxxxxxxxxxxxx] On
Behalf Of Jeremy Schneider
Sent: Thursday, April 13, 2017 11:19 AM
To: John Hallas <john.hallas@xxxxxxxxxxxxxxxxxx>
Cc: jack@xxxxxxxxxxxx; Tim Gorman <tim.evdbt@xxxxxxxxx>;
angelolistas@xxxxxxxxx; Andrew Kerber <andrew.kerber@xxxxxxxxx>; ORACLE-L
Subject: Regular restore tests
On Thu, 13 Apr 2017 09:46:37 +0100 John Hallas wrote:
As an aside we have a server (well 2 actually - one HPUX and one
Linux) that we use for backup proving.
We have a couple of Oracle homes on each and we recover the backups
from our Commvault repository onto these servers. If we have an
18.104.22.168 database we will recover into whichever OH is installed - it
might be 22.214.171.124 or 126.96.36.199. and we do not run any upgrade/downgrade
scripts All we are testing is that the database can be restored and
that the backup is fully functional
In my most recent position before this current freelancing spell, I helped
architect our restore testing process. Every week we randomly picked one of our
several hundred production web applications and did a full out-of-place restore
of both the database and application tiers. We then logged into the restored
application and pulled screenshots showing dates of recent entries (but no
sensitive data), proving the date of the restore. We stored the screenshots as
part of the evidence on our ISO & SAS certifications. I found that auditors
really liked these screenshots too - proving the restore visually without lots
of explanation, unlike log or terminal captures.
One of the counterintuitive principles in devops is that when something is
painful you should consider doing it more instead of less. Because we did these
restores weekly, we got very skilled and fast and reliable in running them. The
restores were also shared across the DBA team so that we all acquired the
skills. Eventually, every step was either automated (by code in svn) or
cut-and-pasted from a change-controlled wiki.
I was really proud of what we did on that operations team!
Ok, I guess I just got so impressed with the size of a 64-bit value that
I was overwhelmed. Consider, for example:
for (i = 1; i != 0; i++);
Now in theory this will count each possible number, but in practice the
machine will die long before it ever finishes.
- George Anzinger on linux-kernel
Other related posts: