[mylvmbackup] Ruby version of mylvmbackup: lvm-backup

I have created a Ruby version of mylvmbackup, preserving about
everything except for the system log. I also added:
 1) Support for saving the backups to Amazon S3 (and breaking up the
massive tar into pieces)
 2) Some infrastructure for restoring.
 3) Binary log support so one can do incremental backups / restores.
 4) Very flexible mechanisms for creating and disposing of backup files.

This is licensed under same conditions as the original mylvmbackup (GNU
Generala Public License).

Attached is an installable gem. You can install via gem if you detach
the .gem into current dir:
 > gem install --local lvmbackup

I assume users of this  are familiar with ruby, gems and the ruby world
in general. If not, I can try and provide some better doc...

I've also attached the current source, for your perusal.. A few notes:

1) Needs some more real world testing. I am not currently using it in
production, as I've delayed my project to use this on Amazon EC2.
I need some real world testers for the current code. Especially in
regards to restoring...

2) It is MySQL specific.

3) I did not retain all the command line overrides of the config file
settings. I could, just seemed like a bit of work.

To extract, do a "tar xzvf lvm-source.tgz".

I also have applied for a RubyForge project for this (lvmbackup).

See the README in the source (and below) for lots of documentation (at
least a lot more than I had before). Also a good place to look is to
look at the ./config/lvm-backup.yml. I tried to comment that decently.

You can either use that file as is(but probably will not do what you
want) or edit it, save it
as /etc/lvm-backup.yml and then run the script as:

 >lvm-backup --help
 >lvm-backup --config-file WHERE_YOU_PUT_THE_EDITTED_LVM-BACKUP.YML
 >lvm-backup   # WILL DO A SNAPSHOT and BACKUP
 >...

A "full restore" is mostly working, albeit needs a bit more testing. An
"incremental" restore from binary logs still needs to be done, but I
think it is pretty simple. The "incremental backup" using binary logs is
working (i.e., I tested this a few weeks ago and all seemed to be
working). But the whole process of restoring a crashed/corrupted server
is a bit site dependent, I think. I just hope lvm-backup can provide the
basics...

Have fun, and I could really use a real-world tester for this. I'd be
happy to fix bugs on it...
I got sidetracked a week or so on another project, and left a few loose
ends dealing with restoring...

README
------
Logical Volume Manager Backup utility for MySQL
Takes snapshots of MySQL volumes mounted via LVM

Copyright (c) 2008 by Jim M Wilson, Lenz Grimmer
open source, see LICENSE file for details

Based on the Perl script mylvmbackup by Lenz Grimmer. Added features
include:
  1) User specified commands to tar/rsync/whatever the backup files
  2) Amazon S3 support for saving backups
  3) A full restore function from the snapshot
  4) Binary log (incremental save) functions
  5) TBD - restore from binary logs
-----

Compatibility with mylvmbackup
------------------------------
In general, all functionality should be there except for:
  1) syslog facility. You can use the ruby logger and set a file name
instead.
  2) command line override of config file options. I only carried over a
few things. 

Config file, YAML, ERB
-------------------

Most options are controlled by a configuration file in YAML format. If
you are not familiar
with YAML, see http://www.yaml.org/spec/current.html. In general, use
the supplied
config/lvm-backup.yml as a guide, noting that the indentation level is
VERY IMPORTANT.

A powerful feature is the ability to perform parameter substitution at
run time(using
ERB = Embedded Ruby). For instance, you might have a config file with
these
options for where mount points are and a command to save binary logs via
tar:

#
# File system specific options
#
:fs:
  :backup_mount:    /mnt/vg/backup   # Path of mounted backup snapshot
  :mysql_mount:     /mnt/vg/mysql    # Path of mounted mysql volume

# The command (shell script) used to backup binary logs
# You may use ERB style substitution within any value:
#   option(category, name) - for config value defined within here
#   file -  a temp file created to contain a tar
#   datadir -  the path to the binary logs
#   bin_logs -  a file containing the list of binary log file names (for
tar)
# within the command string
:bin_log:
  :save: /bin/tar cvpzf <%= file %> -C <%= option(:fs, :mysql_mount) %>
--files-from <%= bin_logs %>

Note that the actual file name (in timestamped form) is substituted, as
well as the
actual location of the files (mysql_mount) from the configuration, and
the logs themselves.
The actual command executed would look something like:

  /bin/tar
cvpzf /tmp/bin-log.XXeEGUzX/osprey-2008-04-23-09-07-03-mysql-bin-98.tgz 
    -C /mnt/vg/mysql  --files-from /tmp/bin-log-list.27651.0


Operations
----------
At present, this script can perform the following functions.

  1) snapshot - Do a LVM snapshot of the MySQL datadir.
  2) logs - Flush the MySQL logs, capture the current log position and
backup all 
            log files via user specified command.
  3) restore - Completely restore MySQL datadir from last snapshot
  4) restore logs - TBD, but will replay binary logs.

Where and how are backups stored/disposed of??.
---------------------
The [:fs][:backup_dir] option is very important in controlling the
disposition of files
created. If it is not provided, then lvm-backup will create a temporary
directory, create
all files there and either copy the file to S3 (if enabled) or via a
user provided
command [:misc][:copy].

Otherwise you may provide a value and backups will be created in that
directory and you 
can optionally copy to S3 or do nothing.

For instance, for binary-logs as an example, here is the result of the
user save script 
when :backup_dir is provided:
  /bin/tar
cvpzf /var/backups/mysql/osprey-2008-04-23-09-13-44-mysql-bin-98.tgz 
   -C /mnt/vg/mysql  --files-from /tmp/bin-log-list.27685.0

And here it is if :backup_dir not provided and a temp is used along with
a user copy.
  /bin/tar
cvpzf /tmp/bin-log.XXBQtL26/osprey-2008-04-23-09-18-01-mysql-bin-98.tgz 
   -C /mnt/vg/mysql  --files-from /tmp/bin-log-list.27702.0
  cp /tmp/bin-log.XXBQtL26/osprey-2008-04-23-09-18-01-mysql-bin-98.tgz
YOUR_DESTINATION/osprey-2008-04-23-09-18-01-mysql-bin-98.tgz

Amazon S3
---------
See the .yml config file for details. You can provide credentials from
either a
s3cfg file (this assumes you have the s3 command line tools installed),
or you
may provide directly. You specify a bucket name to save the files in.

:s3:
  :credentials:  ~/.s3cfg                      # get credentials from a
file (assumes a s3cfg format)
  :bucket:       YOUR_AMAZON_BUCKET           # The Amazon s3 Bucket to
copy backup's to/from
#  :id:           YOUR_AMAZON_ACCESS_KEY       # or specify them here
#  :secret:       YOUR_AMAZON_SECRET_KEY

Backup file disposition
-----------------------
Rather than hard code a specific mechanism to tar/rsync files, via the
config file you merely
specify a command to run (see above ERB example). This way you can tar
any way you like, or
use rsync, or scp, or whatever. The actual data files are plugged into
the command string at
run time.

File formats
------------
All files created will use file names in the following format:
  HOST-TIME-NAME.EXT
where 
 1) HOST is the hostname you are running on
 2) TIME is the timestamp formatted string as controlled by
(:misc[:date_format])
 3) NAME is the user defined name of the snapshot(:backup[:prefix]) or
bin-log([:mysql][:bin_log])
 4) EXT is user provided ([:backup][:save]) or ([:bin_log][:ext])

To use
------
  If installed as a gem
     export RUBYOPT=rubygems
     lvm-backup --config YOUR_CONFIG_FILE

----

For developers: 

    Working on a RubyForge project...

-------

Enjoy!!

Jim Wilson
Colorado Springs, CO 
jim@xxxxxxxxxxxxxx
IM:jwilsonsprings
Registered Linux User # 302849

Other related posts: