X-Git-Url: https://arthur.barton.de/cgi-bin/gitweb.cgi?a=blobdiff_plain;f=README.md;h=9d26917f07ea848d7d622f9d817f5da1567977a3;hb=92499d3cf4f0c697e429e1e344a14cda4c83d958;hp=8187ddfd1c495a58355b8515f40a69b8b5f250be;hpb=07b5ccbf47a8d1084da3ffba4e876d85de847799;p=bup.git diff --git a/README.md b/README.md index 8187ddf..9d26917 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,3 @@ - bup: It backs things up ======================= @@ -72,13 +71,17 @@ Reasons you might want to avoid bup for you, but we don't know why. It is also missing some probably-critical features. - - It requires python >= 2.4, a C compiler, and an installed git version >= - 1.5.3.1. + - It requires python >= 2.5, a C compiler, and an installed git + version >= 1.5.3.1. It also requires par2 if you want fsck to be + able to generate the information needed to recover from some types + of corruption. - It currently only works on Linux, MacOS X >= 10.4, NetBSD, Solaris, or Windows (with Cygwin). Patches to support other platforms are welcome. - + + - Any items in "Things that are stupid" below. + Getting started =============== @@ -89,34 +92,43 @@ From source - Check out the bup source code using git: - git clone git://github.com/apenwarr/bup + git clone git://github.com/bup/bup - - Install the needed python libraries (including the development + - Install the required python libraries (including the development libraries). - On Debian/Ubuntu this is usually sufficient (run as root): + On very recent Debian/Ubuntu versions, this may be sufficient (run + as root): + + apt-get build-dep bup + + Otherwise try this (substitute python2.5-dev or python2.6-dev if + you have an older system): - apt-get install python2.6-dev python-fuse + apt-get install python2.7-dev python-fuse apt-get install python-pyxattr python-pylibacl apt-get install linux-libc-dev - - Substitute python2.5-dev or python2.4-dev if you have an older - system. Alternately, on newer Debian/Ubuntu versions, you can try - this: - - apt-get build-dep bup + apt-get install acl attr + apt-get install python-tornado # optional On CentOS (for CentOS 6, at least), this should be sufficient (run as root): yum groupinstall "Development Tools" - yum install python python-dev + yum install python python-devel yum install fuse-python pyxattr pylibacl yum install perl-Time-HiRes In addition to the default CentOS repositories, you may need to add RPMForge (for fuse-python) and EPEL (for pyxattr and pylibacl). + On Cygwin, install python, make, rsync, and gcc4. + + If you would like to use the optional bup web server on systems + without a tornado package, you may want to try this: + + pip install tornado + - Build the python module and symlinks: make @@ -126,7 +138,7 @@ From source make test (The tests should pass. If they don't pass for you, stop here and send - me an email.) + an email to bup-list@googlegroups.com.) - You can install bup via "make install", and override the default destination with DESTDIR and PREFIX. @@ -150,16 +162,85 @@ Binary packages of bup are known to be built for the following OSes: - pkgsrc (NetBSD, Dragonfly, and others) http://pkgsrc.se/sysutils/bup http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/sysutils/bup/ + - Arch Linux: + https://www.archlinux.org/packages/?sort=&q=bup Using bup --------- - - Try making a local backup as a tar file: + - Get help for any bup command: + + bup help + bup help init + bup help index + bup help save + bup help restore + ... + + - Initialize the default BUP_DIR (~/.bup): + + bup init + + - Make a local backup (-v or -vv will increase the verbosity): + + bup index /etc + bup save -n local-etc /etc + + - Restore a local backup to ./dest: + + bup restore -C ./dest local-etc/latest/etc + ls -l dest/etc + + - Look at how much disk space your backup took: + + du -s ~/.bup + + - Make another backup (which should be mostly identical to the last one; + notice that you don't have to *specify* that this backup is incremental, + it just saves space automatically): + + bup index /etc + bup save -n local-etc /etc + + - Look how little extra space your second backup used (on top of the first): + + du -s ~/.bup + + - Get a list of your previous backups: + + bup ls local-etc + + - Restore your first backup again: + + bup restore -C ./dest-2 local-etc/2013-11-23-11195/etc + + - Make a backup to a remote server (which must already have the 'bup' command + somewhere in the server's PATH (see /etc/profile, etc/environment, + ~/.profile, or ~/.bashrc), and be accessible via ssh. + Make sure to replace SERVERNAME with the actual hostname of your server): + + ssh SERVERNAME bup init + bup index /etc + bup save -r SERVERNAME: -n local-etc /etc + + - Restore a backup from a remote server. (FAIL: unfortunately, + unlike "bup join", "bup restore" does not yet support remote + restores. See both "bup join" and "Things that are stupid" below.) + + - Defend your backups from death rays (OK fine, more likely from the + occasional bad disk block). This writes parity information + (currently via par2) for all of the existing data so that bup may + be able to recover from some amount of repository corruption: + + bup fsck -g + + - Use split/join instead of index/save/restore. Try making a local + backup using tar: tar -cvf - /etc | bup split -n local-etc -vv - - Try restoring your backup tarball: + - Try restoring the tarball: bup join local-etc | tar -tf - @@ -167,29 +248,25 @@ Using bup du -s ~/.bup - - Make another backup (which should be mostly identical to the last one; - notice that you don't have to *specify* that this backup is incremental, - it just saves space automatically): + - Make another tar backup: tar -cvf - /etc | bup split -n local-etc -vv - - Look how little extra space your second backup used on top of the first: + - Look at how little extra space your second backup used on top of + the first: du -s ~/.bup - - Restore your old backup again (the ~1 is git notation for "one older than - the most recent"): + - Restore the first tar backup again (the ~1 is git notation for "one + older than the most recent"): bup join local-etc~1 | tar -tf - - - Get a list of your previous backups: + - Get a list of your previous split-based backups: GIT_DIR=~/.bup git log local-etc - - Make a backup on a remote server (which must already have the 'bup' command - somewhere in the server's PATH (see /etc/profile, etc/environment, - ~/.profile, or ~/.bashrc), and be accessible via ssh. - Make sure to replace SERVERNAME with the actual hostname of your server): + - Make a backup on a remote server: tar -cvf - /etc | bup split -r SERVERNAME: -n local-etc -vv @@ -197,22 +274,6 @@ Using bup bup join -r SERVERNAME: local-etc | tar -tf - - - Try using the new (slightly experimental) 'bup index' and 'bup save' - style backups, which bypass 'tar' but have some missing features (see - "Things that are stupid" below): - - bup index -uv /etc - bup save -n local-etc /etc - - - Do it again and see how fast an incremental backup can be: - - bup index -uv /etc - bup save -n local-etc /etc - - (You can also use the "-r SERVERNAME:" option to 'bup save', just like - with 'bup split' and 'bup join'. The index itself is always local, - so you don't need -r there.) - That's all there is to it! @@ -254,14 +315,36 @@ Notes on NetBSD/pkgsrc traversal code ("fts") in NetBSD's libc will interpret this as a cycle and error out, so "ls -R" and "find" will not work. - - It is not clear if extended attribute and POSIX acl support does - anything useful. + - There is no support for ACLs. If/when some entrprising person + fixes this, adjust t/compare-trees. + + +Notes on Cygwin +--------------- + + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust t/compare-trees. + + - In t/test.sh, two tests have been disabled. These tests check to + see that repeated saves produce identical trees and that an + intervening index doesn't change the SHA1. Apparently Cygwin has + some unusual behaviors with respect to access times (that probably + warrant further investigation). Possibly related: + http://cygwin.com/ml/cygwin/2007-06/msg00436.html + + +Notes on OS X +------------- + + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust t/compare-trees. How it works ============ Basic storage: +-------------- bup stores its data in a git-formatted repository. Unfortunately, git itself doesn't actually behave very well for bup's use case (huge numbers of @@ -272,8 +355,8 @@ python. Basically, 'bup split' reads the data on stdin (or from files specified on the command line), breaks it into chunks using a rolling checksum (similar to -rsync), and saves those chunks into a new git packfile. There is one git -packfile per backup. +rsync), and saves those chunks into a new git packfile. There is at least one +git packfile per backup. When deciding whether to write a particular chunk into the new packfile, bup first checks all the other packfiles that exist to see if they already have that @@ -297,6 +380,7 @@ that tree, respectively, to stdout. You can use this to construct your own scripts that do something with those values. The bup index: +-------------- 'bup index' walks through your filesystem and updates a file (whose name is, by default, ~/.bup/bupindex) to contain the name, attributes, and an @@ -319,21 +403,50 @@ a lot of files have changed. Things that are stupid for now but which we'll fix later --------------------------------------------------------- +======================================================== Help with any of these problems, or others, is very welcome. Join the mailing list (see below) if you'd like to help. + - 'bup restore' can't pull directly from a remote server. + + So in one sense "save -r" is a dead-end right now. Obviously you + can use "ssh SERVER bup restore -C ./dest..." to create a tree you + can transfer elsewhere via rsync/tar/whatever, but that's *lame*. + + Until we fix it, you may be able to mount the remote BUP_DIR via + sshfs and then restore "normally", though that hasn't been + officially tested. + - 'bup save' and 'bup restore' have immature metadata support. On the plus side, they actually do have support now, but it's new, - and not remotely as well tested as tar/rsync/whatever's. If you'd - like to help test, please do; something like 'rsync -niaHAX src/ - restore/' may be useful on that front. + and not remotely as well tested as tar/rsync/whatever's. However, + you have to start somewhere, and as of 0.25, we think it's ready + for more general use. Please let us know if you have any trouble. + + Also, if any strip or graft-style options are specified to 'bup + save', then no metadata will be written for the root directory. + That's obviously less than ideal. - In addition, at the moment, if any strip or graft-style options - are specified to 'bup save', then no metadata will be written for - the root directory. That's obviously less than ideal. + - bup is overly optimistic about mmap. Right now bup just assumes + that it can mmap as large a block as it likes, and that mmap will + never fail. Yeah, right... If nothing else, this has failed on + 32-bit architectures (and 31-bit is even worse -- looking at you, + s390). + + To fix this, we might just implement a FakeMmap[1] class that uses + normal file IO and handles all of the mmap methods[2] that bup + actually calls. Then we'd swap in one of those whenever mmap + fails. + + This would also require implementing some of the methods needed to + support "[]" array access, probably at a minimum __getitem__, + __setitem__, and __setslice__ [3]. + + [1] http://comments.gmane.org/gmane.comp.sysutils.backup.bup/613 + [2] http://docs.python.org/2/library/mmap.html + [3] http://docs.python.org/2/reference/datamodel.html#emulating-container-types - 'bup index' is slower than it should be. @@ -350,7 +463,7 @@ mailing list (see below) if you'd like to help. give the continuous-backup process a really low CPU and I/O priority so you wouldn't even know it was running. - - bup currently has no features that prune away *old* backups. + - bup currently has no way to prune *old* backups. Because of the way the packfile system works, backups become "entangled" in weird ways and it's not actually possible to delete one pack @@ -362,6 +475,9 @@ mailing list (see below) if you'd like to help. We'll have to do it in a totally different way. There are lots of options. For now: make sure you've got lots of disk space :) + Until we fix this, one possible workaround is to just start a new + BUP_DIR occasionally, i.e. bup-2013-10, bup-2013-11... + - bup has never been tested on anything but Linux, MacOS, and Windows+Cygwin. There's nothing that makes it *inherently* non-portable, though, so @@ -371,7 +487,7 @@ mailing list (see below) if you'd like to help. - bup needs better documentation. - According to a recent article about git in Linux Weekly News + According to a recent article about bup in Linux Weekly News (https://lwn.net/Articles/380983/), "it's a bit short on examples and a user guide would be nice." Documentation is the sort of thing that will never be great unless someone from outside contributes it (since @@ -391,7 +507,7 @@ mailing list (see below) if you'd like to help. More Documentation ------------------- +================== bup has an extensive set of man pages. Try using 'bup help' to get started, or use 'bup help SUBCOMMAND' for any bup subcommand (like split, @@ -415,7 +531,7 @@ and you can subscribe by sending a message to: bup-list+subscribe@googlegroups.com -Please see ./HACKING for +Please see ./HACKING for additional information, i.e. how to submit patches (hint - no pull requests), how we handle branches, etc.