X-Git-Url: https://arthur.barton.de/cgi-bin/gitweb.cgi?a=blobdiff_plain;f=README.md;h=f66dd4ebd7e19b4c61744056c7733a5ef9de4b68;hb=4638bf0e7f7e3b0545968982c445e04498ca654c;hp=765d5062b3dbe8a8767532f28d0ba58b2a3bf59c;hpb=ab6dfa597c32a0a9e8424da7350303eebb36dd7b;p=bup.git diff --git a/README.md b/README.md index 765d506..f66dd4e 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,3 @@ - bup: It backs things up ======================= @@ -72,40 +71,174 @@ Reasons you might want to avoid bup for you, but we don't know why. It is also missing some probably-critical features. - - It requires python >= 2.4, a C compiler, and an installed git version >= - 1.5.3.1. + - It requires python >= 2.5, a C compiler, and an installed git + version >= 1.5.3.1. It also requires par2 if you want fsck to be + able to generate the information needed to recover from some types + of corruption. - - It currently only works on Linux, MacOS X >= 10.4, or Windows (with - Cygwin). Patches to support other platforms are welcome. - + - It currently only works on Linux, MacOS X >= 10.4, + NetBSD, Solaris, or Windows (with Cygwin). Patches to support + other platforms are welcome. + + - Any items in "Things that are stupid" below. + Getting started ---------------- +=============== - - check out the bup source code using git: - - git clone git://github.com/apenwarr/bup - - install the python 2.5 development libraries. On Debian or Ubuntu, this - is: - apt-get install python2.5-dev - - - build the python module and symlinks: +From source +----------- + + - Check out the bup source code using git: + git clone git://github.com/bup/bup + + - Install the required python libraries (including the development + libraries). + + On very recent Debian/Ubuntu versions, this may be sufficient (run + as root): + + apt-get build-dep bup + + Otherwise try this (substitute python2.5-dev or python2.6-dev if + you have an older system): + + apt-get install python2.7-dev python-fuse + apt-get install python-pyxattr python-pylibacl + apt-get install linux-libc-dev + apt-get install acl attr + apt-get install python-tornado # optional + + On CentOS (for CentOS 6, at least), this should be sufficient (run + as root): + + yum groupinstall "Development Tools" + yum install python python-devel + yum install fuse-python pyxattr pylibacl + yum install perl-Time-HiRes + + In addition to the default CentOS repositories, you may need to add + RPMForge (for fuse-python) and EPEL (for pyxattr and pylibacl). + + On Cygwin, install python, make, rsync, and gcc4. + + If you would like to use the optional bup web server on systems + without a tornado package, you may want to try this: + + pip install tornado + + - Build the python module and symlinks: + make - - run the tests: + - Run the tests: make test (The tests should pass. If they don't pass for you, stop here and send - me an email.) - - - Try making a local backup as a tar file: + an email to bup-list@googlegroups.com.) + + - You can install bup via "make install", and override the default + destination with DESTDIR and PREFIX. + + Files are normally installed to "$DESTDIR/$PREFIX" where DESTDIR is + empty by default, and PREFIX is set to /usr. So if you wanted to + install bup to /opt/bup, you might do something like this: + + make install DESTDIR=/opt/bup PREFIX='' + + +From binary packages +-------------------- + +Binary packages of bup are known to be built for the following OSes: + + - Debian: + http://packages.debian.org/search?searchon=names&keywords=bup + - Ubuntu: + http://packages.ubuntu.com/search?searchon=names&keywords=bup + - pkgsrc (NetBSD, Dragonfly, and others) + http://pkgsrc.se/sysutils/bup + http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/sysutils/bup/ + + +Using bup +--------- + + - Get help for any bup command: + + bup help + bup help init + bup help index + bup help save + bup help restore + ... + + - Initialize the default BUP_DIR (~/.bup): + + bup init + + - Make a local backup (-v or -vv will increase the verbosity): + + bup index /etc + bup save -n local-etc /etc + + - Restore a local backup to ./dest: + + bup restore -C ./dest local-etc/latest/etc + ls -l dest/etc + + - Look at how much disk space your backup took: + + du -s ~/.bup + + - Make another backup (which should be mostly identical to the last one; + notice that you don't have to *specify* that this backup is incremental, + it just saves space automatically): + + bup index /etc + bup save -n local-etc /etc + + - Look how little extra space your second backup used (on top of the first): + + du -s ~/.bup + + - Get a list of your previous backups: + + bup ls local-etc + + - Restore your first backup again: + + bup restore -C ./dest-2 local-etc/2013-11-23-11195/etc + + - Make a backup on a remote server (which must already have the 'bup' command + somewhere in the server's PATH (see /etc/profile, etc/environment, + ~/.profile, or ~/.bashrc), and be accessible via ssh. + Make sure to replace SERVERNAME with the actual hostname of your server): + + ssh SERVERNAME bup init + bup index /etc + bup save -r SERVERNAME: -n local-etc /etc + + - Restore a backup from a remote server. (FAIL: unfortunately, + unlike "bup join", "bup restore" does not yet support remote + restores. See both "bup join" and "Things that are stupid" below.) + + - Defend your backups from death rays (OK fine, more likely from the + occasional bad disk block). This writes parity information + (currently via par2) for all of the existing data so that bup may + be able to recover from some amount of repository corruption: + + bup fsck -g + + - Use split/join instead of index/save/restore. Try making a local + backup using tar: tar -cvf - /etc | bup split -n local-etc -vv - - Try restoring your backup tarball: + - Try restoring the tarball: bup join local-etc | tar -tf - @@ -113,58 +246,103 @@ Getting started du -s ~/.bup - - Make another backup (which should be mostly identical to the last one; - notice that you don't have to *specify* that this backup is incremental, - it just saves space automatically): + - Make another tar backup: tar -cvf - /etc | bup split -n local-etc -vv - - Look how little extra space your second backup used on top of the first: + - Look at how little extra space your second backup used on top of + the first: du -s ~/.bup - - Restore your old backup again (the ~1 is git notation for "one older than - the most recent"): + - Restore the first tar backup again (the ~1 is git notation for "one + older than the most recent"): bup join local-etc~1 | tar -tf - - - get a list of your previous backups: + - Get a list of your previous split-based backups: GIT_DIR=~/.bup git log local-etc - - make a backup on a remote server (which must already have the 'bup' command - somewhere in the PATH, and be accessible via ssh; make sure to replace - SERVERNAME with the actual hostname of your server): + - Make a backup on a remote server: tar -cvf - /etc | bup split -r SERVERNAME: -n local-etc -vv - - try restoring the remote backup tarball: + - Try restoring the remote backup tarball: bup join -r SERVERNAME: local-etc | tar -tf - - - try using the new (slightly experimental) 'bup index' and 'bup save' - style backups, which bypass 'tar' but have some missing features (see - "Things that are stupid" below): - - bup index -uv /etc - bup save -n local-etc /etc - - - do it again and see how fast an incremental backup can be: - - bup index -uv /etc - bup save -n local-etc /etc - - (You can also use the "-r SERVERNAME:" option to 'bup save', just like - with 'bup split' and 'bup join'. The index itself is always local, - so you don't need -r there.) - That's all there is to it! +Notes on FreeBSD +---------------- + +- FreeBSD's default 'make' command doesn't like bup's Makefile. In order to + compile the code, run tests and install bup, you need to install GNU Make + from the port named 'gmake' and use its executable instead in the commands + seen above. (i.e. 'gmake test' runs bup's test suite) + +- Python's development headers are automatically installed with the 'python' + port so there's no need to install them separately. + +- To use the 'bup fuse' command, you need to install the fuse kernel module + from the 'fusefs-kmod' port in the 'sysutils' section and the libraries from + the port named 'py-fusefs' in the 'devel' section. + +- The 'par2' command can be found in the port named 'par2cmdline'. + +- In order to compile the documentation, you need pandoc which can be found in + the port named 'hs-pandoc' in the 'textproc' section. + + +Notes on NetBSD/pkgsrc +---------------------- + + - See pkgsrc/sysutils/bup, which should be the most recent stable + release and includes man pages. It also has a reasonable set of + dependencies (git, par2, py-fuse-bindings). + + - The "fuse-python" package referred to is hard to locate, and is a + separate tarball for the python language binding distributed by the + fuse project on sourceforge. It is available as + pkgsrc/filesystems/py-fuse-bindings and on NetBSD 5, "bup fuse" + works with it. + + - "bup fuse" presents every directory/file as inode 0. The directory + traversal code ("fts") in NetBSD's libc will interpret this as a + cycle and error out, so "ls -R" and "find" will not work. + + - There is no support for ACLs. If/when some entrprising person + fixes this, adjust t/compare-trees. + + +Notes on Cygwin +--------------- + + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust t/compare-trees. + + - In t/test.sh, two tests have been disabled. These tests check to + see that repeated saves produce identical trees and that an + intervening index doesn't change the SHA1. Apparently Cygwin has + some unusual behaviors with respect to access times (that probably + warrant further investigation). Possibly related: + http://cygwin.com/ml/cygwin/2007-06/msg00436.html + + +Notes on OS X +------------- + + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust t/compare-trees. + + How it works ------------- +============ Basic storage: +-------------- bup stores its data in a git-formatted repository. Unfortunately, git itself doesn't actually behave very well for bup's use case (huge numbers of @@ -200,6 +378,7 @@ that tree, respectively, to stdout. You can use this to construct your own scripts that do something with those values. The bup index: +-------------- 'bup index' walks through your filesystem and updates a file (whose name is, by default, ~/.bup/bupindex) to contain the name, attributes, and an @@ -222,28 +401,51 @@ a lot of files have changed. Things that are stupid for now but which we'll fix later --------------------------------------------------------- +======================================================== Help with any of these problems, or others, is very welcome. Join the mailing list (see below) if you'd like to help. - - 'bup save' doesn't know about file metadata. - - That means we aren't saving file attributes, mtimes, ownership, hard - links, MacOS resource forks, etc. Clearly this needs to be improved. + - 'bup restore' can't pull directly from a remote server. - - There's no 'bup restore' yet. + So in one sense "save -r" is a dead-end right now. Obviously you + can use "ssh SERVER bup restore -C ./dest..." to create a tree you + can transfer elsewhere via rsync/tar/whatever, but that's *lame*. + + Until we fix it, you may be able to mount the remote BUP_DIR via + sshfs and then restore "normally", though that hasn't been + officially tested. + + - 'bup save' and 'bup restore' have immature metadata support. - 'bup save' saves files in the standard git 'tree of blobs' format, so you - could then "restore" the files using something like 'git checkout'. But - that's a git command, not a bup command, so it's hard to explain and - doesn't support retrieving objects from a remote bup server without first - fetching and packing an entire (possibly huge) pack, which could be very - slow. Also, like 'bup save', you would need extra features in order to - properly restore file metadata. And files that bup has split into - chunks will need to be recombined. Although there's no restore tool, - 'bup fuse' does accomplish some of this already. - + On the plus side, they actually do have support now, but it's new, + and not remotely as well tested as tar/rsync/whatever's. However, + you have to start somewhere, and as of 0.25, we think it's ready + for more general use. Please let us know if you have any trouble. + + Also, if any strip or graft-style options are specified to 'bup + save', then no metadata will be written for the root directory. + That's obviously less than ideal. + + - bup is overly optimistic about mmap. Right now bup just assumes + that it can mmap as large a block as it likes, and that mmap will + never fail. Yeah, right... If nothing else, this has failed on + 32-bit architectures (and 31-bit is even worse -- looking at you, + s390). + + To fix this, we might just implement a FakeMmap[1] class that uses + normal file IO and handles all of the mmap methods[2] that bup + actually calls. Then we'd swap in one of those whenever mmap + fails. + + This would also require implementing some of the methods needed to + support "[]" array access, probably at a minimum __getitem__, + __setitem__, and __setslice__ [3]. + + [1] http://comments.gmane.org/gmane.comp.sysutils.backup.bup/613 + [2] http://docs.python.org/2/library/mmap.html + [3] http://docs.python.org/2/reference/datamodel.html#emulating-container-types + - 'bup index' is slower than it should be. It's still rather fast: it can iterate through all the filenames on my @@ -259,7 +461,7 @@ mailing list (see below) if you'd like to help. give the continuous-backup process a really low CPU and I/O priority so you wouldn't even know it was running. - - bup currently has no features that prune away *old* backups. + - bup currently has no way to prune *old* backups. Because of the way the packfile system works, backups become "entangled" in weird ways and it's not actually possible to delete one pack @@ -271,20 +473,49 @@ mailing list (see below) if you'd like to help. We'll have to do it in a totally different way. There are lots of options. For now: make sure you've got lots of disk space :) - - bup has never been tested on anything but Linux, MacOS, and Linux+Cygwin. + Until we fix this, one possible workaround is to just start a new + BUP_DIR occasionally, i.e. bup-2013-10, bup-2013-11... + + - bup has never been tested on anything but Linux, MacOS, and Windows+Cygwin. There's nothing that makes it *inherently* non-portable, though, so that's mostly a matter of someone putting in some effort. (For a "native" Windows port, the most annoying thing is the absence of ssh in a default Windows installation.) + + - bup needs better documentation. + + According to a recent article about bup in Linux Weekly News + (https://lwn.net/Articles/380983/), "it's a bit short on examples and + a user guide would be nice." Documentation is the sort of thing that + will never be great unless someone from outside contributes it (since + the developers can never remember which parts are hard to understand). + + - bup is "relatively speedy" and has "pretty good" compression. + + ...according to the same LWN article. Clearly neither of those is good + enough. We should have awe-inspiring speed and crazy-good compression. + Must work on that. Writing more parts in C might help with the speed. - - bup has no GUI. Actually, that's not stupid, but you might consider it - a limitation. There are a bunch of Linux GUI backup programs; someday - I expect someone will adapt one of them to use bup. + - bup has no GUI. + + Actually, that's not stupid, but you might consider it a limitation. + There are a bunch of Linux GUI backup programs; someday I expect someone + will adapt one of them to use bup. + + +More Documentation +================== + +bup has an extensive set of man pages. Try using 'bup help' to get +started, or use 'bup help SUBCOMMAND' for any bup subcommand (like split, +join, index, save, etc.) to get details on that command. + +For further technical details, please see ./DESIGN. How you can help ----------------- +================ bup is a work in progress and there are many ways it can still be improved. If you'd like to contribute patches, ideas, or bug reports, please join the @@ -298,6 +529,11 @@ and you can subscribe by sending a message to: bup-list+subscribe@googlegroups.com +Please see ./HACKING for +additional information, i.e. how to submit patches (hint - no pull +requests), how we handle branches, etc. + + Have fun, Avery