X-Git-Url: https://arthur.barton.de/cgi-bin/gitweb.cgi?p=bup.git;a=blobdiff_plain;f=README.md;h=6ace0a480c31c8c686f38dd35884ac76cf05c56d;hp=cec2b4cbe4f65a675f293817c2f1b305a26b40aa;hb=HEAD;hpb=473ca23dfc0acaec3b7a91340d03981e4a1dc7b0 diff --git a/README.md b/README.md index cec2b4c..2d46819 100644 --- a/README.md +++ b/README.md @@ -67,75 +67,169 @@ bup has a few advantages over other backup software: Reasons you might want to avoid bup ----------------------------------- - - This is a very early version. Therefore it will most probably not work - for you, but we don't know why. It is also missing some - probably-critical features. + - It's not remotely as well tested as something like tar, so it's + more likely to eat your data. It's also missing some + probably-critical features, though fewer than it used to be. - - It requires python >= 2.4, a C compiler, and an installed git version >= - 1.5.3.1. + - It requires python 3.7 or newer, a C compiler, and an installed git + version >= 1.5.6. It also requires par2 if you want fsck to be + able to generate the information needed to recover from some types + of corruption. - - It currently only works on Linux, MacOS X >= 10.4, - NetBSD, Solaris, or Windows (with Cygwin). Patches to support + - It currently only works on Linux, FreeBSD, NetBSD, OS X >= 10.4, + Solaris, or Windows (with Cygwin, and WSL). Patches to support other platforms are welcome. - - + + - Any items in "Things that are stupid" below. + + +Notable changes introduced by a release +======================================= + + - Changes in 0.33 as compared to 0.32 + - Changes in 0.32 as compared to 0.31 + - Changes in 0.31 as compared to 0.30.1 + - Changes in 0.30.1 as compared to 0.30 + - Changes in 0.30 as compared to 0.29.3 + - Changes in 0.29.3 as compared to 0.29.2 + - Changes in 0.29.2 as compared to 0.29.1 + - Changes in 0.29.1 as compared to 0.29 + - Changes in 0.29 as compared to 0.28.1 + - Changes in 0.28.1 as compared to 0.28 + - Changes in 0.28 as compared to 0.27.1 + - Changes in 0.27.1 as compared to 0.27 + + +Test status +=========== + +| master | +|--------| +| [![master branch test status](https://api.cirrus-ci.com/github/bup/bup.svg?branch=master)](https://cirrus-ci.com/github/bup/bup) | + Getting started =============== - From source ----------- - Check out the bup source code using git: - - git clone git://github.com/bup/bup - - Install the needed python libraries (including the development - libraries). + ```sh + git clone https://github.com/bup/bup + ``` - On Debian/Ubuntu this is usually sufficient (run as root): + - This will leave you on the master branch, which is perfect if you + would like to help with development, but if you'd just like to use + bup, please check out the latest stable release like this: - apt-get install python2.6-dev python-fuse - apt-get install python-pyxattr python-pylibacl - apt-get install linux-libc-dev + ```sh + git checkout 0.33 + ``` - Substitute python2.5-dev or python2.4-dev if you have an older - system. Alternately, on newer Debian/Ubuntu versions, you can try - this: - - apt-get build-dep bup + You can see the latest stable release here: + https://github.com/bup/bup/tags - On CentOS (for CentOS 6, at least), this should be sufficient (run + - Install the required python libraries (including the development + libraries). + + For `bup fuse` you will need to install + [python-fuse](https://github.com/libfuse/python-fuse) rather than + [fusepy](https://github.com/fusepy/fusepy). For example, in + Debian, install python3-fuse rather than python3-fusepy. + + On very recent Debian/Ubuntu versions, this may be sufficient (run as root): - yum groupinstall "Development Tools" - yum install python python-dev - yum install fuse-python pyxattr pylibacl - yum install perl-Time-HiRes + ```sh + apt-get build-dep bup + ``` + + Otherwise try this: + + ```sh + apt-get install python3-dev python3-fuse + apt-get install python3-pyxattr python3-pytest + apt-get install python3-distutils + apt-get install pkg-config linux-libc-dev libacl1-dev + apt-get install gcc make acl attr rsync + apt-get isntall python3-pytest-xdist # optional (parallel tests) + apt-get install par2 # optional (error correction) + apt-get install libreadline-dev # optional (bup ftp) + apt-get install python3-tornado # optional (bup web) + + ``` + + On Cygwin, install python, make, rsync, and gcc4. - In addition to the default CentOS repositories, you may need to add - RPMForge (for fuse-python) and EPEL (for pyxattr and pylibacl). + If you would like to use the optional bup web server on systems + without a tornado package, you may want to try this: - - Build the python module and symlinks: + ```sh + pip install tornado + ``` - make + - Build: + + ```sh + make + ``` - Run the tests: - - make test + + ```sh + make long-check + ``` + + or if you're in a bit more of a hurry: + + ```sh + make check + ``` - (The tests should pass. If they don't pass for you, stop here and send - me an email.) + If you have the Python xdist module installed, then you can + probably run the tests faster by adding the make -j option (see ./HACKING for additional information): + + ```sh + make -j check + ``` + + The tests should pass. If they don't pass for you, stop here and + send an email to bup-list@googlegroups.com. Though if there are + symbolic links along the current working directory path, the tests + may fail. Running something like this before "make test" should + sidestep the problem: + + ```sh + cd "$(pwd -P)" + ``` - You can install bup via "make install", and override the default destination with DESTDIR and PREFIX. Files are normally installed to "$DESTDIR/$PREFIX" where DESTDIR is - empty by default, and PREFIX is set to /usr. So if you wanted to + empty by default, and PREFIX is set to /usr/local. So if you wanted to install bup to /opt/bup, you might do something like this: - make install DESTDIR=/opt/bup PREFIX='' - + ```sh + make install DESTDIR=/opt/bup PREFIX='' + ``` + + - The Python version that bup will use is determined by the + `python-config` program chosen by `./configure`, which will search + for a reasonable version unless `BUP_PYTHON_CONFIG` is set in the + environment. You can see which Python executable was chosen by + looking at the configure output, or examining + `config/config.var/bup-python-config`, and you can change the + selection by re-running `./configure`. + +- If you want to specify your own `CPPFLAGS`, `CFLAGS`, or `LDFLAGS`, + you can set them for individual `make` invocations, e.g. `make + CFLAGS=-O0 check`, or persistently via `./configure` with + `CFLAGS=-O0 ./configure`. At the moment, `make clean` clears the + configuration, but we may change that at some point, perhaps by + adding and requiring a `make distclean` to clear the configuration. From binary packages -------------------- @@ -149,68 +243,180 @@ Binary packages of bup are known to be built for the following OSes: - pkgsrc (NetBSD, Dragonfly, and others) http://pkgsrc.se/sysutils/bup http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/sysutils/bup/ + - Arch Linux: + https://www.archlinux.org/packages/?sort=&q=bup + - Fedora: + https://apps.fedoraproject.org/packages/bup + - macOS (Homebrew): + https://formulae.brew.sh/formula/bup Using bup --------- - - Try making a local backup as a tar file: - - tar -cvf - /etc | bup split -n local-etc -vv - - - Try restoring your backup tarball: - - bup join local-etc | tar -tf - - + - Get help for any bup command: + + ```sh + bup help + bup help init + bup help index + bup help save + bup help restore + ... + ``` + + - Initialize the default bup repository (~/.bup -- you can choose + another by either specifying `bup -d DIR ...` or setting the + `BUP_DIR` environment variable for a command): + + ```sh + bup init + ``` + + - Make a local backup (-v or -vv will increase the verbosity): + + ```sh + bup index /etc + bup save -n local-etc /etc + ``` + + - Restore a local backup to ./dest: + + ```sh + bup restore -C ./dest local-etc/latest/etc + ls -l dest/etc + ``` + - Look at how much disk space your backup took: - - du -s ~/.bup - + + ```sh + du -s ~/.bup + ``` + - Make another backup (which should be mostly identical to the last one; notice that you don't have to *specify* that this backup is incremental, it just saves space automatically): - - tar -cvf - /etc | bup split -n local-etc -vv + + ```sh + bup index /etc + bup save -n local-etc /etc + ``` + + - Look how little extra space your second backup used (on top of the first): + + ```sh + du -s ~/.bup + ``` + + - Get a list of your previous backups: + + ```sh + bup ls local-etc + ``` + + - Restore your first backup again: + + ```sh + bup restore -C ./dest-2 local-etc/2013-11-23-11195/etc + ``` + + - Make a backup to a remote server which must already have the 'bup' command + somewhere in its PATH (see /etc/profile, etc/environment, ~/.profile, or + ~/.bashrc), and be accessible via ssh. + Make sure to replace SERVERNAME with the actual hostname of your server: + + ```sh + bup init -r SERVERNAME:path/to/remote-bup-dir + bup index /etc + bup save -r SERVERNAME:path/to/remote-bup-dir -n local-etc /etc + ``` + + - Make a remote backup to ~/.bup on SERVER: + + ```sh + bup index /etc + bup save -r SERVER: -n local-etc /etc + ``` + + - See what saves are available in ~/.bup on SERVER: + + ```sh + bup ls -r SERVER: + ``` + + - Restore the remote backup to ./dest: + + ```sh + bup restore -r SERVER: -C ./dest local-etc/latest/etc + ls -l dest/etc + ``` + + - Defend your backups from death rays (OK fine, more likely from the + occasional bad disk block). This writes parity information + (currently via par2) for all of the existing data so that bup may + be able to recover from some amount of repository corruption: + + ```sh + bup fsck -g + ``` + + - Use split/join instead of index/save/restore. Try making a local + backup using tar: + + ```sh + tar -cvf - /etc | bup split -n local-etc -vv + ``` - - Look how little extra space your second backup used on top of the first: - - du -s ~/.bup + - Try restoring the tarball: + + ```sh + bup join local-etc | tar -tf - + ``` - - Restore your old backup again (the ~1 is git notation for "one older than - the most recent"): - - bup join local-etc~1 | tar -tf - - - - Get a list of your previous backups: + - Look at how much disk space your backup took: + + ```sh + du -s ~/.bup + ``` + + - Make another tar backup: + + ```sh + tar -cvf - /etc | bup split -n local-etc -vv + ``` + + - Look at how little extra space your second backup used on top of + the first: + + ```sh + du -s ~/.bup + ``` + + - Restore the first tar backup again (the ~1 is git notation for "one + older than the most recent"): + + ```sh + bup join local-etc~1 | tar -tf - + ``` - GIT_DIR=~/.bup git log local-etc + - Get a list of your previous split-based backups: + + ```sh + GIT_DIR=~/.bup git log local-etc + ``` - - Make a backup on a remote server (which must already have the 'bup' command - somewhere in the server's PATH (see /etc/profile, etc/environment, - ~/.profile, or ~/.bashrc), and be accessible via ssh. - Make sure to replace SERVERNAME with the actual hostname of your server): - - tar -cvf - /etc | bup split -r SERVERNAME: -n local-etc -vv - - - Try restoring the remote backup tarball: - - bup join -r SERVERNAME: local-etc | tar -tf - - - - Try using the new (slightly experimental) 'bup index' and 'bup save' - style backups, which bypass 'tar' but have some missing features (see - "Things that are stupid" below): - - bup index -uv /etc - bup save -n local-etc /etc - - - Do it again and see how fast an incremental backup can be: + - Save a tar archive to a remote server (without tar -z to facilitate + deduplication): + + ```sh + tar -cvf - /etc | bup split -r SERVERNAME: -n local-etc -vv + ``` - bup index -uv /etc - bup save -n local-etc /etc - - (You can also use the "-r SERVERNAME:" option to 'bup save', just like - with 'bup split' and 'bup join'. The index itself is always local, - so you don't need -r there.) + - Restore the archive: + + ```sh + bup join -r SERVERNAME: local-etc | tar -tf - + ``` That's all there is to it! @@ -218,10 +424,8 @@ That's all there is to it! Notes on FreeBSD ---------------- -- FreeBSD's default 'make' command doesn't like bup's Makefile. In order to - compile the code, run tests and install bup, you need to install GNU Make - from the port named 'gmake' and use its executable instead in the commands - seen above. (i.e. 'gmake test' runs bup's test suite) +- In order to compile the code, run tests and install bup, you need to + install GNU Make from the `gmake` port. - Python's development headers are automatically installed with the 'python' port so there's no need to install them separately. @@ -253,14 +457,36 @@ Notes on NetBSD/pkgsrc traversal code ("fts") in NetBSD's libc will interpret this as a cycle and error out, so "ls -R" and "find" will not work. - - It is not clear if extended attribute and POSIX acl support does - anything useful. + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust dev/compare-trees. + + +Notes on Cygwin +--------------- + + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust dev/compare-trees. + + - In test/ext/test-misc, two tests have been disabled. These tests + check to see that repeated saves produce identical trees and that + an intervening index doesn't change the SHA1. Apparently Cygwin + has some unusual behaviors with respect to access times (that + probably warrant further investigation). Possibly related: + http://cygwin.com/ml/cygwin/2007-06/msg00436.html + + +Notes on OS X +------------- + + - There is no support for ACLs. If/when some enterprising person + fixes this, adjust dev/compare-trees. How it works ============ Basic storage: +-------------- bup stores its data in a git-formatted repository. Unfortunately, git itself doesn't actually behave very well for bup's use case (huge numbers of @@ -271,8 +497,8 @@ python. Basically, 'bup split' reads the data on stdin (or from files specified on the command line), breaks it into chunks using a rolling checksum (similar to -rsync), and saves those chunks into a new git packfile. There is one git -packfile per backup. +rsync), and saves those chunks into a new git packfile. There is at least one +git packfile per backup. When deciding whether to write a particular chunk into the new packfile, bup first checks all the other packfiles that exist to see if they already have that @@ -296,6 +522,7 @@ that tree, respectively, to stdout. You can use this to construct your own scripts that do something with those values. The bup index: +-------------- 'bup index' walks through your filesystem and updates a file (whose name is, by default, ~/.bup/bupindex) to contain the name, attributes, and an @@ -318,7 +545,7 @@ a lot of files have changed. Things that are stupid for now but which we'll fix later --------------------------------------------------------- +======================================================== Help with any of these problems, or others, is very welcome. Join the mailing list (see below) if you'd like to help. @@ -326,13 +553,32 @@ mailing list (see below) if you'd like to help. - 'bup save' and 'bup restore' have immature metadata support. On the plus side, they actually do have support now, but it's new, - and not remotely as well tested as tar/rsync/whatever's. If you'd - like to help test, please do; something like 'rsync -niaHAX src/ - restore/' may be useful on that front. + and not remotely as well tested as tar/rsync/whatever's. However, + you have to start somewhere, and as of 0.25, we think it's ready + for more general use. Please let us know if you have any trouble. + + Also, if any strip or graft-style options are specified to 'bup + save', then no metadata will be written for the root directory. + That's obviously less than ideal. + + - bup is overly optimistic about mmap. Right now bup just assumes + that it can mmap as large a block as it likes, and that mmap will + never fail. Yeah, right... If nothing else, this has failed on + 32-bit architectures (and 31-bit is even worse -- looking at you, + s390). + + To fix this, we might just implement a FakeMmap[1] class that uses + normal file IO and handles all of the mmap methods[2] that bup + actually calls. Then we'd swap in one of those whenever mmap + fails. + + This would also require implementing some of the methods needed to + support "[]" array access, probably at a minimum __getitem__, + __setitem__, and __setslice__ [3]. - In addition, at the moment, if any strip or graft-style options - are specified to 'bup save', then no metadata will be written for - the root directory. That's obviously less than ideal. + [1] http://comments.gmane.org/gmane.comp.sysutils.backup.bup/613 + [2] http://docs.python.org/3/library/mmap.html + [3] http://docs.python.org/3/reference/datamodel.html#emulating-container-types - 'bup index' is slower than it should be. @@ -349,19 +595,8 @@ mailing list (see below) if you'd like to help. give the continuous-backup process a really low CPU and I/O priority so you wouldn't even know it was running. - - bup currently has no features that prune away *old* backups. - - Because of the way the packfile system works, backups become "entangled" - in weird ways and it's not actually possible to delete one pack - (corresponding approximately to one backup) without risking screwing up - other backups. - - git itself has lots of ways of optimizing this sort of thing, but its - methods aren't really applicable here; bup packfiles are just too huge. - We'll have to do it in a totally different way. There are lots of - options. For now: make sure you've got lots of disk space :) - - - bup has never been tested on anything but Linux, MacOS, and Windows+Cygwin. + - bup has never been tested on anything but Linux, FreeBSD, NetBSD, + OS X, and Windows+Cygwin. There's nothing that makes it *inherently* non-portable, though, so that's mostly a matter of someone putting in some effort. (For a @@ -370,7 +605,7 @@ mailing list (see below) if you'd like to help. - bup needs better documentation. - According to a recent article about git in Linux Weekly News + According to an article about bup in Linux Weekly News (https://lwn.net/Articles/380983/), "it's a bit short on examples and a user guide would be nice." Documentation is the sort of thing that will never be great unless someone from outside contributes it (since @@ -384,13 +619,12 @@ mailing list (see below) if you'd like to help. - bup has no GUI. - Actually, that's not stupid, but you might consider it a limitation. - There are a bunch of Linux GUI backup programs; someday I expect someone - will adapt one of them to use bup. - + Actually, that's not stupid, but you might consider it a + limitation. See the ["Related Projects"](https://bup.github.io/) + list for some possible options. More Documentation ------------------- +================== bup has an extensive set of man pages. Try using 'bup help' to get started, or use 'bup help SUBCOMMAND' for any bup subcommand (like split, @@ -404,7 +638,7 @@ How you can help bup is a work in progress and there are many ways it can still be improved. If you'd like to contribute patches, ideas, or bug reports, please join the -bup mailing list. +bup mailing list: You can find the mailing list archives here: @@ -414,7 +648,12 @@ and you can subscribe by sending a message to: bup-list+subscribe@googlegroups.com -Please see ./HACKING for +You can also reach us via the +\#bup IRC channel at ircs://irc.libera.chat:6697/bup +on the [libera.chat](https://libera.chat/) network or via this +[web interface](https://web.libera.chat/?channels=bup). + +Please see ./HACKING for additional information, i.e. how to submit patches (hint - no pull requests), how we handle branches, etc. @@ -422,3 +661,9 @@ requests), how we handle branches, etc. Have fun, Avery + +