--- /dev/null
+Python code follows PEP8 [1] with regard to coding style and PEP257 [2] with
+regard to docstring style. Multi-line docstrings should have one short summary
+line, followed by a blank line and a series of paragraphs. The last paragraph
+should be followed by a line that closes the docstring (no blank line in
+between). Here's an example from lib/bup/helpers.py:
+
+def unlink(f):
+ """Delete a file at path 'f' if it currently exists.
+
+ Unlike os.unlink(), does not throw an exception if the file didn't already
+ exist.
+ """
+ #code...
+
+Module-level docstrings follow exactly the same guidelines but without the
+blank line between the summary and the details.
+
+
+The C implementations should follow the kernel/git coding style [3].
+
+
+[1]:http://www.python.org/dev/peps/pep-0008/
+[2]:http://www.python.org/dev/peps/pep-0257/
+[3]:http://www.kernel.org/doc/Documentation/CodingStyle
a backup using the 'bup save' command, but that's getting ahead of
ourselves.
-As most backup experts know, backing stuff up is normally about 100x more
-common than restoring stuff, ie. copying from the repository to your
-filesystem. For that reason, and also because bup is so new, there is no
-actual 'bup restore' command that does the obvious inverse operation to 'bup
-save'. There are 'bup ftp' and 'bup fuse', which let you access your
-backed-up data, but they aren't as efficient as a fully optimized restore
-tool intended for high-volume restores. There's nothing stopping us from
-writing one; we just haven't written it yet. Feel free to pester us about
-it on the bup mailing list (see the README to find out about the list).
+For the inverse operation, ie. copying from the repository to your
+filesystem, you have several choices; the main ones are 'bup restore', 'bup
+ftp', 'bup fuse', and 'bup web'.
Now, those are the basics of backups. In other words, we just spent about
half a page telling you that bup backs up and restores data. Are we having
update only the ones that have changed. (Even more interesting things
happen if people are using the files on the restored system and you haven't
updated the index yet; the net result would be an automated merge of all
-non-conflicting files.) This would be a poor man's distributed filesystem.
-The only catch is that nobody has written 'bup restore' yet. Someday!
+non-conflicting files.) This would be a poor man's distributed filesystem.
+The only catch is that nobody has written this feature for 'bup restore'
+yet. Someday!
How 'bup save' works (cmd/save)
--- /dev/null
+% bup-bloom(1) Bup %BUP_VERSION%
+% Brandon Low <lostlogic@lostlogicx.com>
+% %BUP_DATE%
+
+# NAME
+
+bup-bloom - generates, regenerates, updates bloom filters
+
+# SYNOPSIS
+
+bup daemon [-d dir] [-o outfile] [-k hashes]
+
+# DESCRIPTION
+
+`bup bloom` builds a bloom filter file for a bup repo, if
+one already exists, it checks it and updates or regenerates
+it if needed.
+
+# OPTIONS
+
+-d, --dir=*directory*
+: the directory, containing .idx files, to process.
+ defaults to $BUP_DIR/objects/pack
+
+-o, --outfile=*outfile*
+: the file to write the bloom filter to. defaults to
+ $dir/bup.bloom
+
+-k, --hashes=*hashes*
+: number of hash functions to use only 4 and 5 are valid.
+ defaults to 5 for repositories < 2TiB and 4 otherwise.
+ see comments in git.py for more on this value.
+
+# BUP
+
+Part of the `bup`(1) suite.
--- /dev/null
+% bup-daemon(1) Bup %BUP_VERSION%
+% Brandon Low <lostlogic@lostlogicx.com>
+% %BUP_DATE%
+
+# NAME
+
+bup-daemon - listens for connections and runs `bup server`
+
+# SYNOPSIS
+
+bup daemon [-l address] [-p port]
+
+# DESCRIPTION
+
+`bup daemon` is a simple bup server which listens on a
+socket and forks connections to `bup mux server` children.
+
+# OPTIONS
+
+-l, --listen=*address*
+: the address or hostname to listen on
+
+-p, --port=*port*
+: the port to listen on
+
+# BUP
+
+Part of the `bup`(1) suite.
# SYNOPSIS
-bup drecurse [-x] [-q] [--profile] \<path\>
+bup drecurse [-x] [-q] [--exclude *path*]
+[--exclude-from *filename*] [--profile] \<path\>
# DESCRIPTION
-q, --quiet
: don't print filenames as they are encountered. Useful
when testing performance of the traversal algorithms.
+
+--exclude=*path*
+: a path to exclude from the backup (can be used more
+ than once)
+
+--exclude-from=*filename*
+: a file that contains exclude paths (can be used more
+ than once)
--profile
: print profiling information upon completion. Useful
--- /dev/null
+% bup-import-rsnapshot(1) Bup %BUP_VERSION%
+% Zoran Zaric <zz@zoranzaric.de>
+% %BUP_DATE%
+
+# NAME
+
+bup-import-rsnapshot - import a rsnapshot archive
+
+# SYNOPSIS
+
+bup import-rsnapshot [-n] <path to snapshot_root> [<backuptarget>]
+
+# SYNOPSIS
+
+`bup import-rsnapshot` imports a rsnapshot archive. The
+timestamps for the backups are preserved and the path to
+the rsnapshot archive is stripped from the paths.
+
+`bup import-rsnapshot` either imports the whole archive
+or only imports all backups for a given backuptarget.
+
+# OPTIONS
+
+-n,--dry-rung
+: don't do anything just print out what would be done
+
+# EXAMPLE
+
+ $ bup import-rsnapshot /.snapshots
+
+# BUP
+
+Part of the `bup`(1) suite.
# SYNOPSIS
bup index <-p|-m|-u> [-s] [-H] [-l] [-x] [--fake-valid]
-[--check] [-f *indexfile*] [-v] <filenames...>
+[--check] [-f *indexfile*] [--exclude *path*]
+[--exclude-from *filename*] [-v] <filenames...>
# DESCRIPTION
: use a different index filename instead of
`~/.bup/bupindex`.
+--exclude=*path*
+: a path to exclude from the backup (can be used more
+ than once)
+
+--exclude-from=*filename*
+: a file that contains exclude paths (can be used more
+ than once)
+
-v, --verbose
: increase log output during update (can be used more
than once). With one `-v`, print each directory as it
# SYNOPSIS
-bup ls [-s] <paths...>
+bup ls [-s]
[-a] <paths...>
# DESCRIPTION
using the same directory hierarchy as they would have with
`bup-fuse`(1).
-The top level directory is the branch (corresponding to
+The top level directory contains the branch (corresponding to
the `-n` option in `bup save`), the next level is the date
of the backup, and subsequent levels correspond to files in
the backup.
+Note that `bup ls` doesn't show hidden files by default and one needs to use
+the `-a` option to show them. Files are hidden when their name begins with a
+dot. For example, on the topmost level, the special directories named `.commit`
+and `.tag` are hidden directories.
+
Once you have identified the file you want using `bup ls`,
you can view its contents using `bup join` or `git show`.
-s, --hash
: show hash for each file/directory.
+-a, --all
+: show hidden files.
# EXAMPLE
bup ls /myserver/latest/etc/profile
+ bup ls -a /
+
# SEE ALSO
`bup-join`(1), `bup-fuse`(1), `bup-ftp`(1), `bup-save`(1), `git-show`(1)
--- /dev/null
+% bup-mux(1) Bup %BUP_VERSION%
+% Brandon Low <lostlogic@lostlogicx.com>
+% %BUP_DATE%
+
+# NAME
+
+bup-mux - multiplexes data and error streams over a connection
+
+# SYNOPSIS
+
+bup mux \<command\> [options...]
+
+# DESCRIPTION
+
+`bup mux` is used in the bup client-server protocol to
+send both data and debugging/error output over the single
+connection stream.
+
+`bup mux server` might be used in an inetd server setup.
+
+# OPTIONS
+
+command
+: the subcommand to run
+options
+: options for command
+
+# BUP
+
+Part of the `bup`(1) suite.
# SYNOPSIS
-bup random [-S seed] [-f] <numbytes>
+bup random [-S seed] [-fv] <numbytes>
# DESCRIPTION
: generate output even if stdout is a tty. (Generating
random data to a tty is generally considered
ill-advised, but you can do if you really want.)
+
+-v, --verbose
+: print a progress message showing the number of bytes that
+ has been output so far.
# EXAMPLES
# SYNOPSIS
-bup save [-r *host*:*path*] <-t|-c|-n *name*> [-v] [-q]
-
[--smaller=*maxsize*] <paths...>
+bup save [-r *host*:*path*] <-t|-c|-n *name*> [-f *indexfile*]
+
[-v] [-q] [--smaller=*maxsize*] <paths...>
# DESCRIPTION
the same name, and later view the history of that
backup set to see how files have changed over time.)
+-f, --indexfile=*indexfile*
+: use a different index filename instead of
+ `~/.bup/bupindex`.
+
-v, --verbose
: increase verbosity (can be used more than once). With
one -v, prints every directory name as it gets backed up. With
like k, M, or G to specify multiples of 1024,
1024*1024, 1024*1024*1024 respectively.
+--strip
+: strips the path that is given from all files and directories.
+
+ A directory */root/chroot/etc* saved with
+ "bup save -n chroot --strip /root/chroot" would be saved
+ as */etc*.
+
+--strip-prefix=*path-prefix*
+: strips the given path-prefix *path-prefix* from all
+ files and directories.
+
+ A directory */root/chroots/webserver* saved with
+ "bup save -n webserver --strip-path=/root/chroots" would
+ be saved as */webserver/etc*
+
+--graft=*old_path*=*new_path*
+: a graft point *old_path*=*new_path* (can be used more than
+ once).
+
+ A directory */root/chroot/a/etc* saved with
+ "bup save -n chroots --graft /root/chroot/a/etc=/chroots/a"
+ would be saved as */chroots/a/etc*
# EXAMPLE
-
+
$ bup index -ux /etc
Indexing: 1981, done.
-
+
$ bup save -r myserver: -n my-pc-backup --bwlimit=50k /etc
Reading index: 1981, done.
- Saving: 100.00% (998/998k, 1981/1981 files), done.
-
+ Saving: 100.00% (998/998k, 1981/1981 files), done.
+
+
+
+ $ ls /home/joe/chroots/httpd
+ bin var
+
+ $ bup index -ux /home/joe/chroots/httpd
+ Indexing: 1337, done.
+
+ $ bup save --strip -n joes-httpd-chroot /home/joe/chroots/httpd
+ Reading index: 1337, done.
+ Saving: 100.00% (998/998k, 1337/1337 files), done.
+
+ $ bup ls joes-httpd-chroot/latest/
+ bin/
+ var/
+
+
+ $ bup save --strip-prefix=/home/joe/chroots -n joes-chroots \
+ /home/joe/chroots/httpd
+ Reading index: 1337, done.
+ Saving: 100.00% (998/998k, 1337/1337 files), done.
+
+ $ bup ls joes-chroots/latest/
+ httpd/
+
+
+ $ bup save --graft /home/joe/chroots/httpd=/http-chroot \
+ -n joe
+ /home/joe/chroots/httpd
+ Reading index: 1337, done.
+ Saving: 100.00% (998/998k, 1337/1337 files), done.
+
+ $ bup ls joe/latest/
+ http-chroot/
+
# SEE ALSO
There is normally no reason to run `bup server` yourself.
+# MODES
+
+smart
+: In this mode, the server checks each incoming object
+ against the idx files in its repository. If any object
+ already exists, it tells the client about the idx file
+ it was found in, allowing the client to download that
+ idx and avoid sending duplicate data.
+
+dumb
+: In this mode, the server will not check its local index
+ before writing an object. To avoid writing duplicate
+ objects, the server will tell the client to download all
+ of its .idx files at the start of the session. This
+ mode is useful on low powered server hardware (ie
+ router/slow NAS).
+
+# FILES
+
+$BUP_DIR/bup-dumb-server
+: Activate dumb server mode, as discussed above.
+
# SEE ALSO
`bup-save`(1), `bup-split`(1)
bup split [-r *host*:*path*] <-b|-t|-c|-n *name*> [-v] [-q]
[--bench] [--max-pack-size=*bytes*]
- [--max-pack-objects=*n*] [--fanout=*count] [filenames...]
+ [--max-pack-objects=*n*] [--fanout=*count]
+ [--git-ids] [--keep-boundaries] [filenames...]
# DESCRIPTION
the content into chunks of around 8k using a rolling
checksum algorithm, and saves the chunks into a bup
repository. Chunks which have previously been stored are
-not stored again (ie. they are "deduplicated").
+not stored again (ie. they are 'deduplicated').
Because of the way the rolling checksum works, chunks
tend to be very stable across changes to a given file,
-v, --verbose
: increase verbosity (can be used more than once).
+--git-ids
+: stdin is a list of git object ids instead of raw data.
+ `bup split` will read the contents of each named git
+ object (if it exists in the bup repository) and split
+ it. This might be useful for converting a git
+ repository with large binary files to use bup-style
+ hashsplitting instead. This option is probably most
+ useful when combined with `--keep-boundaries`.
+
+--keep-boundaries
+: if multiple filenames are given on the command line,
+ they are normally concatenated together as if the
+ content all came from a single file. That is, the
+ set of blobs/trees produced is identical to what it
+ would have been if there had been a single input file.
+ However, if you use `--keep-boundaries`, each file is
+ split separately. You still only get a single tree or
+ commit or series of blobs, but each blob comes from
+ only one of the files; the end of one of the input
+ files always ends a blob.
+
--noop
: read the data and split it into blocks based on the "bupsplit"
rolling checksum algorithm, but don't do anything with
--- /dev/null
+% bup-tag(1) Bup %BUP_VERSION%
+% Gabriel Filion <lelutin@gmail.com>
+% %BUP_DATE%
+
+# NAME
+
+bup-tag - tag a commit in the bup repository
+
+# SYNOPSIS
+
+bup tag
+
+bup tag \<tag name\> \<committish\>
+
+bup tag -d \<tag name\>
+
+# DESCRIPTION
+
+`bup tag` lists, creates or deletes a tag in the bup repository.
+
+A tag is an easy way to retreive a specific commit. It can be used to mark a
+specific backup for easier retrieval later.
+
+When called without any arguments, the command lists all tags that can
+be found in the repository. When called with a tag name and a commit ID
+or ref name, it creates a new tag with the given name, if it doesn't
+already exist, that points to the commit given in the second argument. When
+called with '-d' and a tag name, it removes the given tag, if it exists.
+
+bup exposes the contents of backups with current tags, via any command that
+lists or shows backups. They can be found under the /.tag directory. For
+example, the 'ftp' command will show the tag named 'tag1' under /.tag/tag1.
+
+Tags are also exposed under the branches from which they can be reached. For
+example, if you create a tag named 'important' under branch 'computerX', you
+will also be able to retrieve the contents of the backup that was tagged under
+/computerX/important. This is done as a convenience, and should the branch
+'computerX' be deleted, the contents of the tagged backup will be available
+through /.tag/important as long as the tag is not deleted.
+
+# OPTIONS
+
+-d, --delete
+: delete a tag
+
+# EXAMPLE
+
+ $ bup tag new-puppet-version hostx-backup
+
+ $ bup tag
+ new-puppet-version
+
+ $ bup ftp "ls /.tag/new-puppet-version"
+ files..
+
+ $ bup tag -d new-puppet-version
+
+# SEE ALSO
+
+`bup-save`(1), `bup-split`(1), `bup-ftp`(1), `bup-fuse`(1), `bup-web`(1)
+
+# BUP
+
+Part of the `bup`(1) suite.
runtests: all runtests-python runtests-cmdline
runtests-python:
- $(PYTHON) wvtest.py $(wildcard t/t*.py lib/*/t/t*.py)
+ $(PYTHON) wvtest.py t/t*.py lib/*/t/t*.py
runtests-cmdline: all
t/test.sh
rm -f $@
ln -s $< $@
-cmds: $(patsubst cmd/%-cmd.py,cmd/bup-%,$(wildcard cmd/*-cmd.py))
+cmds: \
+ $(patsubst cmd/%-cmd.py,cmd/bup-%,$(wildcard cmd/*-cmd.py)) \
+ $(patsubst cmd/%-cmd.sh,cmd/bup-%,$(wildcard cmd/*-cmd.sh))
cmd/bup-%: cmd/%-cmd.py
rm -f $@
rm -f $@
ln -s $< $@
+cmd/bup-%: cmd/%-cmd.sh
+ rm -f $@
+ ln -s $*-cmd.sh $@
+
%.o: %.c
gcc -c -o $@ $< $(CPPFLAGS) $(CFLAGS)
+
+# update the local 'man' and 'html' branches with pregenerated output files, for
+# people who don't have pandoc (and maybe to aid in google searches or something)
+export-docs: Documentation/all
+ git update-ref refs/heads/man origin/man '' 2>/dev/null || true
+ git update-ref refs/heads/html origin/html '' 2>/dev/null || true
+ GIT_INDEX_FILE=gitindex.tmp; export GIT_INDEX_FILE; \
+ rm -f $${GIT_INDEX_FILE} && \
+ git add -f Documentation/*.1 && \
+ git update-ref refs/heads/man \
+ $$(echo "Autogenerated man pages for $$(git describe)" \
+ | git commit-tree $$(git write-tree --prefix=Documentation) \
+ -p refs/heads/man) && \
+ rm -f $${GIT_INDEX_FILE} && \
+ git add -f Documentation/*.html && \
+ git update-ref refs/heads/html \
+ $$(echo "Autogenerated html pages for $$(git describe)" \
+ | git commit-tree $$(git write-tree --prefix=Documentation) \
+ -p refs/heads/html)
+
+# push the pregenerated doc files to origin/man and origin/html
+push-docs: export-docs
+ git push origin man html
+
+# import pregenerated doc files from origin/man and origin/html, in case you
+# don't have pandoc but still want to be able to install the docs.
+import-docs: Documentation/clean
+ git archive origin/html | (cd Documentation; tar -xvf -)
+ git archive origin/man | (cd Documentation; tar -xvf -)
clean: Documentation/clean
rm -f *.o lib/*/*.o *.so lib/*/*.so *.dll *.exe \
- It requires python >= 2.4, a C compiler, and an installed git version >=
1.5.3.1.
- - It currently only works on Linux, MacOS X >= 10.4, or Windows (with
- Cygwin). Patches to support other platforms are welcome.
+ - It currently only works on Linux, MacOS X >= 10.4,
+ Solaris, or Windows (with Cygwin). Patches to support
+ other platforms are welcome.
Getting started
apt-get install python2.6-dev python-fuse
Substitute python2.5-dev or python2.4-dev if you have an older system.
+
+ Or on newer Debian/Ubuntu versions, you can try this:
+
+ apt-get build-dep bup
- Build the python module and symlinks:
That's all there is to it!
+Notes on FreeBSD
+================
+
+- FreeBSD's default 'make' command doesn't like bup's Makefile. In order to
+ compile the code, run tests and install bup, you need to install GNU Make
+ from the port named 'gmake' and use its executable instead in the commands
+ seen above. (i.e. 'gmake test' runs bup's test suite)
+
+- Python's development headers are automatically installed with the 'python'
+ port so there's no need to install them separately.
+
+- To use the 'bup fuse' command, you need to install the fuse kernel module
+ from the 'fusefs-kmod' port in the 'sysutils' section and the libraries from
+ the port named 'py-fusefs' in the 'devel' section.
+
+- The 'par2' command can be found in the port named 'par2cmdline'.
+
+- In order to compile the documentation, you need pandoc which can be found in
+ the port named 'hs-pandoc' in the 'textproc' section.
+
+
How it works
------------
Actually, that's not stupid, but you might consider it a limitation.
There are a bunch of Linux GUI backup programs; someday I expect someone
will adapt one of them to use bup.
+
+
+More Documentation
+------------------
+
+bup has an extensive set of man pages. Try using 'bup help' to get
+started, or use 'bup help SUBCOMMAND' for any bup subcommand (like split,
+join, index, save, etc.) to get details on that command.
How you can help
--- /dev/null
+#!/usr/bin/env python
+import sys, glob, tempfile
+from bup import options, git
+from bup.helpers import *
+
+optspec = """
+bup bloom [options...]
+--
+o,output= output bloom filename (default: auto-generated)
+d,dir= input directory to look for idx files (default: auto-generated)
+k,hashes= number of hash functions to use (4 or 5) (default: auto-generated)
+"""
+
+def do_bloom(path, outfilename):
+ if not outfilename:
+ assert(path)
+ outfilename = os.path.join(path, 'bup.bloom')
+
+ b = None
+ if os.path.exists(outfilename):
+ b = git.ShaBloom(outfilename)
+ if not b.valid():
+ debug1("bloom: Existing invalid bloom found, regenerating.\n")
+ b = None
+
+ add = []
+ rest = []
+ add_count = 0
+ rest_count = 0
+ for name in glob.glob('%s/*.idx' % path):
+ ix = git.open_idx(name)
+ ixbase = os.path.basename(name)
+ if b and (ixbase in b.idxnames):
+ rest.append(name)
+ rest_count += len(ix)
+ else:
+ add.append(name)
+ add_count += len(ix)
+ total = add_count + rest_count
+
+ if not add:
+ log("bloom: Nothing to do\n")
+ return
+
+ if b:
+ if len(b) != rest_count:
+ log("bloom: size %d != idx total %d, regenerating\n"
+ % (len(b), rest_count))
+ b = None
+ elif (b.bits < git.MAX_BLOOM_BITS and
+ b.pfalse_positive(add_count) > git.MAX_PFALSE_POSITIVE):
+ log("bloom: %d more entries => %.2f false positive, regenerating\n"
+ % (add_count, b.pfalse_positive(add_count)))
+ b = None
+ else:
+ b = git.ShaBloom(outfilename, readwrite=True, expected=add_count)
+ if not b: # Need all idxs to build from scratch
+ add += rest
+ add_count += rest_count
+ del rest
+ del rest_count
+
+ msg = b is None and 'creating from' or 'adding'
+ log('bloom: %s %d files (%d objects).\n' % (msg, len(add), add_count))
+
+ tfname = None
+ if b is None:
+ tfname = os.path.join(path, 'bup.tmp.bloom')
+ tf = open(tfname, 'w+')
+ b = git.ShaBloom.create(tfname, f=tf, expected=add_count, k=opt.k)
+ count = 0
+ for name in add:
+ ix = git.open_idx(name)
+ progress('Writing bloom: %d/%d\r' % (count, len(add)))
+ b.add_idx(ix)
+ count += 1
+ log('Writing bloom: %d/%d, done.\n' % (count, len(add)))
+
+ if tfname:
+ os.rename(tfname, outfilename)
+
+
+handle_ctrl_c()
+
+o = options.Options(optspec)
+(opt, flags, extra) = o.parse(sys.argv[1:])
+
+if extra:
+ o.fatal('no positional parameters expected')
+
+if opt.k and opt.k not in (4,5):
+ o.fatal('only k values of 4 and 5 are supported')
+
+git.check_repo_or_die()
+
+do_bloom(opt.dir or git.repo('objects/pack'), opt.output)
--- /dev/null
+#!/usr/bin/env python
+import sys, getopt, socket, subprocess
+from bup import options, path
+from bup.helpers import *
+
+optspec = """
+bup daemon [options...]
+--
+l,listen ip address to listen on, defaults to *
+p,port port to listen on, defaults to 1982
+"""
+o = options.Options(optspec, optfunc=getopt.getopt)
+(opt, flags, extra) = o.parse(sys.argv[1:])
+if extra:
+ o.fatal('no arguments expected')
+
+host = opt.listen
+port = opt.port and int(opt.port) or 1982
+
+import socket
+import sys
+
+socks = []
+for res in socket.getaddrinfo(host, port, socket.AF_UNSPEC,
+ socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
+ af, socktype, proto, canonname, sa = res
+ try:
+ s = socket.socket(af, socktype, proto)
+ except socket.error, msg:
+ continue
+ try:
+ if af == socket.AF_INET6:
+ log("bup daemon: listening on [%s]:%s\n" % sa[:2])
+ else:
+ log("bup daemon: listening on %s:%s\n" % sa[:2])
+ s.bind(sa)
+ s.listen(1)
+ except socket.error, msg:
+ s.close()
+ continue
+ socks.append(s)
+
+if not socks:
+ log('bup daemon: could not open socket\n')
+ sys.exit(1)
+
+try:
+ while True:
+ [rl,wl,xl] = select.select(socks, [], [], 60)
+ for l in rl:
+ s, src = l.accept()
+ log("Socket accepted connection from %s\n" % (src,))
+ sp = subprocess.Popen([path.exe(), 'mux', 'server'],
+ stdin=os.dup(s.fileno()), stdout=os.dup(s.fileno()))
+ s.close()
+finally:
+ for l in socks:
+ l.shutdown(socket.SHUT_RDWR)
+ l.close()
+
+debug1("bup daemon: done")
equal spread damage evenly throughout the file
S,seed= random number seed (for repeatable tests)
"""
-o = options.Options('bup damage', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if not extra:
bup drecurse <path>
--
x,xdev,one-file-system don't cross filesystem boundaries
+exclude= a path to exclude from the backup (can be used more than once)
+exclude-from= a file that contains exclude paths (can be used more than once)
q,quiet don't actually print filenames
profile run under the python profiler
"""
-o = options.Options('bup drecurse', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if len(extra) != 1:
o.fatal("exactly one filename expected")
-it = drecurse.recursive_dirlist(extra, opt.xdev)
+excluded_paths = drecurse.parse_excludes(flags)
+
+it = drecurse.recursive_dirlist(extra, opt.xdev, excluded_paths)
if opt.profile:
import cProfile
def do_it():
par2-ok immediately return 0 if par2 is ok, 1 if not
disable-par2 ignore par2 even if it is available
"""
-o = options.Options('bup fsck', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
par2_setup()
--
a,all include hidden files in the listing
"""
-ls_opt = options.Options('ls', ls_optspec, onabort=OptionError)
+ls_opt = options.Options(ls_optspec, onabort=OptionError)
def do_ls(cmd_args):
try:
optspec = """
bup ftp [commands...]
"""
-o = options.Options('bup ftp', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
git.check_repo_or_die()
f,foreground run in foreground
o,allow-other allow other users to access the filesystem
"""
-o = options.Options('bup fuse', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if len(extra) != 1:
#!/usr/bin/env python
import sys, os, glob
-from bup import options
+from bup import options, path
optspec = """
bup help <command>
"""
-o = options.Options('bup help', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if len(extra) == 0:
os.execvp(os.environ['BUP_MAIN_EXE'], ['bup'])
elif len(extra) == 1:
docname = (extra[0]=='bup' and 'bup' or ('bup-%s' % extra[0]))
- exe = sys.argv[0]
- (exepath, exefile) = os.path.split(exe)
- manpath = os.path.join(exepath, '../Documentation/' + docname + '.[1-9]')
+ manpath = os.path.join(path.exedir(),
+ 'Documentation/' + docname + '.[1-9]')
g = glob.glob(manpath)
if g:
os.execvp('man', ['man', '-l', g[0]])
--- /dev/null
+#!/bin/sh
+# Does an import of a rsnapshot archive.
+
+usage() {
+ echo "Usage: bup import-rsnapshot [-n]" \
+ "<path to snapshot_root> [<backuptarget>]"
+ echo "-n,--dry-run: just print what would be done"
+ exit -1
+}
+
+DRY_RUN=
+while [ "$1" = "-n" -o "$1" = "--dry-run" ]; do
+ DRY_RUN=echo
+ shift
+done
+
+bup()
+{
+ $DRY_RUN "${BUP_MAIN_EXE:=bup}" "$@"
+}
+
+SNAPSHOT_ROOT=$1
+TARGET=$2
+
+[ -n "$SNAPSHOT_ROOT" -a "$#" -le 2 ] || usage
+
+if [ ! -e "$SNAPSHOT_ROOT/." ]; then
+ echo "'$SNAPSHOT_ROOT' isn't a directory!"
+ exit 1
+fi
+
+
+cd "$SNAPSHOT_ROOT" || exit 2
+
+for SNAPSHOT in *; do
+ [ -e "$SNAPSHOT/." ] || continue
+ echo "snapshot='$SNAPSHOT'" >&2
+ for BRANCH_PATH in "$SNAPSHOT/"*; do
+ BRANCH=$(basename "$BRANCH_PATH")
+ [ -e "$BRANCH_PATH/." ] || continue
+ [ -z "$TARGET" -o "$TARGET" = "$BRANCH" ] || continue
+
+ echo "snapshot='$SNAPSHOT' branch='$BRANCH'" >&2
+
+ # Get the snapshot's ctime
+ DATE=$(perl -e '@a=stat($ARGV[0]) or die "$ARGV[0]: $!";
+ print $a[10];' "$BRANCH_PATH")
+ [ -n "$DATE" ] || exit 3
+
+ TMPIDX=bupindex.$BRANCH.tmp
+ bup index -ux -f "$TMPIDX" "$BRANCH_PATH/"
+ bup save --strip --date="$DATE" \
+ -f "$TMPIDX" -n "$BRANCH" \
+ "$BRANCH_PATH/"
+ rm -f "$TMPIDX"
+ done
+done
#!/usr/bin/env python
-import sys, stat, time
+import sys, stat, time, os
from bup import options, git, index, drecurse
from bup.helpers import *
-def merge_indexes(out, r1, r2):
- for e in index.MergeIter([r1, r2]):
- # FIXME: shouldn't we remove deleted entries eventually? When?
- out.add_ixentry(e)
-
-
class IterHelper:
def __init__(self, l):
self.i = iter(l)
log('check: passed.\n')
-def update_index(top):
+def update_index(top, excluded_paths):
ri = index.Reader(indexfile)
wi = index.Writer(indexfile)
rig = IterHelper(ri.iter(name=top))
return (0100644, index.FAKE_SHA)
total = 0
- for (path,pst) in drecurse.recursive_dirlist([top], xdev=opt.xdev):
+ bup_dir = os.path.abspath(git.repo())
+ for (path,pst) in drecurse.recursive_dirlist([top], xdev=opt.xdev,
+ bup_dir=bup_dir,
+ excluded_paths=excluded_paths):
if opt.verbose>=2 or (opt.verbose==1 and stat.S_ISDIR(pst.st_mode)):
sys.stdout.write('%s\n' % path)
sys.stdout.flush()
log('check: before merging: newfile\n')
check_index(wr)
mi = index.Writer(indexfile)
- merge_indexes(mi, ri, wr)
+
+ for e in index.merge(ri, wr):
+ # FIXME: shouldn't we remove deleted entries eventually? When?
+ mi.add_ixentry(e)
+
ri.close()
mi.close()
wr.close()
fake-valid mark all index entries as up-to-date even if they aren't
fake-invalid mark all index entries as invalid
check carefully check index file integrity
-f,indexfile= the name of the index file (default 'index')
+f,indexfile= the name of the index file (normally BUP_DIR/bupindex)
+exclude= a path to exclude from the backup (can be used more than once)
+exclude-from= a file that contains exclude paths (can be used more than once)
v,verbose increase log output (can be used more than once)
"""
-o = options.Options('bup index', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if not (opt.modified or opt['print'] or opt.status or opt.update or opt.check):
log('check: starting initial check.\n')
check_index(index.Reader(indexfile))
+excluded_paths = drecurse.parse_excludes(flags)
+
paths = index.reduce_paths(extra)
if opt.update:
if not extra:
o.fatal('update (-u) requested but no paths given')
for (rp,path) in paths:
- update_index(rp)
+ update_index(rp, excluded_paths)
if opt['print'] or opt.status or opt.modified:
for (name, ent) in index.Reader(indexfile).filter(extra or ['']):
#!/usr/bin/env python
+import sys
+
from bup import git, options, client
from bup.helpers import *
+
optspec = """
[BUP_DIR=...] bup init [-r host:path]
--
r,remote= remote repository path
"""
-o = options.Options('bup init', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
o.fatal("no arguments expected")
-if opt.remote:
- if opt.remote and opt.remote.find(":") == -1:
- o.fatal("--remote argument must contain a colon")
+try:
git.init_repo() # local repo
+except git.GitError, e:
+ log("bup: error: could not init repository: %s" % e)
+ sys.exit(1)
+
+if opt.remote:
git.check_repo_or_die()
- try:
- cli = client.Client(opt.remote, create=True)
- except client.ClientError:
- o.fatal("server exited unexpectedly; see errors above")
+ cli = client.Client(opt.remote, create=True)
cli.close()
-else:
- git.init_repo()
--
r,remote= remote repository path
"""
-o = options.Options('bup join', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
git.check_repo_or_die()
ret = 0
if opt.remote:
- if opt.remote and opt.remote.find(":") == -1:
- o.fatal("--remote argument must contain a colon")
- try:
- cli = client.Client(opt.remote)
- except client.ClientError:
- o.fatal("server exited unexpectedly; see errors above")
+ cli = client.Client(opt.remote)
cat = cli.cat
else:
cp = git.CatPipe()
bup ls <dirs...>
--
s,hash show hash for each file
+a,all show hidden files
"""
-o = options.Options('bup ls', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
git.check_repo_or_die()
n = top.lresolve(d)
if stat.S_ISDIR(n.mode):
for sub in n:
- print_node(sub.name, sub)
+ if opt.all or not sub.name.startswith('.'):
+ print_node(sub.name, sub)
else:
- print_node(d, n)
+ if opt.all or not sub.name.startswith('.'):
+ print_node(d, n)
except vfs.NodeError, e:
log('error: %s\n' % e)
ret = 1
predict Guess object offsets and report the maximum deviation
ignore-midx Don't use midx files; use only plain pack idx files.
"""
-o = options.Options('bup margin', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
#!/usr/bin/env python
-import sys, re, struct, mmap, time, resource
-from bup import git, options
+import sys, re, struct, time, resource
+from bup import git, options, _helpers
from bup.helpers import *
handle_ctrl_c()
-def s_from_bytes(bytes):
- clist = [chr(b) for b in bytes]
- return ''.join(clist)
-
-
_linux_warned = 0
def linux_memstat():
global _linux_warned
_linux_warned = 1
return {}
for line in f:
- k,v = re.split(r':\s*', line.strip(), 1)
- d[k] = v
+ # Note that on Solaris, this file exists but is binary. If that
+ # happens, this split() might not return two elements. We don't
+ # really need to care about the binary format since this output
+ # isn't used for much and report() can deal with missing entries.
+ t = re.split(r':\s*', line.strip(), 1)
+ if len(t) == 2:
+ k,v = t
+ d[k] = v
return d
ignore-midx ignore .midx files, use only .idx files
existing test with existing objects instead of fake ones
"""
-o = options.Options('bup memtest', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
m = git.PackIdxList(git.repo('objects/pack'))
report(-1)
-f = open('/dev/urandom')
-a = mmap.mmap(-1, 20)
+_helpers.random_sha()
report(0)
if opt.existing:
bin = objit.next()
assert(m.exists(bin))
else:
- b = f.read(3)
- a[0:2] = b[0:2]
- a[2] = chr(ord(b[2]) & 0xf0)
- bin = str(a[0:20])
+ bin = _helpers.random_sha()
# technically, a randomly generated object id might exist.
# but the likelihood of that is the likelihood of finding
handle_ctrl_c()
-o = options.Options('bup meta', optspec)
+o = options.Options(optspec)
(opt, flags, remainder) = o.parse(sys.argv[1:])
for flag, value in flags:
#!/usr/bin/env python
import sys, math, struct, glob, resource
-from bup import options, git
+import tempfile
+from bup import options, git, _helpers
from bup.helpers import *
PAGE_SIZE=4096
SHA_PER_PAGE=PAGE_SIZE/20.
+SEEK_END=2 # os.SEEK_END is not defined in python 2.4
optspec = """
bup midx [options...] <idxnames...>
mf -= 6 # minimum safety margin
return mf
-
-def merge(idxlist, bits, table):
- count = 0
- for e in git.idxmerge(idxlist, final_progress=False):
- count += 1
- prefix = git.extract_bits(e, bits)
- table[prefix] = count
- yield e
-
+merge_into = _helpers.merge_into
def _do_midx(outdir, outfilename, infilenames, prefixstr):
if not outfilename:
inp = []
total = 0
- allfilenames = {}
+ allfilenames = []
for name in infilenames:
ix = git.open_idx(name)
+ inp.append((
+ ix.map,
+ len(ix),
+ ix.sha_ofs,
+ isinstance(ix, git.PackMidx) and ix.idxname_ofs or 0,
+ len(allfilenames),
+ ))
for n in ix.idxnames:
- allfilenames[n] = 1
- inp.append(ix)
+ allfilenames.append(os.path.basename(n))
total += len(ix)
+ inp.sort(lambda x,y: cmp(str(y[0][y[2]:y[2]+20]),str(x[0][x[2]:x[2]+20])))
- log('midx: %smerging %d indexes (%d objects).\n'
+ log('midx: %screating from %d files (%d objects).\n'
% (prefixstr, len(infilenames), total))
if (not opt.force and (total < 1024 and len(infilenames) < 3)) \
or len(infilenames) < 2 \
entries = 2**bits
debug1('midx: table size: %d (%d bits)\n' % (entries*4, bits))
- table = [0]*entries
-
try:
os.unlink(outfilename)
except OSError:
pass
- f = open(outfilename + '.tmp', 'w+')
- f.write('MIDX\0\0\0\2')
- f.write(struct.pack('!I', bits))
+ f = open(outfilename + '.tmp', 'w+b')
+ f.write('MIDX')
+ f.write(struct.pack('!II', git.MIDX_VERSION, bits))
assert(f.tell() == 12)
- f.write('\0'*4*entries)
-
- for e in merge(inp, bits, table):
- f.write(e)
-
- f.write('\0'.join(os.path.basename(p) for p in allfilenames.keys()))
- f.seek(12)
- f.write(struct.pack('!%dI' % entries, *table))
+ f.truncate(12 + 4*entries + 20*total + 4*total)
+
+ fmap = mmap_readwrite(f, close=False)
+
+ count = merge_into(fmap, bits, total, inp)
+ fmap.flush()
+ fmap.close()
+
+ f.seek(0, SEEK_END)
+ f.write('\0'.join(allfilenames))
f.close()
os.rename(outfilename + '.tmp', outfilename)
assert(len(p.idxnames) == len(infilenames))
print p.idxnames
assert(len(p) == total)
- pi = iter(p)
- for i in merge(inp, total, bits, table):
+ for pe, e in p, git.idxmerge(inp, final_progress=False):
assert(i == pi.next())
assert(p.exists(i))
- return total,outfilename
+ return total, outfilename
def do_midx(outdir, outfilename, infilenames, prefixstr):
handle_ctrl_c()
-o = options.Options('bup midx', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra and (opt.auto or opt.force):
--- /dev/null
+#!/usr/bin/env python
+import os, sys, subprocess, struct
+from bup import options
+from bup.helpers import *
+
+optspec = """
+bup mux command [command arguments...]
+--
+"""
+o = options.Options(optspec)
+(opt, flags, extra) = o.parse(sys.argv[1:])
+if len(extra) < 1:
+ o.fatal('command is required')
+
+cmdpath, cmdfn = os.path.split(__file__)
+subcmd = extra
+subcmd[0] = os.path.join(cmdpath, 'bup-' + subcmd[0])
+
+debug2('bup mux: starting %r\n' % (extra,))
+
+outr, outw = os.pipe()
+errr, errw = os.pipe()
+def close_fds():
+ os.close(outr)
+ os.close(errr)
+p = subprocess.Popen(subcmd, stdout=outw, stderr=errw, preexec_fn=close_fds)
+os.close(outw)
+os.close(errw)
+sys.stdout.write('BUPMUX')
+sys.stdout.flush()
+mux(p, sys.stdout.fileno(), outr, errr)
+os.close(outr)
+os.close(errr)
+prv = p.wait()
+
+if prv:
+ debug1('%s exited with code %d\n' % (extra[0], prv))
+
+debug1('bup mux: done\n')
+
+sys.exit(prv)
optspec = """
bup newliner
"""
-o = options.Options('bup newliner', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
--
This command is run automatically by 'bup on'
"""
-o = options.Options('bup server-reverse', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
o.fatal('no arguments expected')
#!/usr/bin/env python
import sys, os, struct, getopt, subprocess, signal
-from bup import options, ssh
+from bup import options, ssh, path
from bup.helpers import *
optspec = """
bup on <hostname> save ...
bup on <hostname> split ...
"""
-o = options.Options('bup on', optspec, optfunc=getopt.getopt)
+o = options.Options(optspec, optfunc=getopt.getopt)
(opt, flags, extra) = o.parse(sys.argv[1:])
if len(extra) < 2:
o.fatal('arguments expected')
ret = 99
try:
- hostname = extra[0]
+ hp = extra[0].split(':')
+ if len(hp) == 1:
+ (hostname, port) = (hp[0], None)
+ else:
+ (hostname, port) = hp
+
argv = extra[1:]
- p = ssh.connect(hostname, 'on--server')
+ p = ssh.connect(hostname, port, 'on--server')
argvs = '\0'.join(['bup'] + argv)
p.stdin.write(struct.pack('!I', len(argvs)) + argvs)
p.stdin.flush()
- main_exe = os.environ.get('BUP_MAIN_EXE') or sys.argv[0]
- sp = subprocess.Popen([main_exe, 'server'], stdin=p.stdout, stdout=p.stdin)
-
+ sp = subprocess.Popen([path.exe(), 'server'],
+ stdin=p.stdout, stdout=p.stdin)
p.stdin.close()
p.stdout.close()
--
S,seed= optional random number seed [1]
f,force print random data to stdout even if it's a tty
+v,verbose print byte counter to stderr
"""
-o = options.Options('bup random', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if len(extra) != 1:
if opt.force or (not os.isatty(1) and
not atoi(os.environ.get('BUP_FORCE_TTY')) & 1):
- _helpers.write_random(sys.stdout.fileno(), total, opt.seed)
+ _helpers.write_random(sys.stdout.fileno(), total, opt.seed,
+ opt.verbose and 1 or 0)
else:
log('error: not writing binary data to a terminal. Use -f to force.\n')
sys.exit(1)
handle_ctrl_c()
-o = options.Options('bup restore', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
git.check_repo_or_die()
q,quiet don't show progress meter
smaller= only back up files smaller than n bytes
bwlimit= maximum bytes/sec to transmit to server
+f,indexfile= the name of the index file (normally BUP_DIR/bupindex)
+strip strips the path to every filename given
+strip-path= path-prefix to be stripped when saving
+graft= a graft point *old_path*=*new_path* (can be used morethan once)
"""
-o = options.Options('bup save', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
git.check_repo_or_die()
else:
date = time.time()
+if opt.strip and opt.strip_path:
+ o.fatal("--strip is incompatible with --strip-path")
+
+graft_points = []
+if opt.graft:
+ if opt.strip:
+ o.fatal("--strip is incompatible with --graft")
+
+ if opt.strip_path:
+ o.fatal("--strip-path is incompatible with --graft")
+
+ for (option, parameter) in flags:
+ if option == "--graft":
+ splitted_parameter = parameter.split('=')
+ if len(splitted_parameter) != 2:
+ o.fatal("a graft point must be of the form old_path=new_path")
+ graft_points.append((realpath(splitted_parameter[0]),
+ realpath(splitted_parameter[1])))
+
is_reverse = os.environ.get('BUP_SERVER_REVERSE')
if is_reverse and opt.remote:
o.fatal("don't use -r in reverse mode; it's automatic")
+if opt.name and opt.name.startswith('.'):
+ o.fatal("'%s' is not a valid branch name" % opt.name)
refname = opt.name and 'refs/heads/%s' % opt.name or None
if opt.remote or is_reverse:
- if opt.remote and opt.remote.find(":") == -1:
- o.fatal("--remote argument must contain a colon")
- try:
- cli = client.Client(opt.remote)
- except client.ClientError:
- o.fatal("server exited unexpectedly; see errors above")
+ cli = client.Client(opt.remote)
oldref = refname and cli.read_ref(refname) or None
w = cli.new_packwriter()
else:
shalist = shalists.pop()
tree = force_tree or w.new_tree(shalist)
if shalists:
- shalists[-1].append(('40000', part, tree))
+ shalists[-1].append(('40000',
+ git.mangle_name(part, 040000, 40000),
+ tree))
else: # this was the toplevel, so put it back for sanity
shalists.append(shalist)
return tree
log(s)
-r = index.Reader(git.repo('bupindex'))
+indexfile = opt.indexfile or git.repo('bupindex')
+r = index.Reader(indexfile)
def already_saved(ent):
return ent.is_valid() and w.exists(ent.sha) and ent.sha
continue
assert(dir.startswith('/'))
- dirp = dir.split('/')
+ if opt.strip:
+ stripped_base_path = strip_base_path(dir, extra)
+ dirp = stripped_base_path.split('/')
+ elif opt.strip_path:
+ dirp = strip_path(opt.strip_path, dir).split('/')
+ elif graft_points:
+ grafted = graft_path(graft_points, dir)
+ dirp = grafted.split('/')
+ else:
+ dirp = dir.split('/')
while parts > dirp:
_pop(force_tree = None)
if dir != '/':
add_error(e)
lastskip_name = ent.name
else:
- (mode, id) = hashsplit.split_to_blob_or_tree(w, [f])
+ try:
+ (mode, id) = hashsplit.split_to_blob_or_tree(w, [f],
+ keep_boundaries=False)
+ except IOError, e:
+ add_error('%s: %s' % (ent.name, e))
+ lastskip_name = ent.name
else:
if stat.S_ISDIR(ent.mode):
assert(0) # handled above
print tree.encode('hex')
if opt.commit or opt.name:
msg = 'bup save\n\nGenerated by command:\n%r' % sys.argv
- ref = opt.name and ('refs/heads/%s' % opt.name) or None
commit = w.new_commit(oldref, tree, date, msg)
if opt.commit:
print commit.encode('hex')
#!/usr/bin/env python
-import sys, struct
+import os, sys, struct
from bup import options, git
from bup.helpers import *
suspended_w = None
+dumb_server_mode = False
+
+def _set_mode():
+ global dumb_server_mode
+ dumb_server_mode = os.path.exists(git.repo('bup-dumb-server'))
+ debug1('bup server: serving in %s mode\n'
+ % (dumb_server_mode and 'dumb' or 'smart'))
def init_dir(conn, arg):
git.init_repo(arg)
debug1('bup server: bupdir initialized: %r\n' % git.repodir)
+ _set_mode()
conn.ok()
def set_dir(conn, arg):
git.check_repo_or_die(arg)
debug1('bup server: bupdir is %r\n' % git.repodir)
+ _set_mode()
conn.ok()
def list_indexes(conn, junk):
git.check_repo_or_die()
+ suffix = ''
+ if dumb_server_mode:
+ suffix = ' load'
for f in os.listdir(git.repo('objects/pack')):
if f.endswith('.idx'):
- conn.write('%s\n' % f)
+ conn.write('%s%s\n' % (f, suffix))
conn.ok()
git.check_repo_or_die()
assert(name.find('/') < 0)
assert(name.endswith('.idx'))
- idx = git.PackIdx(git.repo('objects/pack/%s' % name))
+ idx = git.open_idx(git.repo('objects/pack/%s' % name))
conn.write(struct.pack('!I', len(idx.map)))
conn.write(idx.map)
conn.ok()
-def receive_objects(conn, junk):
+def receive_objects_v2(conn, junk):
global suspended_w
git.check_repo_or_die()
- suggested = {}
+ suggested = set()
if suspended_w:
w = suspended_w
suspended_w = None
else:
- w = git.PackWriter()
+ if dumb_server_mode:
+ w = git.PackWriter(objcache_maker=None)
+ else:
+ w = git.PackWriter()
while 1:
ns = conn.read(4)
if not ns:
if not n:
debug1('bup server: received %d object%s.\n'
% (w.count, w.count!=1 and "s" or ''))
- fullpath = w.close()
+ fullpath = w.close(run_midx=not dumb_server_mode)
if fullpath:
(dir, name) = os.path.split(fullpath)
conn.write('%s.idx\n' % name)
conn.ok()
return
+ shar = conn.read(20)
+ crcr = struct.unpack('!I', conn.read(4))[0]
+ n -= 20 + 4
buf = conn.read(n) # object sizes in bup are reasonably small
#debug2('read %d bytes\n' % n)
- if len(buf) < n:
- w.abort()
- raise Exception('object read: expected %d bytes, got %d\n'
- % (n, len(buf)))
- (type, content) = git._decode_packobj(buf)
- sha = git.calc_hash(type, content)
- oldpack = w.exists(sha)
- # FIXME: we only suggest a single index per cycle, because the client
- # is currently dumb to download more than one per cycle anyway.
- # Actually we should fix the client, but this is a minor optimization
- # on the server side.
- if not suggested and \
- oldpack and (oldpack == True or oldpack.endswith('.midx')):
- # FIXME: we shouldn't really have to know about midx files
- # at this layer. But exists() on a midx doesn't return the
- # packname (since it doesn't know)... probably we should just
- # fix that deficiency of midx files eventually, although it'll
- # make the files bigger. This method is certainly not very
- # efficient.
- w.objcache.refresh(skip_midx = True)
- oldpack = w.objcache.exists(sha)
- debug2('new suggestion: %r\n' % oldpack)
- assert(oldpack)
- assert(oldpack != True)
- assert(not oldpack.endswith('.midx'))
- w.objcache.refresh(skip_midx = False)
- if not suggested and oldpack:
- assert(oldpack.endswith('.idx'))
- (dir,name) = os.path.split(oldpack)
- if not (name in suggested):
- debug1("bup server: suggesting index %s\n" % name)
- conn.write('index %s\n' % name)
- suggested[name] = 1
- else:
- w._raw_write([buf])
+ _check(w, n, len(buf), 'object read: expected %d bytes, got %d\n')
+ if not dumb_server_mode:
+ oldpack = w.exists(shar, want_source=True)
+ if oldpack:
+ assert(not oldpack == True)
+ assert(oldpack.endswith('.idx'))
+ (dir,name) = os.path.split(oldpack)
+ if not (name in suggested):
+ debug1("bup server: suggesting index %s\n" % name)
+ conn.write('index %s\n' % name)
+ suggested.add(name)
+ continue
+ nw, crc = w._raw_write((buf,), sha=shar)
+ _check(w, crcr, crc, 'object read: expected crc %d, got %d\n')
# NOTREACHED
+
+
+def _check(w, expected, actual, msg):
+ if expected != actual:
+ w.abort()
+ raise Exception(msg % (expected, actual))
def read_ref(conn, refname):
optspec = """
bup server
"""
-o = options.Options('bup server', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
'set-dir': set_dir,
'list-indexes': list_indexes,
'send-index': send_index,
- 'receive-objects': receive_objects,
+ 'receive-objects-v2': receive_objects_v2,
'read-ref': read_ref,
'update-ref': update_ref,
'cat': cat,
d,date= date for the commit (seconds since the epoch)
q,quiet don't print progress messages
v,verbose increase log output (can be used more than once)
+git-ids read a list of git object ids from stdin and split their contents
+keep-boundaries don't let one chunk span two input files
noop don't actually save the data anywhere
copy just copy input to output, hashsplitting along the way
bench print benchmark timings to stderr
fanout= maximum number of blobs in a single tree
bwlimit= maximum bytes/sec to transmit to server
"""
-o = options.Options('bup split', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
handle_ctrl_c()
if (opt.noop or opt.copy) and (opt.blobs or opt.tree or
opt.commit or opt.name):
o.fatal('-N and --copy are incompatible with -b, -t, -c, -n')
+if extra and opt.git_ids:
+ o.fatal("don't provide filenames when using --git-ids")
if opt.verbose >= 2:
git.verbose = opt.verbose - 1
date = time.time()
+last_prog = total_bytes = 0
+def prog(filenum, nbytes):
+ global last_prog, total_bytes
+ total_bytes += nbytes
+ now = time.time()
+ if now - last_prog < 0.2:
+ return
+ if filenum > 0:
+ progress('Splitting: file #%d, %d kbytes\r'
+ % (filenum+1, total_bytes/1024))
+ else:
+ progress('Splitting: %d kbytes\r' % (total_bytes/1024))
+ last_prog = now
+
+
is_reverse = os.environ.get('BUP_SERVER_REVERSE')
if is_reverse and opt.remote:
o.fatal("don't use -r in reverse mode; it's automatic")
start_time = time.time()
+if opt.name and opt.name.startswith('.'):
+ o.fatal("'%s' is not a valid branch name." % opt.name)
refname = opt.name and 'refs/heads/%s' % opt.name or None
if opt.noop or opt.copy:
cli = pack_writer = oldref = None
elif opt.remote or is_reverse:
- if opt.remote and opt.remote.find(":") == -1:
- o.fatal("--remote argument must contain a colon")
- try:
- cli = client.Client(opt.remote)
- except client.ClientError:
- o.fatal("server exited unexpectedly; see errors above")
+ cli = client.Client(opt.remote)
oldref = refname and cli.read_ref(refname) or None
pack_writer = cli.new_packwriter()
else:
oldref = refname and git.read_ref(refname) or None
pack_writer = git.PackWriter()
-files = extra and (open(fn) for fn in extra) or [sys.stdin]
+if opt.git_ids:
+ # the input is actually a series of git object ids that we should retrieve
+ # and split.
+ #
+ # This is a bit messy, but basically it converts from a series of
+ # CatPipe.get() iterators into a series of file-type objects.
+ # It would be less ugly if either CatPipe.get() returned a file-like object
+ # (not very efficient), or split_to_shalist() expected an iterator instead
+ # of a file.
+ cp = git.CatPipe()
+ class IterToFile:
+ def __init__(self, it):
+ self.it = iter(it)
+ def read(self, size):
+ v = next(self.it)
+ return v or ''
+ def read_ids():
+ while 1:
+ line = sys.stdin.readline()
+ if not line:
+ break
+ if line:
+ line = line.strip()
+ try:
+ it = cp.get(line.strip())
+ next(it) # skip the file type
+ except KeyError, e:
+ add_error('error: %s' % e)
+ continue
+ yield IterToFile(it)
+ files = read_ids()
+else:
+ # the input either comes from a series of files or from stdin.
+ files = extra and (open(fn) for fn in extra) or [sys.stdin]
+
if pack_writer:
- shalist = hashsplit.split_to_shalist(pack_writer, files)
+ shalist = hashsplit.split_to_shalist(pack_writer, files,
+ keep_boundaries=opt.keep_boundaries,
+ progress=prog)
tree = pack_writer.new_tree(shalist)
else:
last = 0
- for (blob, bits) in hashsplit.hashsplit_iter(files):
+ for (blob, bits) in hashsplit.hashsplit_iter(files,
+ keep_boundaries=opt.keep_boundaries,
+ progress=prog):
hashsplit.total_split += len(blob)
if opt.copy:
sys.stdout.write(str(blob))
if opt.bench:
log('\nbup: %.2fkbytes in %.2f secs = %.2f kbytes/sec\n'
% (size/1024., secs, size/1024./secs))
+
+if saved_errors:
+ log('WARNING: %d errors encountered while saving.\n' % len(saved_errors))
+ sys.exit(1)
--- /dev/null
+#!/usr/bin/env python
+"""Tag a commit in the bup repository.
+Creating a tag on a commit can be used for avoiding automatic cleanup from
+removing this commit due to old age.
+"""
+import sys
+import os
+
+from bup import git, options
+from bup.helpers import *
+
+
+handle_ctrl_c()
+
+optspec = """
+bup tag
+bup tag <tag name> <commit>
+bup tag -d <tag name>
+--
+d,delete= Delete a tag
+"""
+
+o = options.Options(optspec)
+(opt, flags, extra) = o.parse(sys.argv[1:])
+
+git.check_repo_or_die()
+
+if opt.delete:
+ tag_file = git.repo('refs/tags/%s' % opt.delete)
+ debug1("tag file: %s\n" % tag_file)
+ if not os.path.exists(tag_file):
+ log("bup: error: tag '%s' not found." % opt.delete)
+ sys.exit(1)
+
+ try:
+ os.unlink(tag_file)
+ except OSError, e:
+ log("bup: error: unable to delete tag '%s': %s" % (opt.delete, e))
+ sys.exit(1)
+
+ sys.exit(0)
+
+tags = [t for sublist in git.tags().values() for t in sublist]
+
+if not extra:
+ for t in tags:
+ print t
+ sys.exit(0)
+elif len(extra) < 2:
+ o.fatal('no commit ref or hash given.')
+
+(tag_name, commit) = extra[:2]
+if not tag_name:
+ o.fatal("tag name must not be empty.")
+debug1("args: tag name = %s; commit = %s\n" % (tag_name, commit))
+
+if tag_name in tags:
+ log("bup: error: tag '%s' already exists" % tag_name)
+ sys.exit(1)
+
+if tag_name.startswith('.'):
+ o.fatal("'%s' is not a valid tag name." % tag_name)
+
+try:
+ hash = git.rev_parse(commit)
+except git.GitError, e:
+ log("bup: error: %s" % e)
+ sys.exit(2)
+
+if not hash:
+ log("bup: error: commit %s not found." % commit)
+ sys.exit(2)
+
+pL = git.PackIdxList(git.repo('objects/pack'))
+if not pL.exists(hash):
+ log("bup: error: commit %s not found." % commit)
+ sys.exit(2)
+
+tag_file = git.repo('refs/tags/%s' % tag_name)
+try:
+ tag = file(tag_file, 'w')
+except OSError, e:
+ log("bup: error: could not create tag '%s': %s" % (tag_name, e))
+ sys.exit(3)
+
+tag.write(hash.encode('hex'))
+tag.close()
optspec = """
bup tick
"""
-o = options.Options('bup tick', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if extra:
commit display the git commit id of this version of bup
tag display the tag name of this version. If no tag is available, display the commit id
"""
-o = options.Options('bup version', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
return False
-def _compute_dir_contents(n, show_hidden=False):
+def _compute_dir_contents(n, path, show_hidden=False):
"""Given a vfs node, returns an iterator for display info of all subs."""
+ url_append = ""
+ if show_hidden:
+ url_append = "?hidden=1"
+
+ if path != "/":
+ yield('..', '../' + url_append, '')
for sub in n:
display = link = sub.name
if not show_hidden and len(display)>1 and display.startswith('.'):
continue
- url_append = ""
- if show_hidden:
- url_append = "?hidden=1"
-
size = None
if stat.S_ISDIR(sub.mode):
display = sub.name + '/'
def head(self, path):
return self._process_request(path)
-
+
+ @tornado.web.asynchronous
def _process_request(self, path):
path = urllib.unquote(path)
print 'Handling request for %s' % path
breadcrumbs=_compute_breadcrumbs(path, show_hidden),
files_hidden=_contains_hidden_files(n),
hidden_shown=show_hidden,
- dir_contents=_compute_dir_contents(n, show_hidden))
+ dir_contents=_compute_dir_contents(n, path, show_hidden))
def _get_file(self, path, n):
"""Process a request on a file.
self.set_header("Content-Type", ctype)
size = n.size()
self.set_header("Content-Length", str(size))
+ assert(len(n.hash) == 20)
+ self.set_header("Etag", n.hash.encode('hex'))
if self.request.method != 'HEAD':
+ self.flush()
f = n.open()
- for blob in chunkyreader(f):
- self.write(blob)
- f.close()
+ it = chunkyreader(f)
+ def write_more(me):
+ try:
+ blob = it.next()
+ except StopIteration:
+ f.close()
+ self.finish()
+ return
+ self.request.connection.stream.write(blob,
+ callback=lambda: me(me))
+ write_more(write_more)
def _guess_type(self, path):
"""Guess the type of a file.
bup web [[hostname]:port]
--
"""
-o = options.Options('bup web', optspec)
+o = options.Options(optspec)
(opt, flags, extra) = o.parse(sys.argv[1:])
if len(extra) > 1:
handle_ctrl_c()
-o = options.Options('bup pathinfo', optspec)
+o = options.Options(optspec)
(opt, flags, remainder) = o.parse(sys.argv[1:])
treat_include_fields_as_definitive = True
#include <fcntl.h>
#include <arpa/inet.h>
#include <stdint.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <stdio.h>
#ifdef linux
#include <linux/fs.h>
#include <sys/time.h>
#endif
+static int istty = 0;
static PyObject *selftest(PyObject *self, PyObject *args)
{
return NULL;
v = ntohl(*(uint32_t *)buf);
- return Py_BuildValue("I", v);
+ return PyLong_FromUnsignedLong(v);
}
+typedef struct {
+ uint32_t high;
+ unsigned char low;
+} bits40_t;
+
+
+static void to_bloom_address_bitmask4(const bits40_t *buf,
+ const int nbits, uint64_t *v, unsigned char *bitmask)
+{
+ int bit;
+ uint64_t raw, mask;
+
+ mask = (1<<nbits) - 1;
+ raw = (((uint64_t)ntohl(buf->high)) << 8) | buf->low;
+ bit = (raw >> (37-nbits)) & 0x7;
+ *v = (raw >> (40-nbits)) & mask;
+ *bitmask = 1 << bit;
+}
+
+static void to_bloom_address_bitmask5(const uint32_t *buf,
+ const int nbits, uint32_t *v, unsigned char *bitmask)
+{
+ int bit;
+ uint32_t raw, mask;
+
+ mask = (1<<nbits) - 1;
+ raw = ntohl(*buf);
+ bit = (raw >> (29-nbits)) & 0x7;
+ *v = (raw >> (32-nbits)) & mask;
+ *bitmask = 1 << bit;
+}
+
+
+#define BLOOM_SET_BIT(name, address, itype, otype) \
+static void name(unsigned char *bloom, const void *buf, const int nbits)\
+{\
+ unsigned char bitmask;\
+ otype v;\
+ address((itype *)buf, nbits, &v, &bitmask);\
+ bloom[16+v] |= bitmask;\
+}
+BLOOM_SET_BIT(bloom_set_bit4, to_bloom_address_bitmask4, bits40_t, uint64_t)
+BLOOM_SET_BIT(bloom_set_bit5, to_bloom_address_bitmask5, uint32_t, uint32_t)
+
+
+#define BLOOM_GET_BIT(name, address, itype, otype) \
+static int name(const unsigned char *bloom, const void *buf, const int nbits)\
+{\
+ unsigned char bitmask;\
+ otype v;\
+ address((itype *)buf, nbits, &v, &bitmask);\
+ return bloom[16+v] & bitmask;\
+}
+BLOOM_GET_BIT(bloom_get_bit4, to_bloom_address_bitmask4, bits40_t, uint64_t)
+BLOOM_GET_BIT(bloom_get_bit5, to_bloom_address_bitmask5, uint32_t, uint32_t)
+
+
+static PyObject *bloom_add(PyObject *self, PyObject *args)
+{
+ unsigned char *sha = NULL, *bloom = NULL;
+ unsigned char *end;
+ int len = 0, blen = 0, nbits = 0, k = 0;
+
+ if (!PyArg_ParseTuple(args, "w#s#ii", &bloom, &blen, &sha, &len, &nbits, &k))
+ return NULL;
+
+ if (blen < 16+(1<<nbits) || len % 20 != 0)
+ return NULL;
+
+ if (k == 5)
+ {
+ if (nbits > 29)
+ return NULL;
+ for (end = sha + len; sha < end; sha += 20/k)
+ bloom_set_bit5(bloom, sha, nbits);
+ }
+ else if (k == 4)
+ {
+ if (nbits > 37)
+ return NULL;
+ for (end = sha + len; sha < end; sha += 20/k)
+ bloom_set_bit4(bloom, sha, nbits);
+ }
+ else
+ return NULL;
+
+
+ return Py_BuildValue("i", len/20);
+}
+
+static PyObject *bloom_contains(PyObject *self, PyObject *args)
+{
+ unsigned char *sha = NULL, *bloom = NULL;
+ int len = 0, blen = 0, nbits = 0, k = 0;
+ unsigned char *end;
+ int steps;
+
+ if (!PyArg_ParseTuple(args, "t#s#ii", &bloom, &blen, &sha, &len, &nbits, &k))
+ return NULL;
+
+ if (len != 20)
+ return NULL;
+
+ if (k == 5)
+ {
+ if (nbits > 29)
+ return NULL;
+ for (steps = 1, end = sha + 20; sha < end; sha += 20/k, steps++)
+ if (!bloom_get_bit5(bloom, sha, nbits))
+ return Py_BuildValue("Oi", Py_None, steps);
+ }
+ else if (k == 4)
+ {
+ if (nbits > 37)
+ return NULL;
+ for (steps = 1, end = sha + 20; sha < end; sha += 20/k, steps++)
+ if (!bloom_get_bit4(bloom, sha, nbits))
+ return Py_BuildValue("Oi", Py_None, steps);
+ }
+ else
+ return NULL;
+
+ return Py_BuildValue("Oi", Py_True, k);
+}
+
+
+static uint32_t _extract_bits(unsigned char *buf, int nbits)
+{
+ uint32_t v, mask;
+
+ mask = (1<<nbits) - 1;
+ v = ntohl(*(uint32_t *)buf);
+ v = (v >> (32-nbits)) & mask;
+ return v;
+}
static PyObject *extract_bits(PyObject *self, PyObject *args)
{
unsigned char *buf = NULL;
int len = 0, nbits = 0;
- uint32_t v, mask;
if (!PyArg_ParseTuple(args, "t#i", &buf, &len, &nbits))
return NULL;
if (len < 4)
return NULL;
- mask = (1<<nbits) - 1;
- v = ntohl(*(uint32_t *)buf);
- v = (v >> (32-nbits)) & mask;
- return Py_BuildValue("I", v);
+ return PyLong_FromUnsignedLong(_extract_bits(buf, nbits));
+}
+
+
+struct sha {
+ unsigned char bytes[20];
+};
+struct idx {
+ unsigned char *map;
+ struct sha *cur;
+ struct sha *end;
+ uint32_t *cur_name;
+ long bytes;
+ int name_base;
+};
+
+
+static int _cmp_sha(const struct sha *sha1, const struct sha *sha2)
+{
+ int i;
+ for (i = 0; i < 20; i++)
+ if (sha1->bytes[i] != sha2->bytes[i])
+ return sha1->bytes[i] - sha2->bytes[i];
+ return 0;
+}
+
+
+static void _fix_idx_order(struct idx **idxs, int *last_i)
+{
+ struct idx *idx;
+ int low, mid, high, c = 0;
+
+ idx = idxs[*last_i];
+ if (idxs[*last_i]->cur >= idxs[*last_i]->end)
+ {
+ idxs[*last_i] = NULL;
+ PyMem_Free(idx);
+ --*last_i;
+ return;
+ }
+ if (*last_i == 0)
+ return;
+
+ low = *last_i-1;
+ mid = *last_i;
+ high = 0;
+ while (low >= high)
+ {
+ mid = (low + high) / 2;
+ c = _cmp_sha(idx->cur, idxs[mid]->cur);
+ if (c < 0)
+ high = mid + 1;
+ else if (c > 0)
+ low = mid - 1;
+ else
+ break;
+ }
+ if (c < 0)
+ ++mid;
+ if (mid == *last_i)
+ return;
+ memmove(&idxs[mid+1], &idxs[mid], (*last_i-mid)*sizeof(struct idx *));
+ idxs[mid] = idx;
+}
+
+
+static uint32_t _get_idx_i(struct idx *idx)
+{
+ if (idx->cur_name == NULL)
+ return idx->name_base;
+ return ntohl(*idx->cur_name) + idx->name_base;
+}
+
+
+static PyObject *merge_into(PyObject *self, PyObject *args)
+{
+ PyObject *ilist = NULL;
+ unsigned char *fmap = NULL;
+ struct sha *sha_ptr, *last = NULL;
+ uint32_t *table_ptr, *name_ptr;
+ struct idx **idxs = NULL;
+ int flen = 0, bits = 0, i;
+ uint32_t total, count, prefix;
+ int num_i;
+ int last_i;
+
+ if (!PyArg_ParseTuple(args, "w#iIO", &fmap, &flen, &bits, &total, &ilist))
+ return NULL;
+
+ num_i = PyList_Size(ilist);
+ idxs = (struct idx **)PyMem_Malloc(num_i * sizeof(struct idx *));
+
+ for (i = 0; i < num_i; i++)
+ {
+ long len, sha_ofs, name_map_ofs;
+ idxs[i] = (struct idx *)PyMem_Malloc(sizeof(struct idx));
+ PyObject *itup = PyList_GetItem(ilist, i);
+ if (!PyArg_ParseTuple(itup, "t#llli", &idxs[i]->map, &idxs[i]->bytes,
+ &len, &sha_ofs, &name_map_ofs, &idxs[i]->name_base))
+ return NULL;
+ idxs[i]->cur = (struct sha *)&idxs[i]->map[sha_ofs];
+ idxs[i]->end = &idxs[i]->cur[len];
+ if (name_map_ofs)
+ idxs[i]->cur_name = (uint32_t *)&idxs[i]->map[name_map_ofs];
+ else
+ idxs[i]->cur_name = NULL;
+ }
+ table_ptr = (uint32_t *)&fmap[12];
+ sha_ptr = (struct sha *)&table_ptr[1<<bits];
+ name_ptr = (uint32_t *)&sha_ptr[total];
+
+ last_i = num_i-1;
+ count = 0;
+ prefix = 0;
+ while (last_i >= 0)
+ {
+ struct idx *idx;
+ uint32_t new_prefix;
+ if (count % 102424 == 0 && istty)
+ fprintf(stderr, "midx: writing %.2f%% (%d/%d)\r",
+ count*100.0/total, count, total);
+ idx = idxs[last_i];
+ new_prefix = _extract_bits((unsigned char *)idx->cur, bits);
+ while (prefix < new_prefix)
+ table_ptr[prefix++] = htonl(count);
+ if (last == NULL || _cmp_sha(last, idx->cur) != 0)
+ {
+ memcpy(sha_ptr++, idx->cur, 20);
+ *name_ptr++ = htonl(_get_idx_i(idx));
+ last = idx->cur;
+ }
+ ++idx->cur;
+ if (idx->cur_name != NULL)
+ ++idx->cur_name;
+ _fix_idx_order(idxs, &last_i);
+ ++count;
+ }
+ table_ptr[prefix] = htonl(count);
+
+ PyMem_Free(idxs);
+ return PyLong_FromUnsignedLong(count);
}
static PyObject *write_random(PyObject *self, PyObject *args)
{
uint32_t buf[1024/4];
- int fd = -1, seed = 0;
+ int fd = -1, seed = 0, verbose = 0;
ssize_t ret;
long long len = 0, kbytes = 0, written = 0;
- if (!PyArg_ParseTuple(args, "iLi", &fd, &len, &seed))
+ if (!PyArg_ParseTuple(args, "iLii", &fd, &len, &seed, &verbose))
return NULL;
srandom(seed);
written += ret;
if (ret < (int)sizeof(buf))
break;
- if (kbytes/1024 > 0 && !(kbytes%1024))
+ if (verbose && kbytes/1024 > 0 && !(kbytes%1024))
fprintf(stderr, "Random: %lld Mbytes\r", kbytes/1024);
}
}
+static PyObject *random_sha(PyObject *self, PyObject *args)
+{
+ static int seeded = 0;
+ uint32_t shabuf[20/4];
+ int i;
+
+ if (!seeded)
+ {
+ assert(sizeof(shabuf) == 20);
+ srandom(time(NULL));
+ seeded = 1;
+ }
+
+ if (!PyArg_ParseTuple(args, ""))
+ return NULL;
+
+ memset(shabuf, 0, sizeof(shabuf));
+ for (i=0; i < 20/4; i++)
+ shabuf[i] = random();
+ return Py_BuildValue("s#", shabuf, 20);
+}
+
+
static PyObject *open_noatime(PyObject *self, PyObject *args)
{
char *filename = NULL;
#endif /* def linux */
-static PyMethodDef helper_methods[] = {
+static PyMethodDef faster_methods[] = {
{ "selftest", selftest, METH_VARARGS,
"Check that the rolling checksum rolls correctly (for unit tests)." },
{ "blobbits", blobbits, METH_VARARGS,
"Count the number of matching prefix bits between two strings." },
{ "firstword", firstword, METH_VARARGS,
"Return an int corresponding to the first 32 bits of buf." },
+ { "bloom_contains", bloom_contains, METH_VARARGS,
+ "Check if a bloom filter of 2^nbits bytes contains an object" },
+ { "bloom_add", bloom_add, METH_VARARGS,
+ "Add an object to a bloom filter of 2^nbits bytes" },
{ "extract_bits", extract_bits, METH_VARARGS,
"Take the first 'nbits' bits from 'buf' and return them as an int." },
+ { "merge_into", merge_into, METH_VARARGS,
+ "Merges a bunch of idx and midx files into a single midx." },
{ "write_random", write_random, METH_VARARGS,
"Write random bytes to the given file descriptor" },
+ { "random_sha", random_sha, METH_VARARGS,
+ "Return a random 20-byte string" },
{ "open_noatime", open_noatime, METH_VARARGS,
"open() the given filename for read with O_NOATIME if possible" },
{ "fadvise_done", fadvise_done, METH_VARARGS,
PyMODINIT_FUNC init_helpers(void)
{
- PyObject *m = Py_InitModule("_helpers", helper_methods);
+ PyObject *m = Py_InitModule("_helpers", faster_methods);
if (m == NULL)
return;
#ifdef HAVE_BUP_UTIMENSAT
Py_INCREF(Py_False);
PyModule_AddObject(m, "_have_ns_fs_timestamps", Py_False);
#endif
+ istty = isatty(2) || getenv("BUP_FORCE_TTY");
}
-import re, struct, errno, time
+import re, struct, errno, time, zlib
from bup import git, ssh
from bup.helpers import *
bwcount = len(sub) # might be less than 4096
bwtime = next
return (bwcount, bwtime)
-
+
+
+def parse_remote(remote):
+ protocol = r'([a-z]+)://'
+ host = r'(?P<sb>\[)?((?(sb)[0-9a-f:]+|[^:/]+))(?(sb)\])'
+ port = r'(?::(\d+))?'
+ path = r'(/.*)?'
+ url_match = re.match(
+ '%s(?:%s%s)?%s' % (protocol, host, port, path), remote, re.I)
+ if url_match:
+ assert(url_match.group(1) in ('ssh', 'bup', 'file'))
+ return url_match.group(1,3,4,5)
+ else:
+ rs = remote.split(':', 1)
+ if len(rs) == 1 or rs[0] in ('', '-'):
+ return 'file', None, None, rs[-1]
+ else:
+ return 'ssh', rs[0], None, rs[1]
+
class Client:
def __init__(self, remote, create=False):
- self._busy = self.conn = self.p = self.pout = self.pin = None
+ self._busy = self.conn = None
+ self.sock = self.p = self.pout = self.pin = None
is_reverse = os.environ.get('BUP_SERVER_REVERSE')
if is_reverse:
assert(not remote)
remote = '%s:' % is_reverse
- rs = remote.split(':', 1)
- if len(rs) == 1:
- (host, dir) = (None, remote)
- else:
- (host, dir) = rs
- (self.host, self.dir) = (host, dir)
+ (self.protocol, self.host, self.port, self.dir) = parse_remote(remote)
self.cachedir = git.repo('index-cache/%s'
% re.sub(r'[^@\w]', '_',
- "%s:%s" % (host, dir)))
+ "%s:%s" % (self.host, self.dir)))
if is_reverse:
self.pout = os.fdopen(3, 'rb')
self.pin = os.fdopen(4, 'wb')
+ self.conn = Conn(self.pout, self.pin)
else:
- try:
- self.p = ssh.connect(host, 'server')
- self.pout = self.p.stdout
- self.pin = self.p.stdin
- except OSError, e:
- raise ClientError, 'connect: %s' % e, sys.exc_info()[2]
- self.conn = Conn(self.pout, self.pin)
- if dir:
- dir = re.sub(r'[\r\n]', ' ', dir)
+ if self.protocol in ('ssh', 'file'):
+ try:
+ # FIXME: ssh and file shouldn't use the same module
+ self.p = ssh.connect(self.host, self.port, 'server')
+ self.pout = self.p.stdout
+ self.pin = self.p.stdin
+ self.conn = Conn(self.pout, self.pin)
+ except OSError, e:
+ raise ClientError, 'connect: %s' % e, sys.exc_info()[2]
+ elif self.protocol == 'bup':
+ self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ self.sock.connect((self.host, atoi(self.port) or 1982))
+ self.sockw = self.sock.makefile('wb')
+ self.conn = DemuxConn(self.sock.fileno(), self.sockw)
+ if self.dir:
+ self.dir = re.sub(r'[\r\n]', ' ', self.dir)
if create:
- self.conn.write('init-dir %s\n' % dir)
+ self.conn.write('init-dir %s\n' % self.dir)
else:
- self.conn.write('set-dir %s\n' % dir)
+ self.conn.write('set-dir %s\n' % self.dir)
self.check_ok()
- self.sync_indexes_del()
+ self.sync_indexes()
def __del__(self):
try:
def close(self):
if self.conn and not self._busy:
self.conn.write('quit\n')
- if self.pin and self.pout:
+ if self.pin:
self.pin.close()
- while self.pout.read(65536):
- pass
+ if self.sock and self.sockw:
+ self.sockw.close()
+ self.sock.shutdown(socket.SHUT_WR)
+ if self.conn:
+ self.conn.close()
+ if self.pout:
self.pout.close()
+ if self.sock:
+ self.sock.close()
if self.p:
self.p.wait()
rv = self.p.wait()
if rv:
raise ClientError('server tunnel returned exit code %d' % rv)
self.conn = None
- self.p = self.pin = self.pout = None
+ self.sock = self.p = self.pin = self.pout = None
def check_ok(self):
if self.p:
def _not_busy(self):
self._busy = None
- def sync_indexes_del(self):
+ def sync_indexes(self):
self.check_busy()
conn = self.conn
+ mkdirp(self.cachedir)
+ # All cached idxs are extra until proven otherwise
+ extra = set()
+ for f in os.listdir(self.cachedir):
+ debug1('%s\n' % f)
+ if f.endswith('.idx'):
+ extra.add(f)
+ needed = set()
conn.write('list-indexes\n')
- packdir = git.repo('objects/pack')
- all = {}
- needed = {}
for line in linereader(conn):
if not line:
break
- all[line] = 1
assert(line.find('/') < 0)
- if not os.path.exists(os.path.join(self.cachedir, line)):
- needed[line] = 1
+ parts = line.split(' ')
+ idx = parts[0]
+ if len(parts) == 2 and parts[1] == 'load' and idx not in extra:
+ # If the server requests that we load an idx and we don't
+ # already have a copy of it, it is needed
+ needed.add(idx)
+ # Any idx that the server has heard of is proven not extra
+ extra.discard(idx)
+
self.check_ok()
+ debug1('client: removing extra indexes: %s\n' % extra)
+ for idx in extra:
+ os.unlink(os.path.join(self.cachedir, idx))
+ debug1('client: server requested load of: %s\n' % needed)
+ for idx in needed:
+ self.sync_index(idx)
+ git.auto_midx(self.cachedir)
- mkdirp(self.cachedir)
- for f in os.listdir(self.cachedir):
- if f.endswith('.idx') and not f in all:
- debug1('client: pruning old index: %r\n' % f)
- os.unlink(os.path.join(self.cachedir, f))
def sync_index(self, name):
#debug1('requesting %r\n' % name)
self.check_ok()
f.close()
os.rename(fn + '.tmp', fn)
- git.auto_midx(self.cachedir)
def _make_objcache(self):
- ob = self._busy
- self._busy = None
- #self.sync_indexes()
- self._busy = ob
return git.PackIdxList(self.cachedir)
- def _suggest_pack(self, indexname):
- debug1('client: received index suggestion: %s\n' % indexname)
+ def _suggest_packs(self):
ob = self._busy
if ob:
- assert(ob == 'receive-objects')
- self.conn.write('\xff\xff\xff\xff') # suspend receive-objects
+ assert(ob == 'receive-objects-v2')
+ self.conn.write('\xff\xff\xff\xff') # suspend receive-objects-v2
+ suggested = []
+ for line in linereader(self.conn):
+ if not line:
+ break
+ debug2('%s\n' % line)
+ if line.startswith('index '):
+ idx = line[6:]
+ debug1('client: received index suggestion: %s\n' % idx)
+ suggested.append(idx)
+ else:
+ assert(line.endswith('.idx'))
+ debug1('client: completed writing pack, idx: %s\n' % line)
+ suggested.append(line)
+ self.check_ok()
+ if ob:
self._busy = None
- self.conn.drain_and_check_ok()
- self.sync_index(indexname)
+ idx = None
+ for idx in suggested:
+ self.sync_index(idx)
+ git.auto_midx(self.cachedir)
if ob:
self._busy = ob
- self.conn.write('receive-objects\n')
+ self.conn.write('%s\n' % ob)
+ return idx
def new_packwriter(self):
self.check_busy()
def _set_busy():
- self._busy = 'receive-objects'
- self.conn.write('receive-objects\n')
+ self._busy = 'receive-objects-v2'
+ self.conn.write('receive-objects-v2\n')
return PackWriter_Remote(self.conn,
objcache_maker = self._make_objcache,
- suggest_pack = self._suggest_pack,
+ suggest_packs = self._suggest_packs,
onopen = _set_busy,
onclose = self._not_busy,
ensure_busy = self.ensure_busy)
class PackWriter_Remote(git.PackWriter):
- def __init__(self, conn, objcache_maker, suggest_pack,
+ def __init__(self, conn, objcache_maker, suggest_packs,
onopen, onclose,
ensure_busy):
git.PackWriter.__init__(self, objcache_maker)
self.file = conn
self.filename = 'remote socket'
- self.suggest_pack = suggest_pack
+ self.suggest_packs = suggest_packs
self.onopen = onopen
self.onclose = onclose
self.ensure_busy = ensure_busy
def _open(self):
if not self._packopen:
- self._make_objcache()
- if self.onopen:
- self.onopen()
+ self.onopen()
self._packopen = True
def _end(self):
if self._packopen and self.file:
self.file.write('\0\0\0\0')
self._packopen = False
- while True:
- line = self.file.readline().strip()
- if line.startswith('index '):
- pass
- else:
- break
- id = line
- self.file.check_ok()
+ self.onclose() # Unbusy
self.objcache = None
- if self.onclose:
- self.onclose()
- if id and self.suggest_pack:
- self.suggest_pack(id)
- return id
+ return self.suggest_packs() # Returns last idx received
def close(self):
id = self._end()
return id
def abort(self):
- raise GitError("don't know how to abort remote pack writing")
+ raise ClientError("don't know how to abort remote pack writing")
- def _raw_write(self, datalist):
+ def _raw_write(self, datalist, sha):
assert(self.file)
if not self._packopen:
self._open()
- if self.ensure_busy:
- self.ensure_busy()
+ self.ensure_busy()
data = ''.join(datalist)
- assert(len(data))
- outbuf = struct.pack('!I', len(data)) + data
- (self._bwcount, self._bwtime) = \
- _raw_write_bwlimit(self.file, outbuf, self._bwcount, self._bwtime)
+ assert(data)
+ assert(sha)
+ crc = zlib.crc32(data) & 0xffffffff
+ outbuf = ''.join((struct.pack('!I', len(data) + 20 + 4),
+ sha,
+ struct.pack('!I', crc),
+ data))
+ try:
+ (self._bwcount, self._bwtime) = _raw_write_bwlimit(
+ self.file, outbuf, self._bwcount, self._bwtime)
+ except IOError, e:
+ raise ClientError, e, sys.exc_info()[2]
self.outbytes += len(data)
self.count += 1
if self.file.has_input():
- line = self.file.readline().strip()
- assert(line.startswith('index '))
- idxname = line[6:]
- if self.suggest_pack:
- self.suggest_pack(idxname)
- self.objcache.refresh()
+ self.suggest_packs()
+ self.objcache.refresh()
+
+ return sha, crc
-import stat
+import stat, os
from bup.helpers import *
import bup.xstat as xstat
O_LARGEFILE = os.O_LARGEFILE
except AttributeError:
O_LARGEFILE = 0
+try:
+ O_NOFOLLOW = os.O_NOFOLLOW
+except AttributeError:
+ O_NOFOLLOW = 0
# the use of fchdir() and lstat() is for two reasons:
class OsFile:
def __init__(self, path):
self.fd = None
- self.fd = os.open(path,
- os.O_RDONLY|O_LARGEFILE|os.O_NOFOLLOW|os.O_NDELAY)
+ self.fd = os.open(path, os.O_RDONLY|O_LARGEFILE|O_NOFOLLOW|os.O_NDELAY)
def __del__(self):
if self.fd:
return l
-def _recursive_dirlist(prepend, xdev):
+def _recursive_dirlist(prepend, xdev, bup_dir=None, excluded_paths=None):
for (name,pst) in _dirlist():
if name.endswith('/'):
if xdev != None and pst.st_dev != xdev:
log('Skipping %r: different filesystem.\n' % (prepend+name))
continue
+ if bup_dir != None:
+ if os.path.normpath(prepend+name) == bup_dir:
+ log('Skipping BUP_DIR.\n')
+ continue
+ if excluded_paths:
+ if os.path.normpath(prepend+name) in excluded_paths:
+ log('Skipping %r: excluded.\n' % (prepend+name))
+ continue
try:
OsFile(name).fchdir()
except OSError, e:
add_error('%s: %s' % (prepend, e))
else:
- for i in _recursive_dirlist(prepend=prepend+name, xdev=xdev):
+ for i in _recursive_dirlist(prepend=prepend+name, xdev=xdev,
+ bup_dir=bup_dir,
+ excluded_paths=excluded_paths):
yield i
os.chdir('..')
yield (prepend + name, pst)
-def recursive_dirlist(paths, xdev):
+def recursive_dirlist(paths, xdev, bup_dir=None, excluded_paths=None):
startdir = OsFile('.')
try:
assert(type(paths) != type(''))
if stat.S_ISDIR(pst.st_mode):
pfile.fchdir()
prepend = os.path.join(path, '')
- for i in _recursive_dirlist(prepend=prepend, xdev=xdev):
+ for i in _recursive_dirlist(prepend=prepend, xdev=xdev,
+ bup_dir=bup_dir,
+ excluded_paths=excluded_paths):
yield i
startdir.fchdir()
else:
except:
pass
raise
+
+def parse_excludes(flags):
+ excluded_paths = []
+
+ for flag in flags:
+ (option, parameter) = flag
+ if option == '--exclude':
+ excluded_paths.append(realpath(parameter))
+
+ if option == '--exclude-from':
+ try:
+ try:
+ f = open(realpath(parameter))
+ for exclude_path in f.readlines():
+ excluded_paths.append(realpath(exclude_path.strip()))
+ except Error, e:
+ log("warning: couldn't read %s" % parameter)
+ finally:
+ f.close()
+
+ return excluded_paths
+
bup repositories are in Git format. This library allows us to
interact with the Git data structures.
"""
-import os, zlib, time, subprocess, struct, stat, re, tempfile
-import heapq
+import os, sys, zlib, time, subprocess, struct, stat, re, tempfile, math, glob
from bup.helpers import *
-from bup import _helpers
-
-MIDX_VERSION = 2
+from bup import _helpers, path
+
+MIDX_VERSION = 4
+
+"""Discussion of bloom constants for bup:
+
+There are four basic things to consider when building a bloom filter:
+The size, in bits, of the filter
+The capacity, in entries, of the filter
+The probability of a false positive that is tolerable
+The number of bits readily available to use for addresing filter bits
+
+There is one major tunable that is not directly related to the above:
+k: the number of bits set in the filter per entry
+
+Here's a wall of numbers showing the relationship between k; the ratio between
+the filter size in bits and the entries in the filter; and pfalse_positive:
+
+mn|k=3 |k=4 |k=5 |k=6 |k=7 |k=8 |k=9 |k=10 |k=11
+ 8|3.05794|2.39687|2.16792|2.15771|2.29297|2.54917|2.92244|3.41909|4.05091
+ 9|2.27780|1.65770|1.40703|1.32721|1.34892|1.44631|1.61138|1.84491|2.15259
+10|1.74106|1.18133|0.94309|0.84362|0.81937|0.84555|0.91270|1.01859|1.16495
+11|1.36005|0.86373|0.65018|0.55222|0.51259|0.50864|0.53098|0.57616|0.64387
+12|1.08231|0.64568|0.45945|0.37108|0.32939|0.31424|0.31695|0.33387|0.36380
+13|0.87517|0.49210|0.33183|0.25527|0.21689|0.19897|0.19384|0.19804|0.21013
+14|0.71759|0.38147|0.24433|0.17934|0.14601|0.12887|0.12127|0.12012|0.12399
+15|0.59562|0.30019|0.18303|0.12840|0.10028|0.08523|0.07749|0.07440|0.07468
+16|0.49977|0.23941|0.13925|0.09351|0.07015|0.05745|0.05049|0.04700|0.04587
+17|0.42340|0.19323|0.10742|0.06916|0.04990|0.03941|0.03350|0.03024|0.02870
+18|0.36181|0.15765|0.08392|0.05188|0.03604|0.02748|0.02260|0.01980|0.01827
+19|0.31160|0.12989|0.06632|0.03942|0.02640|0.01945|0.01549|0.01317|0.01182
+20|0.27026|0.10797|0.05296|0.03031|0.01959|0.01396|0.01077|0.00889|0.00777
+21|0.23591|0.09048|0.04269|0.02356|0.01471|0.01014|0.00759|0.00609|0.00518
+22|0.20714|0.07639|0.03473|0.01850|0.01117|0.00746|0.00542|0.00423|0.00350
+23|0.18287|0.06493|0.02847|0.01466|0.00856|0.00555|0.00392|0.00297|0.00240
+24|0.16224|0.05554|0.02352|0.01171|0.00663|0.00417|0.00286|0.00211|0.00166
+25|0.14459|0.04779|0.01957|0.00944|0.00518|0.00316|0.00211|0.00152|0.00116
+26|0.12942|0.04135|0.01639|0.00766|0.00408|0.00242|0.00157|0.00110|0.00082
+27|0.11629|0.03595|0.01381|0.00626|0.00324|0.00187|0.00118|0.00081|0.00059
+28|0.10489|0.03141|0.01170|0.00515|0.00259|0.00146|0.00090|0.00060|0.00043
+29|0.09492|0.02756|0.00996|0.00426|0.00209|0.00114|0.00069|0.00045|0.00031
+30|0.08618|0.02428|0.00853|0.00355|0.00169|0.00090|0.00053|0.00034|0.00023
+31|0.07848|0.02147|0.00733|0.00297|0.00138|0.00072|0.00041|0.00025|0.00017
+32|0.07167|0.01906|0.00633|0.00250|0.00113|0.00057|0.00032|0.00019|0.00013
+
+Here's a table showing available repository size for a given pfalse_positive
+and three values of k (assuming we only use the 160 bit SHA1 for addressing the
+filter and 8192bytes per object):
+
+pfalse|obj k=4 |cap k=4 |obj k=5 |cap k=5 |obj k=6 |cap k=6
+2.500%|139333497228|1038.11 TiB|558711157|4262.63 GiB|13815755|105.41 GiB
+1.000%|104489450934| 778.50 TiB|436090254|3327.10 GiB|11077519| 84.51 GiB
+0.125%| 57254889824| 426.58 TiB|261732190|1996.86 GiB| 7063017| 55.89 GiB
+
+This eliminates pretty neatly any k>6 as long as we use the raw SHA for
+addressing.
+
+filter size scales linearly with repository size for a given k and pfalse.
+
+Here's a table of filter sizes for a 1 TiB repository:
+
+pfalse| k=3 | k=4 | k=5 | k=6
+2.500%| 138.78 MiB | 126.26 MiB | 123.00 MiB | 123.37 MiB
+1.000%| 197.83 MiB | 168.36 MiB | 157.58 MiB | 153.87 MiB
+0.125%| 421.14 MiB | 307.26 MiB | 262.56 MiB | 241.32 MiB
+
+For bup:
+* We want the bloom filter to fit in memory; if it doesn't, the k pagefaults
+per lookup will be worse than the two required for midx.
+* We want the pfalse_positive to be low enough that the cost of sometimes
+faulting on the midx doesn't overcome the benefit of the bloom filter.
+* We have readily available 160 bits for addressing the filter.
+* We want to be able to have a single bloom address entire repositories of
+reasonable size.
+
+Based on these parameters, a combination of k=4 and k=5 provides the behavior
+that bup needs. As such, I've implemented bloom addressing, adding and
+checking functions in C for these two values. Because k=5 requires less space
+and gives better overall pfalse_positive perofrmance, it is preferred if a
+table with k=5 can represent the repository.
+
+None of this tells us what max_pfalse_positive to choose.
+
+Brandon Low <lostlogic@lostlogicx.com> 04-02-2011
+"""
+BLOOM_VERSION = 2
+MAX_BITS_EACH = 32 # Kinda arbitrary, but 4 bytes per entry is pretty big
+MAX_BLOOM_BITS = {4: 37, 5: 29} # 160/k-log2(8)
+MAX_PFALSE_POSITIVE = 1. # Totally arbitrary, needs benchmarking
verbose = 0
ignore_midx = 0
def auto_midx(objdir):
- main_exe = os.environ.get('BUP_MAIN_EXE') or sys.argv[0]
- args = [main_exe, 'midx', '--auto', '--dir', objdir]
- rv = subprocess.call(args, stdout=open('/dev/null', 'w'))
+ args = [path.exe(), 'midx', '--auto', '--dir', objdir]
+ try:
+ rv = subprocess.call(args, stdout=open('/dev/null', 'w'))
+ except OSError, e:
+ # make sure 'args' gets printed to help with debugging
+ add_error('%r: exception: %s' % (args, e))
+ raise
+ if rv:
+ add_error('%r: returned %d' % (args, rv))
+
+ args = [path.exe(), 'bloom', '--dir', objdir]
+ try:
+ rv = subprocess.call(args, stdout=open('/dev/null', 'w'))
+ except OSError, e:
+ # make sure 'args' gets printed to help with debugging
+ add_error('%r: exception: %s' % (args, e))
+ raise
if rv:
add_error('%r: returned %d' % (args, rv))
class PackIdx:
- """Object representation of a Git pack index file."""
- def __init__(self, filename):
- self.name = filename
- self.idxnames = [self.name]
- self.map = mmap_read(open(filename))
- assert(str(self.map[0:8]) == '\377tOc\0\0\0\2')
- self.fanout = list(struct.unpack('!256I',
- str(buffer(self.map, 8, 256*4))))
- self.fanout.append(0) # entry "-1"
- nsha = self.fanout[255]
- self.ofstable = buffer(self.map,
- 8 + 256*4 + nsha*20 + nsha*4,
- nsha*4)
- self.ofs64table = buffer(self.map,
- 8 + 256*4 + nsha*20 + nsha*4 + nsha*4)
+ def __init__(self):
+ assert(0)
- def _ofs_from_idx(self, idx):
- ofs = struct.unpack('!I', str(buffer(self.ofstable, idx*4, 4)))[0]
- if ofs & 0x80000000:
- idx64 = ofs & 0x7fffffff
- ofs = struct.unpack('!I',
- str(buffer(self.ofs64table, idx64*8, 8)))[0]
- return ofs
+ def find_offset(self, hash):
+ """Get the offset of an object inside the index file."""
+ idx = self._idx_from_hash(hash)
+ if idx != None:
+ return self._ofs_from_idx(idx)
+ return None
+
+ def exists(self, hash, want_source=False):
+ """Return nonempty if the object exists in this index."""
+ if hash and (self._idx_from_hash(hash) != None):
+ return want_source and os.path.basename(self.name) or True
+ return None
+
+ def __len__(self):
+ return int(self.fanout[255])
def _idx_from_hash(self, hash):
global _total_searches, _total_steps
b1 = ord(hash[0])
start = self.fanout[b1-1] # range -1..254
end = self.fanout[b1] # range 0..255
- buf = buffer(self.map, 8 + 256*4, end*20)
want = str(hash)
_total_steps += 1 # lookup table is a step
while start < end:
_total_steps += 1
mid = start + (end-start)/2
- v = str(buf[mid*20:(mid+1)*20])
+ v = self._idx_to_hash(mid)
if v < want:
start = mid+1
elif v > want:
return mid
return None
- def find_offset(self, hash):
- """Get the offset of an object inside the index file."""
- idx = self._idx_from_hash(hash)
- if idx != None:
- return self._ofs_from_idx(idx)
- return None
- def exists(self, hash):
- """Return nonempty if the object exists in this index."""
- return hash and (self._idx_from_hash(hash) != None) and True or None
+class PackIdxV1(PackIdx):
+ """Object representation of a Git pack index (version 1) file."""
+ def __init__(self, filename, f):
+ self.name = filename
+ self.idxnames = [self.name]
+ self.map = mmap_read(f)
+ self.fanout = list(struct.unpack('!256I',
+ str(buffer(self.map, 0, 256*4))))
+ self.fanout.append(0) # entry "-1"
+ nsha = self.fanout[255]
+ self.sha_ofs = 256*4
+ self.shatable = buffer(self.map, self.sha_ofs, nsha*24)
+
+ def _ofs_from_idx(self, idx):
+ return struct.unpack('!I', str(self.shatable[idx*24 : idx*24+4]))[0]
+
+ def _idx_to_hash(self, idx):
+ return str(self.shatable[idx*24+4 : idx*24+24])
def __iter__(self):
for i in xrange(self.fanout[255]):
- yield buffer(self.map, 8 + 256*4 + 20*i, 20)
+ yield buffer(self.map, 256*4 + 24*i + 4, 20)
- def __len__(self):
- return int(self.fanout[255])
+
+class PackIdxV2(PackIdx):
+ """Object representation of a Git pack index (version 2) file."""
+ def __init__(self, filename, f):
+ self.name = filename
+ self.idxnames = [self.name]
+ self.map = mmap_read(f)
+ assert(str(self.map[0:8]) == '\377tOc\0\0\0\2')
+ self.fanout = list(struct.unpack('!256I',
+ str(buffer(self.map, 8, 256*4))))
+ self.fanout.append(0) # entry "-1"
+ nsha = self.fanout[255]
+ self.sha_ofs = 8 + 256*4
+ self.shatable = buffer(self.map, self.sha_ofs, nsha*20)
+ self.ofstable = buffer(self.map,
+ self.sha_ofs + nsha*20 + nsha*4,
+ nsha*4)
+ self.ofs64table = buffer(self.map,
+ 8 + 256*4 + nsha*20 + nsha*4 + nsha*4)
+
+ def _ofs_from_idx(self, idx):
+ ofs = struct.unpack('!I', str(buffer(self.ofstable, idx*4, 4)))[0]
+ if ofs & 0x80000000:
+ idx64 = ofs & 0x7fffffff
+ ofs = struct.unpack('!Q',
+ str(buffer(self.ofs64table, idx64*8, 8)))[0]
+ return ofs
+
+ def _idx_to_hash(self, idx):
+ return str(self.shatable[idx*20:(idx+1)*20])
+
+ def __iter__(self):
+ for i in xrange(self.fanout[255]):
+ yield buffer(self.map, 8 + 256*4 + 20*i, 20)
extract_bits = _helpers.extract_bits
+bloom_contains = _helpers.bloom_contains
+bloom_add = _helpers.bloom_add
+
+
+class ShaBloom:
+ """Wrapper which contains data from multiple index files.
+ """
+ def __init__(self, filename, f=None, readwrite=False, expected=-1):
+ self.name = filename
+ self.rwfile = None
+ self.map = None
+ assert(filename.endswith('.bloom'))
+ if readwrite:
+ assert(expected > 0)
+ self.rwfile = f = f or open(filename, 'r+b')
+ f.seek(0)
+
+ # Decide if we want to mmap() the pages as writable ('immediate'
+ # write) or else map them privately for later writing back to
+ # the file ('delayed' write). A bloom table's write access
+ # pattern is such that we dirty almost all the pages after adding
+ # very few entries. But the table is so big that dirtying
+ # *all* the pages often exceeds Linux's default
+ # /proc/sys/vm/dirty_ratio or /proc/sys/vm/dirty_background_ratio,
+ # thus causing it to start flushing the table before we're
+ # finished... even though there's more than enough space to
+ # store the bloom table in RAM.
+ #
+ # To work around that behaviour, if we calculate that we'll
+ # probably end up touching the whole table anyway (at least
+ # one bit flipped per memory page), let's use a "private" mmap,
+ # which defeats Linux's ability to flush it to disk. Then we'll
+ # flush it as one big lump during close().
+ pages = os.fstat(f.fileno()).st_size / 4096 * 5 # assume k=5
+ self.delaywrite = expected > pages
+ debug1('bloom: delaywrite=%r\n' % self.delaywrite)
+ if self.delaywrite:
+ self.map = mmap_readwrite_private(self.rwfile, close=False)
+ else:
+ self.map = mmap_readwrite(self.rwfile, close=False)
+ else:
+ self.rwfile = None
+ f = f or open(filename, 'rb')
+ self.map = mmap_read(f)
+ got = str(self.map[0:4])
+ if got != 'BLOM':
+ log('Warning: invalid BLOM header (%r) in %r\n' % (got, filename))
+ return self._init_failed()
+ ver = struct.unpack('!I', self.map[4:8])[0]
+ if ver < BLOOM_VERSION:
+ log('Warning: ignoring old-style (v%d) bloom %r\n'
+ % (ver, filename))
+ return self._init_failed()
+ if ver > BLOOM_VERSION:
+ log('Warning: ignoring too-new (v%d) bloom %r\n'
+ % (ver, filename))
+ return self._init_failed()
+
+ self.bits, self.k, self.entries = struct.unpack('!HHI', self.map[8:16])
+ idxnamestr = str(self.map[16 + 2**self.bits:])
+ if idxnamestr:
+ self.idxnames = idxnamestr.split('\0')
+ else:
+ self.idxnames = []
+
+ def _init_failed(self):
+ if self.map:
+ self.map = None
+ if self.rwfile:
+ self.rwfile.close()
+ self.rwfile = None
+ self.idxnames = []
+ self.bits = self.entries = 0
+
+ def valid(self):
+ return self.map and self.bits
+
+ def __del__(self):
+ self.close()
+
+ def close(self):
+ if self.map and self.rwfile:
+ debug2("bloom: closing with %d entries\n" % self.entries)
+ self.map[12:16] = struct.pack('!I', self.entries)
+ if self.delaywrite:
+ self.rwfile.seek(0)
+ self.rwfile.write(self.map)
+ else:
+ self.map.flush()
+ self.rwfile.seek(16 + 2**self.bits)
+ if self.idxnames:
+ self.rwfile.write('\0'.join(self.idxnames))
+ self._init_failed()
+
+ def pfalse_positive(self, additional=0):
+ n = self.entries + additional
+ m = 8*2**self.bits
+ k = self.k
+ return 100*(1-math.exp(-k*float(n)/m))**k
+
+ def add_idx(self, ix):
+ """Add the object to the filter, return current pfalse_positive."""
+ if not self.map: raise Exception, "Cannot add to closed bloom"
+ self.entries += bloom_add(self.map, ix.shatable, self.bits, self.k)
+ self.idxnames.append(os.path.basename(ix.name))
+
+ def exists(self, sha):
+ """Return nonempty if the object probably exists in the bloom filter."""
+ global _total_searches, _total_steps
+ _total_searches += 1
+ if not self.map: return None
+ found, steps = bloom_contains(self.map, str(sha), self.bits, self.k)
+ _total_steps += steps
+ return found
+
+ @classmethod
+ def create(cls, name, expected, delaywrite=None, f=None, k=None):
+ """Create and return a bloom filter for `expected` entries."""
+ bits = int(math.floor(math.log(expected*MAX_BITS_EACH/8,2)))
+ k = k or ((bits <= MAX_BLOOM_BITS[5]) and 5 or 4)
+ if bits > MAX_BLOOM_BITS[k]:
+ log('bloom: warning, max bits exceeded, non-optimal\n')
+ bits = MAX_BLOOM_BITS[k]
+ debug1('bloom: using 2^%d bytes and %d hash functions\n' % (bits, k))
+ f = f or open(name, 'w+b')
+ f.write('BLOM')
+ f.write(struct.pack('!IHHI', BLOOM_VERSION, bits, k, 0))
+ assert(f.tell() == 16)
+ # NOTE: On some systems this will not extend+zerofill, but it does on
+ # darwin, linux, bsd and solaris.
+ f.truncate(16+2**bits)
+ f.seek(0)
+ if delaywrite != None and not delaywrite:
+ # tell it to expect very few objects, forcing a direct mmap
+ expected = 1
+ return cls(name, f=f, readwrite=True, expected=expected)
+
+ def __len__(self):
+ return int(self.entries)
+
class PackMidx:
"""Wrapper which contains data from multiple index files.
self.bits = _helpers.firstword(self.map[8:12])
self.entries = 2**self.bits
self.fanout = buffer(self.map, 12, self.entries*4)
- shaofs = 12 + self.entries*4
- nsha = self._fanget(self.entries-1)
- self.shalist = buffer(self.map, shaofs, nsha*20)
- self.idxnames = str(self.map[shaofs + 20*nsha:]).split('\0')
+ self.sha_ofs = 12 + self.entries*4
+ self.nsha = nsha = self._fanget(self.entries-1)
+ self.shatable = buffer(self.map, self.sha_ofs, nsha*20)
+ self.whichlist = buffer(self.map, self.sha_ofs + nsha*20, nsha*4)
+ self.idxname_ofs = self.sha_ofs + 24*nsha
+ self.idxnames = str(self.map[self.idxname_ofs:]).split('\0')
def _init_failed(self):
self.bits = 0
self.entries = 1
self.fanout = buffer('\0\0\0\0')
- self.shalist = buffer('\0'*20)
+ self.shatable = buffer('\0'*20)
self.idxnames = []
def _fanget(self, i):
return _helpers.firstword(s)
def _get(self, i):
- return str(self.shalist[i*20:(i+1)*20])
+ return str(self.shatable[i*20:(i+1)*20])
- def exists(self, hash):
+ def _get_idx_i(self, i):
+ return struct.unpack('!I', self.whichlist[i*4:(i+1)*4])[0]
+
+ def _get_idxname(self, i):
+ return self.idxnames[self._get_idx_i(i)]
+
+ def exists(self, hash, want_source=False):
"""Return nonempty if the object exists in the index files."""
global _total_searches, _total_steps
_total_searches += 1
end = mid
endv = _helpers.firstword(v)
else: # got it!
- return True
+ return want_source and self._get_idxname(mid) or True
return None
def __iter__(self):
for i in xrange(self._fanget(self.entries-1)):
- yield buffer(self.shalist, i*20, 20)
+ yield buffer(self.shatable, i*20, 20)
def __len__(self):
return int(self._fanget(self.entries-1))
assert(_mpi_count == 0) # these things suck tons of VM; don't waste it
_mpi_count += 1
self.dir = dir
- self.also = {}
+ self.also = set()
self.packs = []
+ self.do_bloom = False
+ self.bloom = None
self.refresh()
def __del__(self):
def __len__(self):
return sum(len(pack) for pack in self.packs)
- def exists(self, hash):
+ def exists(self, hash, want_source=False):
"""Return nonempty if the object exists in the index files."""
global _total_searches
_total_searches += 1
if hash in self.also:
return True
- for i in range(len(self.packs)):
+ if self.do_bloom and self.bloom is not None:
+ _total_searches -= 1 # will be incremented by bloom
+ if self.bloom.exists(hash):
+ self.do_bloom = False
+ else:
+ return None
+ for i in xrange(len(self.packs)):
p = self.packs[i]
_total_searches -= 1 # will be incremented by sub-pack
- if p.exists(hash):
+ ix = p.exists(hash, want_source=want_source)
+ if ix:
# reorder so most recently used packs are searched first
self.packs = [p] + self.packs[:i] + self.packs[i+1:]
- return p.name
+ return ix
+ self.do_bloom = True
return None
def refresh(self, skip_midx = False):
The module-global variable 'ignore_midx' can force this function to
always act as if skip_midx was True.
"""
+ self.bloom = None # Always reopen the bloom as it may have been relaced
+ self.do_bloom = False
skip_midx = skip_midx or ignore_midx
d = dict((p.name, p) for p in self.packs
if not skip_midx or not isinstance(p, PackMidx))
if isinstance(ix, PackMidx):
for name in ix.idxnames:
d[os.path.join(self.dir, name)] = ix
- for f in os.listdir(self.dir):
- full = os.path.join(self.dir, f)
- if f.endswith('.midx') and not d.get(full):
+ for full in glob.glob(os.path.join(self.dir,'*.midx')):
+ if not d.get(full):
mx = PackMidx(full)
(mxd, mxf) = os.path.split(mx.name)
- broken = 0
+ broken = False
for n in mx.idxnames:
if not os.path.exists(os.path.join(mxd, n)):
log(('warning: index %s missing\n' +
' used by %s\n') % (n, mxf))
- broken += 1
- if not broken:
+ broken = True
+ if broken:
+ del mx
+ unlink(full)
+ else:
midxl.append(mx)
midxl.sort(lambda x,y: -cmp(len(x),len(y)))
for ix in midxl:
- any = 0
+ any_needed = False
for sub in ix.idxnames:
found = d.get(os.path.join(self.dir, sub))
if not found or isinstance(found, PackIdx):
# doesn't exist, or exists but not in a midx
- d[ix.name] = ix
- for name in ix.idxnames:
- d[os.path.join(self.dir, name)] = ix
- any += 1
+ any_needed = True
break
- if not any and not ix.force_keep:
+ if any_needed:
+ d[ix.name] = ix
+ for name in ix.idxnames:
+ d[os.path.join(self.dir, name)] = ix
+ elif not ix.force_keep:
debug1('midx: removing redundant: %s\n'
% os.path.basename(ix.name))
unlink(ix.name)
- for f in os.listdir(self.dir):
- full = os.path.join(self.dir, f)
- if f.endswith('.idx') and not d.get(full):
- ix = PackIdx(full)
+ for full in glob.glob(os.path.join(self.dir,'*.idx')):
+ if not d.get(full):
+ try:
+ ix = open_idx(full)
+ except GitError, e:
+ add_error(e)
+ continue
d[full] = ix
+ bfull = os.path.join(self.dir, 'bup.bloom')
+ if self.bloom is None and os.path.exists(bfull):
+ self.bloom = ShaBloom(bfull)
self.packs = list(set(d.values()))
+ self.packs.sort(lambda x,y: -cmp(len(x),len(y)))
+ if self.bloom and self.bloom.valid() and len(self.bloom) >= len(self):
+ self.do_bloom = True
+ else:
+ self.bloom = None
debug1('PackIdxList: using %d index%s.\n'
% (len(self.packs), len(self.packs)!=1 and 'es' or ''))
def add(self, hash):
"""Insert an additional object in the list."""
- self.also[hash] = 1
-
- def zap_also(self):
- """Remove all additional objects from the list."""
- self.also = {}
+ self.also.add(hash)
def calc_hash(type, content):
def open_idx(filename):
if filename.endswith('.idx'):
- return PackIdx(filename)
+ f = open(filename, 'rb')
+ header = f.read(8)
+ if header[0:4] == '\377tOc':
+ version = struct.unpack('!I', header[4:8])[0]
+ if version == 2:
+ return PackIdxV2(filename, f)
+ else:
+ raise GitError('%s: expected idx file version 2, got %d'
+ % (filename, version))
+ elif len(header) == 8 and header[0:4] < '\377tOc':
+ return PackIdxV1(filename, f)
+ else:
+ raise GitError('%s: unrecognized idx file header' % filename)
elif filename.endswith('.midx'):
return PackMidx(filename)
else:
def idxmerge(idxlist, final_progress=True):
"""Generate a list of all the objects reachable in a PackIdxList."""
- total = sum(len(i) for i in idxlist)
- iters = (iter(i) for i in idxlist)
- heap = [(next(it), it) for it in iters]
- heapq.heapify(heap)
- count = 0
- last = None
- while heap:
- if (count % 10024) == 0:
- progress('Reading indexes: %.2f%% (%d/%d)\r'
- % (count*100.0/total, count, total))
- (e, it) = heap[0]
- if e != last:
- yield e
- last = e
- count += 1
- e = next(it)
- if e:
- heapq.heapreplace(heap, (e, it))
- else:
- heapq.heappop(heap)
- if final_progress:
- log('Reading indexes: %.2f%% (%d/%d), done.\n' % (100, total, total))
+ def pfunc(count, total):
+ progress('Reading indexes: %.2f%% (%d/%d)\r'
+ % (count*100.0/total, count, total))
+ def pfinal(count, total):
+ if final_progress:
+ log('Reading indexes: %.2f%% (%d/%d), done.\n' % (100, total, total))
+ return merge_iter(idxlist, 10024, pfunc, pfinal)
+
+def _make_objcache():
+ return PackIdxList(repo('objects/pack'))
class PackWriter:
"""Writes Git objects insid a pack file."""
- def __init__(self, objcache_maker=None):
+ def __init__(self, objcache_maker=_make_objcache):
self.count = 0
self.outbytes = 0
self.filename = None
self.file = None
+ self.idx = None
self.objcache_maker = objcache_maker
self.objcache = None
def __del__(self):
self.close()
- def _make_objcache(self):
- if self.objcache == None:
- if self.objcache_maker:
- self.objcache = self.objcache_maker()
- else:
- self.objcache = PackIdxList(repo('objects/pack'))
-
def _open(self):
if not self.file:
- self._make_objcache()
(fd,name) = tempfile.mkstemp(suffix='.pack', dir=repo('objects'))
self.file = os.fdopen(fd, 'w+b')
assert(name.endswith('.pack'))
self.filename = name[:-5]
self.file.write('PACK\0\0\0\2\0\0\0\0')
+ self.idx = list(list() for i in xrange(256))
- def _raw_write(self, datalist):
+ def _raw_write(self, datalist, sha):
self._open()
f = self.file
# in case we get interrupted (eg. KeyboardInterrupt), it's best if
# to our hashsplit algorithm.) f.write() does its own buffering,
# but that's okay because we'll flush it in _end().
oneblob = ''.join(datalist)
- f.write(oneblob)
- self.outbytes += len(oneblob)
+ try:
+ f.write(oneblob)
+ except IOError, e:
+ raise GitError, e, sys.exc_info()[2]
+ nw = len(oneblob)
+ crc = zlib.crc32(oneblob) & 0xffffffff
+ self._update_idx(sha, crc, nw)
+ self.outbytes += nw
self.count += 1
+ return nw, crc
- def _write(self, bin, type, content):
+ def _update_idx(self, sha, crc, size):
+ assert(sha)
+ if self.idx:
+ self.idx[ord(sha[0])].append((sha, crc, self.file.tell() - size))
+
+ def _write(self, sha, type, content):
if verbose:
log('>')
- self._raw_write(_encode_packobj(type, content))
- return bin
+ if not sha:
+ sha = calc_hash(type, content)
+ size, crc = self._raw_write(_encode_packobj(type, content), sha=sha)
+ return sha
def breakpoint(self):
"""Clear byte and object counts and return the last processed id."""
self.outbytes = self.count = 0
return id
- def write(self, type, content):
- """Write an object in this pack file."""
- return self._write(calc_hash(type, content), type, content)
+ def _require_objcache(self):
+ if self.objcache is None and self.objcache_maker:
+ self.objcache = self.objcache_maker()
+ if self.objcache is None:
+ raise GitError(
+ "PackWriter not opened or can't check exists w/o objcache")
- def exists(self, id):
+ def exists(self, id, want_source=False):
"""Return non-empty if an object is found in the object cache."""
- if not self.objcache:
- self._make_objcache()
- return self.objcache.exists(id)
+ self._require_objcache()
+ return self.objcache.exists(id, want_source=want_source)
def maybe_write(self, type, content):
"""Write an object to the pack file if not present and return its id."""
- bin = calc_hash(type, content)
- if not self.exists(bin):
- self._write(bin, type, content)
- self.objcache.add(bin)
- return bin
+ self._require_objcache()
+ sha = calc_hash(type, content)
+ if not self.exists(sha):
+ self._write(sha, type, content)
+ self.objcache.add(sha)
+ return sha
def new_blob(self, blob):
"""Create a blob object in the pack with the supplied content."""
"""Remove the pack file from disk."""
f = self.file
if f:
+ self.idx = None
self.file = None
f.close()
os.unlink(self.filename + '.pack')
- def _end(self):
+ def _end(self, run_midx=True):
f = self.file
if not f: return None
self.file = None
self.objcache = None
+ idx = self.idx
+ self.idx = None
# update object count
f.seek(8)
# calculate the pack sha1sum
f.seek(0)
sum = Sha1()
- while 1:
- b = f.read(65536)
+ for b in chunkyreader(f):
sum.update(b)
- if not b: break
- f.write(sum.digest())
-
+ packbin = sum.digest()
+ f.write(packbin)
f.close()
- p = subprocess.Popen(['git', 'index-pack', '-v',
- '--index-version=2',
- self.filename + '.pack'],
- preexec_fn = _gitenv,
- stdout = subprocess.PIPE)
- out = p.stdout.read().strip()
- _git_wait('git index-pack', p)
- if not out:
- raise GitError('git index-pack produced no output')
- nameprefix = repo('objects/pack/%s' % out)
+ idx_f = open(self.filename + '.idx', 'wb')
+ obj_list_sha = self._write_pack_idx_v2(idx_f, idx, packbin)
+ idx_f.close()
+
+ nameprefix = repo('objects/pack/pack-%s' % obj_list_sha)
if os.path.exists(self.filename + '.map'):
os.unlink(self.filename + '.map')
os.rename(self.filename + '.pack', nameprefix + '.pack')
os.rename(self.filename + '.idx', nameprefix + '.idx')
- auto_midx(repo('objects/pack'))
+ if run_midx:
+ auto_midx(repo('objects/pack'))
return nameprefix
- def close(self):
+ def close(self, run_midx=True):
"""Close the pack file and move it to its definitive path."""
- return self._end()
+ return self._end(run_midx=run_midx)
+
+ def _write_pack_idx_v2(self, file, idx, packbin):
+ sum = Sha1()
+
+ def write(data):
+ file.write(data)
+ sum.update(data)
+
+ write('\377tOc\0\0\0\2')
+
+ n = 0
+ for part in idx:
+ n += len(part)
+ write(struct.pack('!i', n))
+ part.sort(key=lambda x: x[0])
+
+ obj_list_sum = Sha1()
+ for part in idx:
+ for entry in part:
+ write(entry[0])
+ obj_list_sum.update(entry[0])
+ for part in idx:
+ for entry in part:
+ write(struct.pack('!I', entry[1]))
+ ofs64_list = []
+ for part in idx:
+ for entry in part:
+ if entry[2] & 0x80000000:
+ write(struct.pack('!I', 0x80000000 | len(ofs64_list)))
+ ofs64_list.append(struct.pack('!Q', entry[2]))
+ else:
+ write(struct.pack('!i', entry[2]))
+ for ofs64 in ofs64_list:
+ write(ofs64)
+
+ write(packbin)
+ file.write(sum.digest())
+ return obj_list_sum.hexdigest()
def _git_date(date):
raise GitError, 'no such commit %r' % ref
+def rev_parse(committish):
+ """Resolve the full hash for 'committish', if it exists.
+
+ Should be roughly equivalent to 'git rev-parse'.
+
+ Returns the hex value of the hash if it is found, None if 'committish' does
+ not correspond to anything.
+ """
+ head = read_ref(committish)
+ if head:
+ debug2("resolved from ref: commit = %s\n" % head.encode('hex'))
+ return head
+
+ pL = PackIdxList(repo('objects/pack'))
+
+ if len(committish) == 40:
+ try:
+ hash = committish.decode('hex')
+ except TypeError:
+ return None
+
+ if pL.exists(hash):
+ return hash
+
+ return None
+
+
def update_ref(refname, newval, oldval):
"""Change the commit pointed to by a branch."""
if not oldval:
def init_repo(path=None):
"""Create the Git bare repository for bup in a given path."""
guess_repo(path)
- d = repo()
+ d = repo() # appends a / to the path
+ parent = os.path.dirname(os.path.dirname(d))
+ if parent and not os.path.exists(parent):
+ raise GitError('parent directory "%s" does not exist\n' % parent)
if os.path.exists(d) and not os.path.isdir(os.path.join(d, '.')):
raise GitError('"%d" exists but is not a directory\n' % d)
p = subprocess.Popen(['git', '--bare', 'init'], stdout=sys.stderr,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
close_fds = True,
+ bufsize = 4096,
preexec_fn = _gitenv)
def _fast_get(self, id):
assert(not self.inprogress)
assert(id.find('\n') < 0)
assert(id.find('\r') < 0)
- assert(id[0] != '-')
+ assert(not id.startswith('-'))
self.inprogress = id
self.p.stdin.write('%s\n' % id)
+ self.p.stdin.flush()
hdr = self.p.stdout.readline()
if hdr.endswith(' missing\n'):
+ self.inprogress = None
raise KeyError('blob %r is missing' % id)
spl = hdr.split(' ')
if len(spl) != 3 or len(spl[0]) != 40:
yield d
except StopIteration:
log('booger!\n')
+
+def tags():
+ """Return a dictionary of all tags in the form {hash: [tag_names, ...]}."""
+ tags = {}
+ for (n,c) in list_refs():
+ if n.startswith('refs/tags/'):
+ name = n[10:]
+ if not c in tags:
+ tags[c] = []
+
+ tags[c].append(name) # more than one tag can point at 'c'
+
+ return tags
return (None, 0)
-def blobiter(files):
- for f in files:
+def blobiter(files, progress=None):
+ for filenum,f in enumerate(files):
ofs = 0
+ b = ''
while 1:
+ if progress:
+ progress(filenum, len(b))
fadvise_done(f, max(0, ofs - 1024*1024))
b = f.read(BLOB_HWM)
ofs += len(b)
yield (buf.get(buf.used()), 0)
-def hashsplit_iter(files):
+def _hashsplit_iter(files, progress):
assert(BLOB_HWM > BLOB_MAX)
buf = Buf()
- fi = blobiter(files)
+ fi = blobiter(files, progress)
while 1:
for i in drainbuf(buf, finalize=False):
yield i
buf.put(bnew)
+def _hashsplit_iter_keep_boundaries(files, progress):
+ for real_filenum,f in enumerate(files):
+ if progress:
+ def prog(filenum, nbytes):
+ # the inner _hashsplit_iter doesn't know the real file count,
+ # so we'll replace it here.
+ return progress(real_filenum, nbytes)
+ else:
+ prog = None
+ for i in _hashsplit_iter([f], progress=prog):
+ yield i
+
+
+def hashsplit_iter(files, keep_boundaries, progress):
+ if keep_boundaries:
+ return _hashsplit_iter_keep_boundaries(files, progress)
+ else:
+ return _hashsplit_iter(files, progress)
+
+
total_split = 0
-def _split_to_blobs(w, files):
+def _split_to_blobs(w, files, keep_boundaries, progress):
global total_split
- for (blob, bits) in hashsplit_iter(files):
+ for (blob, bits) in hashsplit_iter(files, keep_boundaries, progress):
sha = w.new_blob(blob)
total_split += len(blob)
if w.outbytes >= max_pack_size or w.count >= max_pack_objects:
i += 1
-def split_to_shalist(w, files):
- sl = _split_to_blobs(w, files)
+def split_to_shalist(w, files, keep_boundaries, progress=None):
+ sl = _split_to_blobs(w, files, keep_boundaries, progress)
if not fanout:
shal = []
for (sha,size,bits) in sl:
return _make_shalist(stacks[-1])[0]
-def split_to_blob_or_tree(w, files):
- shalist = list(split_to_shalist(w, files))
+def split_to_blob_or_tree(w, files, keep_boundaries):
+ shalist = list(split_to_shalist(w, files, keep_boundaries))
if len(shalist) == 1:
return (shalist[0][0], shalist[0][2])
elif len(shalist) == 0:
def fadvise_done(f, ofs):
assert(ofs >= 0)
- if ofs > 0:
+ if ofs > 0 and hasattr(f, 'fileno'):
_helpers.fadvise_done(f.fileno(), ofs)
"""Helper functions and classes for bup."""
-import sys, os, pwd, subprocess, errno, socket, select, mmap, stat, re
+
+import sys, os, pwd, subprocess, errno, socket, select, mmap, stat, re, struct
+import heapq, operator
from bup import _version
import bup._helpers as _helpers
return None
+def merge_iter(iters, pfreq, pfunc, pfinal, key=None):
+ if key:
+ samekey = lambda e, pe: getattr(e, key) == getattr(pe, key, None)
+ else:
+ samekey = operator.eq
+ count = 0
+ total = sum(len(it) for it in iters)
+ iters = (iter(it) for it in iters)
+ heap = ((next(it),it) for it in iters)
+ heap = [(e,it) for e,it in heap if e]
+
+ heapq.heapify(heap)
+ pe = None
+ while heap:
+ if not count % pfreq:
+ pfunc(count, total)
+ e, it = heap[0]
+ if not samekey(e, pe):
+ pe = e
+ yield e
+ count += 1
+ try:
+ e = it.next() # Don't use next() function, it's too expensive
+ except StopIteration:
+ heapq.heappop(heap) # remove current
+ else:
+ heapq.heapreplace(heap, (e, it)) # shift current to new location
+ pfinal(count, total)
+
+
def unlink(f):
"""Delete a file at path 'f' if it currently exists.
_resource_path = os.environ.get('BUP_RESOURCE_PATH') or '.'
return os.path.join(_resource_path, subdir)
+
class NotOk(Exception):
pass
-class Conn:
- """A helper class for bup's client-server protocol."""
- def __init__(self, inp, outp):
- self.inp = inp
+
+class BaseConn:
+ def __init__(self, outp):
self.outp = outp
+ def close(self):
+ while self._read(65536): pass
+
def read(self, size):
"""Read 'size' bytes from input stream."""
self.outp.flush()
- return self.inp.read(size)
+ return self._read(size)
def readline(self):
"""Read from input stream until a newline is found."""
self.outp.flush()
- return self.inp.readline()
+ return self._readline()
def write(self, data):
"""Write 'data' to output stream."""
def has_input(self):
"""Return true if input stream is readable."""
- [rl, wl, xl] = select.select([self.inp.fileno()], [], [], 0)
- if rl:
- assert(rl[0] == self.inp.fileno())
- return True
- else:
- return None
+ raise NotImplemented("Subclasses must implement has_input")
def ok(self):
"""Indicate end of output from last sent command."""
def _check_ok(self, onempty):
self.outp.flush()
rl = ''
- for rl in linereader(self.inp):
+ for rl in linereader(self):
#log('%d got line: %r\n' % (os.getpid(), rl))
if not rl: # empty line
continue
return self._check_ok(onempty)
+class Conn(BaseConn):
+ def __init__(self, inp, outp):
+ BaseConn.__init__(self, outp)
+ self.inp = inp
+
+ def _read(self, size):
+ return self.inp.read(size)
+
+ def _readline(self):
+ return self.inp.readline()
+
+ def has_input(self):
+ [rl, wl, xl] = select.select([self.inp.fileno()], [], [], 0)
+ if rl:
+ assert(rl[0] == self.inp.fileno())
+ return True
+ else:
+ return None
+
+
+def checked_reader(fd, n):
+ while n > 0:
+ rl, _, _ = select.select([fd], [], [])
+ assert(rl[0] == fd)
+ buf = os.read(fd, n)
+ if not buf: raise Exception("Unexpected EOF reading %d more bytes" % n)
+ yield buf
+ n -= len(buf)
+
+
+MAX_PACKET = 128 * 1024
+def mux(p, outfd, outr, errr):
+ try:
+ fds = [outr, errr]
+ while p.poll() is None:
+ rl, _, _ = select.select(fds, [], [])
+ for fd in rl:
+ if fd == outr:
+ buf = os.read(outr, MAX_PACKET)
+ if not buf: break
+ os.write(outfd, struct.pack('!IB', len(buf), 1) + buf)
+ elif fd == errr:
+ buf = os.read(errr, 1024)
+ if not buf: break
+ os.write(outfd, struct.pack('!IB', len(buf), 2) + buf)
+ finally:
+ os.write(outfd, struct.pack('!IB', 0, 3))
+
+
+class DemuxConn(BaseConn):
+ """A helper class for bup's client-server protocol."""
+ def __init__(self, infd, outp):
+ BaseConn.__init__(self, outp)
+ # Anything that comes through before the sync string was not
+ # multiplexed and can be assumed to be debug/log before mux init.
+ tail = ''
+ while tail != 'BUPMUX':
+ b = os.read(infd, (len(tail) < 6) and (6-len(tail)) or 1)
+ if not b:
+ raise IOError('demux: unexpected EOF during initialization')
+ tail += b
+ sys.stderr.write(tail[:-6]) # pre-mux log messages
+ tail = tail[-6:]
+ self.infd = infd
+ self.reader = None
+ self.buf = None
+ self.closed = False
+
+ def write(self, data):
+ self._load_buf(0)
+ BaseConn.write(self, data)
+
+ def _next_packet(self, timeout):
+ if self.closed: return False
+ rl, wl, xl = select.select([self.infd], [], [], timeout)
+ if not rl: return False
+ assert(rl[0] == self.infd)
+ ns = ''.join(checked_reader(self.infd, 5))
+ n, fdw = struct.unpack('!IB', ns)
+ assert(n <= MAX_PACKET)
+ if fdw == 1:
+ self.reader = checked_reader(self.infd, n)
+ elif fdw == 2:
+ for buf in checked_reader(self.infd, n):
+ sys.stderr.write(buf)
+ elif fdw == 3:
+ self.closed = True
+ debug2("DemuxConn: marked closed\n")
+ return True
+
+ def _load_buf(self, timeout):
+ if self.buf is not None:
+ return True
+ while not self.closed:
+ while not self.reader:
+ if not self._next_packet(timeout):
+ return False
+ try:
+ self.buf = self.reader.next()
+ return True
+ except StopIteration:
+ self.reader = None
+ return False
+
+ def _read_parts(self, ix_fn):
+ while self._load_buf(None):
+ assert(self.buf is not None)
+ i = ix_fn(self.buf)
+ if i is None or i == len(self.buf):
+ yv = self.buf
+ self.buf = None
+ else:
+ yv = self.buf[:i]
+ self.buf = self.buf[i:]
+ yield yv
+ if i is not None:
+ break
+
+ def _readline(self):
+ def find_eol(buf):
+ try:
+ return buf.index('\n')+1
+ except ValueError:
+ return None
+ return ''.join(self._read_parts(find_eol))
+
+ def _read(self, size):
+ csize = [size]
+ def until_size(buf): # Closes on csize
+ if len(buf) < csize[0]:
+ csize[0] -= len(buf)
+ return None
+ else:
+ return csize[0]
+ return ''.join(self._read_parts(until_size))
+
+ def has_input(self):
+ return self._load_buf(0)
+
+
def linereader(f):
"""Generate a list of input lines from 'f' without terminating newlines."""
while 1:
return s
-def _mmap_do(f, sz, flags, prot):
+def _mmap_do(f, sz, flags, prot, close):
if not sz:
st = os.fstat(f.fileno())
sz = st.st_size
+ if not sz:
+ # trying to open a zero-length map gives an error, but an empty
+ # string has all the same behaviour of a zero-length map, ie. it has
+ # no elements :)
+ return ''
map = mmap.mmap(f.fileno(), sz, flags, prot)
- f.close() # map will persist beyond file close
+ if close:
+ f.close() # map will persist beyond file close
return map
-def mmap_read(f, sz = 0):
+def mmap_read(f, sz = 0, close=True):
"""Create a read-only memory mapped region on file 'f'.
-
If sz is 0, the region will cover the entire file.
"""
- return _mmap_do(f, sz, mmap.MAP_PRIVATE, mmap.PROT_READ)
+ return _mmap_do(f, sz, mmap.MAP_PRIVATE, mmap.PROT_READ, close)
-def mmap_readwrite(f, sz = 0):
+def mmap_readwrite(f, sz = 0, close=True):
"""Create a read-write memory mapped region on file 'f'.
+ If sz is 0, the region will cover the entire file.
+ """
+ return _mmap_do(f, sz, mmap.MAP_SHARED, mmap.PROT_READ|mmap.PROT_WRITE,
+ close)
+
+def mmap_readwrite_private(f, sz = 0, close=True):
+ """Create a read-write memory mapped region on file 'f'.
If sz is 0, the region will cover the entire file.
+ The map is private, which means the changes are never flushed back to the
+ file.
"""
- return _mmap_do(f, sz, mmap.MAP_SHARED, mmap.PROT_READ|mmap.PROT_WRITE)
+ return _mmap_do(f, sz, mmap.MAP_PRIVATE, mmap.PROT_READ|mmap.PROT_WRITE,
+ close)
def parse_num(s):
out += prefix + ''.join(('%-*s' % (clen+2, s)) for s in row) + '\n'
return out
+
def parse_date_or_fatal(str, fatal):
"""Parses the given date or calls Option.fatal().
For now we expect a string that contains a float."""
return date
+def strip_path(prefix, path):
+ """Strips a given prefix from a path.
+
+ First both paths are normalized.
+
+ Raises an Exception if no prefix is given.
+ """
+ if prefix == None:
+ raise Exception('no path given')
+
+ normalized_prefix = os.path.realpath(prefix)
+ debug2("normalized_prefix: %s\n" % normalized_prefix)
+ normalized_path = os.path.realpath(path)
+ debug2("normalized_path: %s\n" % normalized_path)
+ if normalized_path.startswith(normalized_prefix):
+ return normalized_path[len(normalized_prefix):]
+ else:
+ return path
+
+
+def strip_base_path(path, base_paths):
+ """Strips the base path from a given path.
+
+
+ Determines the base path for the given string and then strips it
+ using strip_path().
+ Iterates over all base_paths from long to short, to prevent that
+ a too short base_path is removed.
+ """
+ normalized_path = os.path.realpath(path)
+ sorted_base_paths = sorted(base_paths, key=len, reverse=True)
+ for bp in sorted_base_paths:
+ if normalized_path.startswith(os.path.realpath(bp)):
+ return strip_path(bp, normalized_path)
+ return path
+
+
+def graft_path(graft_points, path):
+ normalized_path = os.path.realpath(path)
+ for graft_point in graft_points:
+ old_prefix, new_prefix = graft_point
+ if normalized_path.startswith(old_prefix):
+ return re.sub(r'^' + old_prefix, new_prefix, normalized_path)
+ return normalized_path
+
+
# hashlib is only available in python 2.5 or higher, but the 'sha' module
# produces a DeprecationWarning in python 2.6 or higher. We want to support
# python 2.4 and above without any stupid warnings, so let's try using hashlib
"""Format bup's version date string for output."""
return _version.DATE.split(' ')[0]
+
def version_commit():
"""Get the commit hash of bup's current version."""
return _version.COMMIT
+
def version_tag():
"""Format bup's version tag (the official version number).
return not self.ctime
def __cmp__(a, b):
- return (cmp(a.name, b.name)
- or -cmp(a.is_valid(), b.is_valid())
- or -cmp(a.is_fake(), b.is_fake()))
+ return (cmp(b.name, a.name)
+ or cmp(a.is_valid(), b.is_valid())
+ or cmp(a.is_fake(), b.is_fake()))
def write(self, f):
f.write(self.basename + '\0' + self.packed())
paths.sort(reverse=True)
return paths
-
-class MergeIter:
- def __init__(self, iters):
- self.iters = iters
-
- def __len__(self):
- # FIXME: doesn't remove duplicated entries between iters.
- # That only happens for parent directories, but will mean the
- # actual iteration returns fewer entries than this function counts.
- return sum(len(it) for it in self.iters)
-
- def __iter__(self):
- total = len(self)
- l = [iter(it) for it in self.iters]
- l = [(next(it),it) for it in l]
- l = filter(lambda x: x[0], l)
- count = 0
- lastname = None
- while l:
- if not (count % 1024):
- progress('bup: merging indexes (%d/%d)\r' % (count, total))
- l.sort()
- (e,it) = l.pop()
- if not e:
- continue
- if e.name != lastname:
- yield e
- lastname = e.name
- n = next(it)
- if n:
- l.append((n,it))
- count += 1
+def merge(*iters):
+ def pfunc(count, total):
+ progress('bup: merging indexes (%d/%d)\r' % (count, total))
+ def pfinal(count, total):
log('bup: merging indexes (%d/%d), done.\n' % (count, total))
+ return merge_iter(iters, 1024, pfunc, pfinal, key='name')
"""Command-line options parser.
With the help of an options spec string, easily parse command-line options.
+
+An options spec is made up of two parts, separated by a line with two dashes.
+The first part is the synopsis of the command and the second one specifies
+options, one per line.
+
+Each non-empty line in the synopsis gives a set of options that can be used
+together.
+
+Option flags must be at the begining of the line and multiple flags are
+separated by commas. Usually, options have a short, one character flag, and a
+longer one, but the short one can be omitted.
+
+Long option flags are used as the option's key for the OptDict produced when
+parsing options.
+
+When the flag definition is ended with an equal sign, the option takes one
+string as an argument. Otherwise, the option does not take an argument and
+corresponds to a boolean flag that is true when the option is given on the
+command line.
+
+The option's description is found at the right of its flags definition, after
+one or more spaces. The description ends at the end of the line. If the
+description contains text enclosed in square brackets, the enclosed text will
+be used as the option's default value.
+
+Options can be put in different groups. Options in the same group must be on
+consecutive lines. Groups are formed by inserting a line that begins with a
+space. The text on that line will be output after an empty line.
"""
import sys, os, textwrap, getopt, re, struct
class OptDict:
+ """Dictionary that exposes keys as attributes.
+
+ Keys can bet set or accessed with a "no-" or "no_" prefix to negate the
+ value.
+ """
def __init__(self):
self._opts = {}
except (IOError, ImportError):
return _atoi(os.environ.get('WIDTH')) or 70
(ysize,xsize,ypix,xpix) = struct.unpack('HHHH', s)
- return xsize
+ return xsize or 70
class Options:
"""Option parser.
- When constructed, two strings are mandatory. The first one is the command
- name showed before error messages. The second one is a string called an
- optspec that specifies the synopsis and option flags and their description.
- For more information about optspecs, consult the bup-options(1) man page.
+ When constructed, a string called an option spec must be given. It
+ specifies the synopsis and option flags and their description. For more
+ information about option specs, see the docstring at the top of this file.
Two optional arguments specify an alternative parsing function and an
alternative behaviour on abort (after having output the usage string).
By default, the parser function is getopt.gnu_getopt, and the abort
behaviour is to exit the program.
"""
- def __init__(self, exe, optspec, optfunc=getopt.gnu_getopt,
+ def __init__(self, optspec, optfunc=getopt.gnu_getopt,
onabort=_default_onabort):
- self.exe = exe
self.optspec = optspec
self._onabort = onabort
self.optfunc = optfunc
defval = None
flagl = flags.split(',')
flagl_nice = []
- for f in flagl:
- f,dvi = _remove_negative_kv(f, _intify(defval))
+ for _f in flagl:
+ f,dvi = _remove_negative_kv(_f, _intify(defval))
self._aliases[f] = _remove_negative_k(flagl[0])
self._hasparms[f] = has_parm
self._defaults[f] = dvi
self._aliases[f_nice] = _remove_negative_k(flagl[0])
self._longopts.append(f + (has_parm and '=' or ''))
self._longopts.append('no-' + f)
- flagl_nice.append('--' + f)
+ flagl_nice.append('--' + _f)
flags_nice = ', '.join(flagl_nice)
if has_parm:
flags_nice += ' ...'
--- /dev/null
+"""This is a separate module so we can cleanly getcwd() before anyone
+ does chdir().
+"""
+import sys, os
+
+startdir = os.getcwd()
+
+def exe():
+ return (os.environ.get('BUP_MAIN_EXE') or
+ os.path.join(startdir, sys.argv[0]))
+
+def exedir():
+ return os.path.split(exe())[0]
+
+def exefile():
+ return os.path.split(exe())[1]
"""SSH connection.
Connect to a remote host via SSH and execute a command on the host.
"""
-import os
-import sys
-import re
-import subprocess
+import sys, os, re, subprocess
+from bup import helpers, path
-from bup import helpers
-
-def connect(rhost, subcmd):
+def connect(rhost, port, subcmd):
"""Connect to 'rhost' and execute the bup subcommand 'subcmd' on it."""
assert(not re.search(r'[^\w-]', subcmd))
- main_exe = os.environ.get('BUP_MAIN_EXE') or sys.argv[0]
- nicedir = os.path.split(os.path.abspath(main_exe))[0]
- nicedir = re.sub(r':', "_", nicedir)
+ nicedir = re.sub(r':', "_", path.exedir())
if rhost == '-':
rhost = None
if not rhost:
cmd = r"""
sh -c PATH=%s:'$PATH BUP_DEBUG=%s BUP_FORCE_TTY=%s bup %s'
""" % (escapedir, buglvl, force_tty, subcmd)
- argv = ['ssh', rhost, '--', cmd.strip()]
+ argv = ['ssh']
+ if port:
+ argv.extend(('-p', port))
+ argv.extend((rhost, '--', cmd.strip()))
#helpers.log('argv is: %r\n' % argv)
def setup():
# runs in the child process
-import sys, os, time, random, subprocess
+import sys, os, stat, time, random, subprocess, glob
from bup import client, git
from wvtest import *
s1 = randbytes(10000)
s2 = randbytes(10000)
+s3 = randbytes(10000)
+
+IDX_PAT = '/*.idx'
@wvtest
def test_server_split_with_indexes():
rw.new_blob(s1)
+@wvtest
+def test_multiple_suggestions():
+ os.environ['BUP_MAIN_EXE'] = '../../../bup'
+ os.environ['BUP_DIR'] = bupdir = 'buptest_tclient.tmp'
+ subprocess.call(['rm', '-rf', bupdir])
+ git.init_repo(bupdir)
+
+ lw = git.PackWriter()
+ lw.new_blob(s1)
+ lw.close()
+ lw = git.PackWriter()
+ lw.new_blob(s2)
+ lw.close()
+ WVPASSEQ(len(glob.glob(git.repo('objects/pack'+IDX_PAT))), 2)
+
+ c = client.Client(bupdir, create=True)
+ WVPASSEQ(len(glob.glob(c.cachedir+IDX_PAT)), 0)
+ rw = c.new_packwriter()
+ rw.new_blob(s1)
+ rw.new_blob(s2)
+ # This is a little hacky, but ensures that we test the code under test
+ while len(glob.glob(c.cachedir+IDX_PAT)) < 2 and not c.conn.has_input(): pass
+ rw.new_blob(s3)
+ WVPASSEQ(len(glob.glob(c.cachedir+IDX_PAT)), 2)
+ rw.close()
+ WVPASSEQ(len(glob.glob(c.cachedir+IDX_PAT)), 3)
+
+
+@wvtest
+def test_dumb_client_server():
+ os.environ['BUP_MAIN_EXE'] = '../../../bup'
+ os.environ['BUP_DIR'] = bupdir = 'buptest_tclient.tmp'
+ subprocess.call(['rm', '-rf', bupdir])
+ git.init_repo(bupdir)
+ open(git.repo('bup-dumb-server'), 'w').close()
+
+ lw = git.PackWriter()
+ lw.new_blob(s1)
+ lw.close()
+
+ c = client.Client(bupdir, create=True)
+ rw = c.new_packwriter()
+ WVPASSEQ(len(glob.glob(c.cachedir+IDX_PAT)), 1)
+ rw.new_blob(s1)
+ WVPASSEQ(len(glob.glob(c.cachedir+IDX_PAT)), 1)
+ rw.new_blob(s2)
+ rw.close()
+ WVPASSEQ(len(glob.glob(c.cachedir+IDX_PAT)), 2)
+
+
@wvtest
def test_midx_refreshing():
os.environ['BUP_MAIN_EXE'] = bupmain = '../../../bup'
WVPASSEQ(len(pi.packs), 2)
pi.refresh(skip_midx=False)
WVPASSEQ(len(pi.packs), 1)
+
+@wvtest
+def test_remote_parsing():
+ tests = (
+ (':/bup', ('file', None, None, '/bup')),
+ ('file:///bup', ('file', None, None, '/bup')),
+ ('192.168.1.1:/bup', ('ssh', '192.168.1.1', None, '/bup')),
+ ('ssh://192.168.1.1:2222/bup', ('ssh', '192.168.1.1', '2222', '/bup')),
+ ('ssh://[ff:fe::1]:2222/bup', ('ssh', 'ff:fe::1', '2222', '/bup')),
+ ('bup://foo.com:1950', ('bup', 'foo.com', '1950', None)),
+ ('bup://foo.com:1950/bup', ('bup', 'foo.com', '1950', '/bup')),
+ ('bup://[ff:fe::1]/bup', ('bup', 'ff:fe::1', None, '/bup')),
+ )
+ for remote, values in tests:
+ WVPASSEQ(client.parse_remote(remote), values)
+ try:
+ client.parse_remote('http://asdf.com/bup')
+ WVFAIL()
+ except AssertionError:
+ WVPASS()
-import time
+import struct, os, tempfile, time
from bup import git
from bup.helpers import *
from wvtest import *
@wvtest
def testpacks():
+ subprocess.call(['rm','-rf', 'pybuptest.tmp'])
git.init_repo('pybuptest.tmp')
git.verbose = 1
- now = str(time.time()) # hopefully not in any packs yet
w = git.PackWriter()
- w.write('blob', now)
- w.write('blob', now)
+ w.new_blob(os.urandom(100))
+ w.new_blob(os.urandom(100))
w.abort()
w = git.PackWriter()
hashes = []
nobj = 1000
for i in range(nobj):
- hashes.append(w.write('blob', str(i)))
+ hashes.append(w.new_blob(str(i)))
log('\n')
nameprefix = w.close()
print repr(nameprefix)
WVPASS(os.path.exists(nameprefix + '.pack'))
WVPASS(os.path.exists(nameprefix + '.idx'))
- r = git.PackIdx(nameprefix + '.idx')
+ r = git.open_idx(nameprefix + '.idx')
print repr(r.fanout)
for i in range(nobj):
WVPASS(r.exists(hashes[5]))
WVPASS(r.exists(hashes[6]))
WVFAIL(r.exists('\0'*20))
+
+
+@wvtest
+def test_pack_name_lookup():
+ os.environ['BUP_MAIN_EXE'] = bupmain = '../../../bup'
+ os.environ['BUP_DIR'] = bupdir = 'pybuptest.tmp'
+ subprocess.call(['rm','-rf', bupdir])
+ git.init_repo(bupdir)
+ git.verbose = 1
+
+ idxnames = []
+
+ w = git.PackWriter()
+ hashes = []
+ for i in range(2):
+ hashes.append(w.new_blob(str(i)))
+ log('\n')
+ idxnames.append(w.close() + '.idx')
+
+ w = git.PackWriter()
+ for i in range(2,4):
+ hashes.append(w.new_blob(str(i)))
+ log('\n')
+ idxnames.append(w.close() + '.idx')
+
+ idxnames = [os.path.basename(ix) for ix in idxnames]
+
+ def verify(r):
+ for i in range(2):
+ WVPASSEQ(r.exists(hashes[i], want_source=True), idxnames[0])
+ for i in range(2,4):
+ WVPASSEQ(r.exists(hashes[i], want_source=True), idxnames[1])
+
+ r = git.PackIdxList('pybuptest.tmp/objects/pack')
+ WVPASSEQ(len(r.packs), 2)
+ verify(r)
+ del r
+
+ subprocess.call([bupmain, 'midx', '-f'])
+
+ r = git.PackIdxList('pybuptest.tmp/objects/pack')
+ WVPASSEQ(len(r.packs), 1)
+ verify(r)
+
+
+@wvtest
+def test_long_index():
+ w = git.PackWriter()
+ obj_bin = struct.pack('!IIIII',
+ 0x00112233, 0x44556677, 0x88990011, 0x22334455, 0x66778899)
+ obj2_bin = struct.pack('!IIIII',
+ 0x11223344, 0x55667788, 0x99001122, 0x33445566, 0x77889900)
+ obj3_bin = struct.pack('!IIIII',
+ 0x22334455, 0x66778899, 0x00112233, 0x44556677, 0x88990011)
+ pack_bin = struct.pack('!IIIII',
+ 0x99887766, 0x55443322, 0x11009988, 0x77665544, 0x33221100)
+ idx = list(list() for i in xrange(256))
+ idx[0].append((obj_bin, 1, 0xfffffffff))
+ idx[0x11].append((obj2_bin, 2, 0xffffffffff))
+ idx[0x22].append((obj3_bin, 3, 0xff))
+ (fd,name) = tempfile.mkstemp(suffix='.idx', dir=git.repo('objects'))
+ f = os.fdopen(fd, 'w+b')
+ r = w._write_pack_idx_v2(f, idx, pack_bin)
+ f.seek(0)
+ i = git.PackIdxV2(name, f)
+ WVPASSEQ(i.find_offset(obj_bin), 0xfffffffff)
+ WVPASSEQ(i.find_offset(obj2_bin), 0xffffffffff)
+ WVPASSEQ(i.find_offset(obj3_bin), 0xff)
+ f.close()
+ os.remove(name)
+
+@wvtest
+def test_bloom():
+ hashes = [os.urandom(20) for i in range(100)]
+ class Idx:
+ pass
+ ix = Idx()
+ ix.name='dummy.idx'
+ ix.shatable = ''.join(hashes)
+ for k in (4, 5):
+ b = git.ShaBloom.create('pybuptest.bloom', expected=100, k=k)
+ b.add_idx(ix)
+ WVPASSLT(b.pfalse_positive(), .1)
+ b.close()
+ b = git.ShaBloom('pybuptest.bloom')
+ all_present = True
+ for h in hashes:
+ all_present &= b.exists(h)
+ WVPASS(all_present)
+ false_positives = 0
+ for h in [os.urandom(20) for i in range(1000)]:
+ if b.exists(h):
+ false_positives += 1
+ WVPASSLT(false_positives, 5)
+ os.unlink('pybuptest.bloom')
+
+ tf = tempfile.TemporaryFile()
+ b = git.ShaBloom.create('bup.bloom', f=tf, expected=100)
+ WVPASSEQ(b.rwfile, tf)
+ WVPASSEQ(b.k, 5)
+ tf = tempfile.TemporaryFile()
+ b = git.ShaBloom.create('bup.bloom', f=tf, expected=2**28,
+ delaywrite=False)
+ WVPASSEQ(b.k, 4)
-import os, math
+import math
+import os
import bup._helpers as _helpers
-
from bup.helpers import *
from wvtest import *
WVPASSEQ(pn('1e+9 k'), 1000000000 * 1024)
WVPASSEQ(pn('-3e-3mb'), int(-0.003 * 1024 * 1024))
-
@wvtest
def test_detect_fakeroot():
if os.getenv('FAKEROOTKEY'):
WVPASS(detect_fakeroot())
else:
WVPASS(not detect_fakeroot())
+
+@wvtest
+def test_strip_path():
+ prefix = "/var/backup/daily.0/localhost"
+ empty_prefix = ""
+ non_matching_prefix = "/home"
+ path = "/var/backup/daily.0/localhost/etc/"
+
+ WVPASSEQ(strip_path(prefix, path), '/etc')
+ WVPASSEQ(strip_path(empty_prefix, path), path)
+ WVPASSEQ(strip_path(non_matching_prefix, path), path)
+ WVEXCEPT(Exception, strip_path, None, path)
+
+@wvtest
+def test_strip_base_path():
+ path = "/var/backup/daily.0/localhost/etc/"
+ base_paths = ["/var", "/var/backup", "/var/backup/daily.0/localhost"]
+ WVPASSEQ(strip_base_path(path, base_paths), '/etc')
+
+@wvtest
+def test_strip_symlinked_base_path():
+ tmpdir = os.path.join(os.getcwd(),"test_strip_symlinked_base_path.tmp")
+ symlink_src = os.path.join(tmpdir, "private", "var")
+ symlink_dst = os.path.join(tmpdir, "var")
+ path = os.path.join(symlink_dst, "a")
+
+ os.mkdir(tmpdir)
+ os.mkdir(os.path.join(tmpdir, "private"))
+ os.mkdir(symlink_src)
+ os.symlink(symlink_src, symlink_dst)
+
+ result = strip_base_path(path, [symlink_dst])
+
+ os.remove(symlink_dst)
+ os.rmdir(symlink_src)
+ os.rmdir(os.path.join(tmpdir, "private"))
+ os.rmdir(tmpdir)
+
+ WVPASSEQ(result, "/a")
+
+@wvtest
+def test_graft_path():
+ middle_matching_old_path = "/user"
+ non_matching_old_path = "/usr"
+ matching_old_path = "/home"
+ matching_full_path = "/home/user"
+ new_path = "/opt"
+
+ all_graft_points = [(middle_matching_old_path, new_path),
+ (non_matching_old_path, new_path),
+ (matching_old_path, new_path)]
+
+ path = "/home/user/"
+
+ WVPASSEQ(graft_path([(middle_matching_old_path, new_path)], path),
+ "/home/user")
+ WVPASSEQ(graft_path([(non_matching_old_path, new_path)], path),
+ "/home/user")
+ WVPASSEQ(graft_path([(matching_old_path, new_path)], path), "/opt/user")
+ WVPASSEQ(graft_path(all_graft_points, path), "/opt/user")
+ WVPASSEQ(graft_path([(matching_full_path, new_path)], path),
+ "/opt")
r3all = [e.name for e in r3]
WVPASSEQ(r3all,
['/a/c/n/3', '/a/c/n/', '/a/c/', '/a/', '/'])
- m = index.MergeIter([r2,r1,r3])
- all = [e.name for e in m]
+ all = [e.name for e in index.merge(r2, r1, r3)]
WVPASSEQ(all,
['/a/c/n/3', '/a/c/n/', '/a/c/',
'/a/b/x', '/a/b/n/2', '/a/b/n/', '/a/b/c',
print [hex(e.flags) for e in r1]
WVPASSEQ([e.name for e in r1 if e.is_valid()], r1all)
WVPASSEQ([e.name for e in r1 if not e.is_valid()], [])
- WVPASSEQ([e.name for e in m if not e.is_valid()],
+ WVPASSEQ([e.name for e in index.merge(r2, r1, r3) if not e.is_valid()],
['/a/c/n/3', '/a/c/n/', '/a/c/',
'/a/b/n/2', '/a/b/n/', '/a/b/', '/a/', '/'])
expect_invalid = ['/'] + r2all + r3all
expect_real = (set(r1all) - set(r2all) - set(r3all)) \
| set(['/a/b/n/2', '/a/c/n/3'])
- dump(m)
- for e in m:
+ dump(index.merge(r2, r1, r3))
+ for e in index.merge(r2, r1, r3):
print e.name, hex(e.flags), e.ctime
eiv = e.name in expect_invalid
er = e.name in expect_real
WVPASSEQ(eiv, not e.is_valid())
WVPASSEQ(er, e.is_real())
fake_validate(r2, r3)
- dump(m)
- WVPASSEQ([e.name for e in m if not e.is_valid()], [])
+ dump(index.merge(r2, r1, r3))
+ WVPASSEQ([e.name for e in index.merge(r2, r1, r3) if not e.is_valid()], [])
- e = eget(m, '/a/b/c')
+ e = eget(index.merge(r2, r1, r3), '/a/b/c')
e.invalidate()
e.repack()
- dump(m)
- WVPASSEQ([e.name for e in m if not e.is_valid()],
+ dump(index.merge(r2, r1, r3))
+ WVPASSEQ([e.name for e in index.merge(r2, r1, r3) if not e.is_valid()],
['/a/b/c', '/a/b/', '/a/', '/'])
@wvtest
def test_options():
- o = options.Options('exename', optspec)
+ o = options.Options(optspec)
(opt,flags,extra) = o.parse(['-tttqp', 7, '--longoption', '19',
'hanky', '--onlylong'])
WVPASSEQ(flags[0], ('-t', ''))
yield ''.join(cp().join(sha.encode('hex')))[skipmore:]
-class _ChunkReader(object):
+class _ChunkReader:
def __init__(self, hash, isdir, startofs):
if isdir:
self.it = _chunkiter(hash, startofs)
pass
-class Node(object):
+class Node:
"""Base class for file representation."""
def __init__(self, parent, name, mode, hash):
self.parent = parent
def size(self):
"""Get this file's size."""
if self._cached_size == None:
- debug1('<<<<File.size() is calculating...\n')
+ debug1('<<<<File.size() is calculating (for %r)...\n' % self.name)
if self.bupmode == git.BUP_CHUNKED:
self._cached_size = _total_size(self.hash)
else:
self._subs[name] = File(self, name, mode, sha, bupmode)
+class CommitDir(Node):
+ """A directory that contains all commits that are reachable by a ref.
+
+ Contains a set of subdirectories named after the commits' first byte in
+ hexadecimal. Each of those directories contain all commits with hashes that
+ start the same as the directory name. The name used for those
+ subdirectories is the hash of the commit without the first byte. This
+ separation helps us avoid having too much directories on the same level as
+ the number of commits grows big.
+ """
+ def __init__(self, parent, name):
+ Node.__init__(self, parent, name, 040000, EMPTY_SHA)
+
+ def _mksubs(self):
+ self._subs = {}
+ refs = git.list_refs()
+ for ref in refs:
+ #debug2('ref name: %s\n' % ref[0])
+ revs = git.rev_list(ref[1].encode('hex'))
+ for (date, commit) in revs:
+ #debug2('commit: %s date: %s\n' % (commit.encode('hex'), date))
+ commithex = commit.encode('hex')
+ containername = commithex[:2]
+ dirname = commithex[2:]
+ n1 = self._subs.get(containername)
+ if not n1:
+ n1 = CommitList(self, containername)
+ self._subs[containername] = n1
+
+ if n1.commits.get(dirname):
+ # Stop work for this ref, the rest should already be present
+ break
+
+ n1.commits[dirname] = (commit, date)
+
+
class CommitList(Node):
- """A reverse-chronological list of commits on a branch in bup's repository.
+ """A list of commits with hashes that start with the current node's name."""
+ def __init__(self, parent, name):
+ Node.__init__(self, parent, name, 040000, EMPTY_SHA)
+ self.commits = {}
+
+ def _mksubs(self):
+ self._subs = {}
+ for (name, (hash, date)) in self.commits.items():
+ n1 = Dir(self, name, 040000, hash)
+ n1.ctime = n1.mtime = date
+ self._subs[name] = n1
+
+
+class TagDir(Node):
+ """A directory that contains all tags in the repository."""
+ def __init__(self, parent, name):
+ Node.__init__(self, parent, name, 040000, EMPTY_SHA)
+
+ def _mksubs(self):
+ self._subs = {}
+ for (name, sha) in git.list_refs():
+ if name.startswith('refs/tags/'):
+ name = name[10:]
+ date = git.rev_get_date(sha.encode('hex'))
+ commithex = sha.encode('hex')
+ target = '../.commit/%s/%s' % (commithex[:2], commithex[2:])
+ tag1 = FakeSymlink(self, name, target)
+ tag1.ctime = tag1.mtime = date
+ self._subs[name] = tag1
+
- Represents each commit as a directory and a symlink that points to the
- directory. The symlink is named after the date. Prepends a dot to each hash
- to make commits look like hidden directories.
+class BranchList(Node):
+ """A list of links to commits reachable by a branch in bup's repository.
+
+ Represents each commit as a symlink that points to the commit directory in
+ /.commit/??/ . The symlink is named after the commit date.
"""
def __init__(self, parent, name, hash):
Node.__init__(self, parent, name, 040000, hash)
def _mksubs(self):
self._subs = {}
+
+ tags = git.tags()
+
revs = list(git.rev_list(self.hash.encode('hex')))
for (date, commit) in revs:
l = time.localtime(date)
ls = time.strftime('%Y-%m-%d-%H%M%S', l)
- commithex = '.' + commit.encode('hex')
- n1 = Dir(self, commithex, 040000, commit)
- n2 = FakeSymlink(self, ls, commithex)
- n1.ctime = n1.mtime = n2.ctime = n2.mtime = date
- self._subs[commithex] = n1
- self._subs[ls] = n2
- latest = max(revs)
+ commithex = commit.encode('hex')
+ target = '../.commit/%s/%s' % (commithex[:2], commithex[2:])
+ n1 = FakeSymlink(self, ls, target)
+ n1.ctime = n1.mtime = date
+ self._subs[ls] = n1
+
+ for tag in tags.get(commit, []):
+ t1 = FakeSymlink(self, tag, target)
+ t1.ctime = t1.mtime = date
+ self._subs[tag] = t1
+
+ latest = max(revs)
if latest:
(date, commit) = latest
- commithex = '.' + commit.encode('hex')
- n2 = FakeSymlink(self, 'latest', commithex)
- n2.ctime = n2.mtime = date
- self._subs['latest'] = n2
+ commithex = commit.encode('hex')
+ target = '../.commit/%s/%s' % (commithex[:2], commithex[2:])
+ n1 = FakeSymlink(self, 'latest', target)
+ n1.ctime = n1.mtime = date
+ self._subs['latest'] = n1
class RefList(Node):
The sub-nodes of the ref list are a series of CommitList for each commit
hash pointed to by a branch.
+
+ Also, a special sub-node named '.commit' contains all commit directories
+ that are reachable via a ref (e.g. a branch). See CommitDir for details.
"""
def __init__(self, parent):
Node.__init__(self, parent, '/', 040000, EMPTY_SHA)
def _mksubs(self):
self._subs = {}
+
+ commit_dir = CommitDir(self, '.commit')
+ self._subs['.commit'] = commit_dir
+
+ tag_dir = TagDir(self, '.tag')
+ self._subs['.tag'] = tag_dir
+
for (name,sha) in git.list_refs():
if name.startswith('refs/heads/'):
name = name[11:]
date = git.rev_get_date(sha.encode('hex'))
- n1 = CommitList(self, name, sha)
+ n1 = BranchList(self, name, sha)
n1.ctime = n1.mtime = date
self._subs[name] = n1
-
-
}
#wrapper {
- width: 960px;
+ width: 90%;
margin: auto;
}
}
table {
- width: 100%;
+ width: auto;
}
th {
text-align: left;
}
-.dir-name {
- width:80%;
-}
-
.dir-size {
- width:20%;
+ padding-left:15px;
}
\ No newline at end of file
os.environ['WIDTH'] = str(tty_width())
def usage():
- log('Usage: bup [-?|--help] [-d BUP_DIR] [--debug] '
+ log('Usage: bup [-?|--help] [-d BUP_DIR] [--debug] [--profile]'
'<command> [options...]\n\n')
common = dict(
ftp = 'Browse backup sets using an ftp-like client',
on = 'Backup a remote machine to the local one',
restore = 'Extract files from a backup set',
save = 'Save files into a backup set (note: run "bup index" first)',
+ tag = 'Tag commits for easier access',
web = 'Launch a web server to examine backup sets',
)
# Handle global options.
try:
- global_args, subcmd = getopt.getopt(argv[1:], '?VDd:',
- ['help', 'version', 'debug', 'bup-dir='])
+ optspec = ['help', 'version', 'debug', 'profile', 'bup-dir=']
+ global_args, subcmd = getopt.getopt(argv[1:], '?VDd:', optspec)
except getopt.GetoptError, ex:
log('error: ' + ex.msg + '\n')
usage()
help_requested = None
dest_dir = None
+do_profile = False
for opt in global_args:
if opt[0] in ['-?', '--help']:
elif opt[0] in ['-D', '--debug']:
helpers.buglvl += 1
os.environ['BUP_DEBUG'] = str(helpers.buglvl)
+ elif opt[0] in ['--profile']:
+ do_profile = True
elif opt[0] in ['-d', '--bup-dir']:
dest_dir = opt[1]
else:
sp = os.path.join(cmdpath, 'bup-%s' % s)
return sp
-if not os.path.exists(subpath(subcmd_name)):
+subcmd[0] = subpath(subcmd_name)
+if not os.path.exists(subcmd[0]):
log('error: unknown command "%s"\n' % subcmd_name)
usage()
already_fixed = atoi(os.environ.get('BUP_FORCE_TTY'))
-if subcmd_name in ['ftp', 'help']:
+if subcmd_name in ['mux', 'ftp', 'help']:
already_fixed = True
fix_stdout = not already_fixed and os.isatty(1)
fix_stderr = not already_fixed and os.isatty(2)
p = None
try:
try:
- p = subprocess.Popen([subpath(subcmd_name)] + subcmd[1:],
- stdout=outf, stderr=errf, preexec_fn=force_tty)
+ c = (do_profile and [sys.executable, '-m', 'cProfile'] or []) + subcmd
+ p = subprocess.Popen(c, stdout=outf, stderr=errf, preexec_fn=force_tty)
while 1:
# if we get a signal while waiting, we have to keep waiting, just
# in case our child doesn't die.
os.kill(p.pid, e.signum)
ret = 94
except OSError, e:
- log('%s: %s\n' % (subpath(subcmd_name), e))
+ log('%s: %s\n' % (subcmd[0], e))
ret = 98
finally:
if p and p.poll() == None:
. wvtest.sh
#set -e
-TOP="$(pwd)"
+TOP="$(/bin/pwd)"
export BUP_DIR="$TOP/buptest.tmp"
bup()
WVPASSEQ "$(bup index -s / | grep ^D)" ""
tree3=$(bup save -t /)
WVPASSEQ "$tree1" "$tree3"
-WVFAIL bup save -r localhost -n r-test $D
WVPASS bup save -r :$BUP_DIR -n r-test $D
WVFAIL bup save -r :$BUP_DIR/fake/path -n r-test $D
WVFAIL bup save -r :$BUP_DIR -n r-test $D/fake/path
WVSTART "split"
+echo a >a.tmp
+echo b >b.tmp
+WVPASS bup split -b a.tmp >taga.tmp
+WVPASS bup split -b b.tmp >tagb.tmp
+cat a.tmp b.tmp | WVPASS bup split -b >tagab.tmp
+WVPASSEQ $(cat taga.tmp | wc -l) 1
+WVPASSEQ $(cat tagb.tmp | wc -l) 1
+WVPASSEQ $(cat tagab.tmp | wc -l) 1
+WVPASSEQ $(cat tag[ab].tmp | wc -l) 2
+WVPASSEQ "$(bup split -b a.tmp b.tmp)" "$(cat tagab.tmp)"
+WVPASSEQ "$(bup split -b --keep-boundaries a.tmp b.tmp)" "$(cat tag[ab].tmp)"
+WVPASSEQ "$(cat tag[ab].tmp | bup split -b --keep-boundaries --git-ids)" \
+ "$(cat tag[ab].tmp)"
+WVPASSEQ "$(cat tag[ab].tmp | bup split -b --git-ids)" \
+ "$(cat tagab.tmp)"
WVPASS bup split --bench -b <t/testfile1 >tags1.tmp
WVPASS bup split -vvvv -b t/testfile2 >tags2.tmp
WVPASS bup margin
WVPASS bup margin
WVPASS bup split -t t/testfile2 >tags2t.tmp
WVPASS bup split -t t/testfile2 --fanout 3 >tags2tf.tmp
-WVFAIL bup split -r $BUP_DIR -c t/testfile2 >tags2c.tmp
+WVPASS bup split -r "$BUP_DIR" -c t/testfile2 >tags2c.tmp
WVPASS bup split -r :$BUP_DIR -c t/testfile2 >tags2c.tmp
WVPASS ls -lR \
| WVPASS bup split -r :$BUP_DIR -c --fanout 3 --max-pack-objects 3 -n lslr
WVFAIL diff -u tags1.tmp tags2.tmp
# fanout must be different from non-fanout
-WVFAIL diff -q tags2t.tmp tags2tf.tmp
+WVFAIL diff tags2t.tmp tags2tf.tmp
wc -c t/testfile1 t/testfile2
wc -l tags1.tmp tags2.tmp
WVPASS bup join $(cat tags1.tmp) >out1.tmp
WVPASS bup join <tags2.tmp >out2.tmp
WVPASS bup join <tags2t.tmp >out2t.tmp
-WVFAIL bup join -r "$BUP_DIR" <tags2c.tmp >out2c.tmp
+WVPASS bup join -r "$BUP_DIR" <tags2c.tmp >out2c.tmp
WVPASS bup join -r ":$BUP_DIR" <tags2c.tmp >out2c.tmp
WVPASS diff -u t/testfile1 out1.tmp
WVPASS diff -u t/testfile2 out2.tmp
#git prune
(cd "$TOP/t/sampledata" && WVPASS bup save -vvn master /) || WVFAIL
n=$(git fsck --full --strict 2>&1 |
- egrep -v 'dangling (commit|tree)' |
- tee -a /dev/stderr |
- wc -l)
+ egrep -v 'dangling (commit|tree)' |
+ tee -a /dev/stderr |
+ wc -l)
WVPASS [ "$n" -eq 0 ]
) || exit 1
WVPASSEQ "$(cat $D/f.new{,} | sha1sum)" "$(sha1sum <$D/f2.new)"
WVPASSEQ "$(sha1sum <$D/a)" "$(sha1sum <$D/a.new)"
+WVSTART "tag"
+WVFAIL bup tag -d v0.n 2>/dev/null
+WVFAIL bup tag v0.n non-existant 2>/dev/null
+WVPASSEQ "$(bup tag)" ""
+WVPASS bup tag v0.1 master
+WVPASSEQ "$(bup tag)" "v0.1"
+WVPASS bup tag -d v0.1
+
+# This section destroys data in the bup repository, so it is done last.
WVSTART "fsck"
WVPASS bup fsck
WVPASS bup fsck --quick
WVFAIL bup fsck --quick -r # still fails because par2 was missing
fi
-exit 0
+WVSTART "exclude-bupdir"
+D=exclude-bupdir.tmp
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+WVPASS bup random 128k >$D/b
+mkdir $D/d $D/d/e
+WVPASS bup random 512 >$D/f
+WVPASS bup index -ux $D
+bup save -n exclude-bupdir $D
+WVPASSEQ "$(bup ls exclude-bupdir/latest/$TOP/$D/)" "a
+b
+d/
+f"
+
+WVSTART "exclude"
+D=exclude.tmp
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+WVPASS bup random 128k >$D/b
+mkdir $D/d $D/d/e
+WVPASS bup random 512 >$D/f
+WVPASS bup index -ux --exclude $D/d $D
+bup save -n exclude $D
+WVPASSEQ "$(bup ls exclude/latest/$TOP/$D/)" "a
+b
+f"
+mkdir $D/g $D/h
+WVPASS bup index -ux --exclude $D/d --exclude $TOP/$D/g --exclude $D/h $D
+bup save -n exclude $D
+WVPASSEQ "$(bup ls exclude/latest/$TOP/$D/)" "a
+b
+f"
+
+WVSTART "exclude-from"
+D=exclude-fromdir.tmp
+EXCLUDE_FILE=exclude-from.tmp
+echo "$D/d
+ $TOP/$D/g
+$D/h" > $EXCLUDE_FILE
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+WVPASS bup random 128k >$D/b
+mkdir $D/d $D/d/e
+WVPASS bup random 512 >$D/f
+mkdir $D/g $D/h
+WVPASS bup index -ux --exclude-from $EXCLUDE_FILE $D
+bup save -n exclude-from $D
+WVPASSEQ "$(bup ls exclude-from/latest/$TOP/$D/)" "a
+b
+f"
+rm $EXCLUDE_FILE
+
+WVSTART "strip"
+D=strip.tmp
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+WVPASS bup random 128k >$D/b
+mkdir $D/d $D/d/e
+WVPASS bup random 512 >$D/f
+WVPASS bup index -ux $D
+bup save --strip -n strip $D
+WVPASSEQ "$(bup ls strip/latest/)" "a
+b
+d/
+f"
+
+WVSTART "strip-path"
+D=strip-path.tmp
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+WVPASS bup random 128k >$D/b
+mkdir $D/d $D/d/e
+WVPASS bup random 512 >$D/f
+WVPASS bup index -ux $D
+bup save --strip-path $TOP -n strip-path $D
+WVPASSEQ "$(bup ls strip-path/latest/$D/)" "a
+b
+d/
+f"
+
+WVSTART "graft_points"
+D=graft-points.tmp
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+WVPASS bup random 128k >$D/b
+mkdir $D/d $D/d/e
+WVPASS bup random 512 >$D/f
+WVPASS bup index -ux $D
+bup save --graft $TOP/$D=/grafted -n graft-point-absolute $D
+WVPASSEQ "$(bup ls graft-point-absolute/latest/grafted/)" "a
+b
+d/
+f"
+bup save --graft $D=grafted -n graft-point-relative $D
+WVPASSEQ "$(bup ls graft-point-relative/latest/$TOP/grafted/)" "a
+b
+d/
+f"
+
+WVSTART "indexfile"
+D=indexfile.tmp
+INDEXFILE=tmpindexfile.tmp
+rm -f $INDEXFILE
+rm -rf $D
+mkdir $D
+export BUP_DIR="$D/.bup"
+WVPASS bup init
+touch $D/a
+touch $D/b
+mkdir $D/c
+WVPASS bup index -ux $D
+bup save --strip -n bupdir $D
+WVPASSEQ "$(bup ls bupdir/latest/)" "a
+b
+c/"
+WVPASS bup index -f $INDEXFILE --exclude=$D/c -ux $D
+bup save --strip -n indexfile -f $INDEXFILE $D
+WVPASSEQ "$(bup ls indexfile/latest/)" "a
+b"
+
+
+WVSTART "import-rsnapshot"
+D=rsnapshot.tmp
+export BUP_DIR="$TOP/$D/.bup"
+rm -rf $D
+mkdir $D
+WVPASS bup init
+mkdir -p $D/hourly.0/buptest/a
+touch $D/hourly.0/buptest/a/b
+mkdir -p $D/hourly.0/buptest/c/d
+touch $D/hourly.0/buptest/c/d/e
+WVPASS true
+WVPASS bup import-rsnapshot $D/
+WVPASSEQ "$(bup ls buptest/latest/)" "a/
+c/"
print
print traceback.format_exc()
tb = sys.exc_info()[2]
- wvtest._result(e, traceback.extract_tb(tb)[1],
- 'EXCEPTION')
+ wvtest._result(e, traceback.extract_tb(tb)[1], 'EXCEPTION')
# main code
for modname in sys.argv[1:]:
oldwd = os.getcwd()
oldpath = sys.path
try:
- modpath = os.path.abspath(modname).split('/')[:-1]
- os.chdir('/'.join(modpath))
- sys.path += ['/'.join(modpath),
- '/'.join(modpath[:-1])]
- mod = __import__(modname.replace('/', '.'), None, None, [])
+ path, mod = os.path.split(os.path.abspath(modname))
+ os.chdir(path)
+ sys.path += [path, os.path.split(path)[0]]
+ mod = __import__(modname.replace(os.path.sep, '.'), None, None, [])
for t in wvtest._registered:
_runtest(modname, t.func_name, t)
print
projects / bup.git / commitdiff