David Roda [Tue, 14 Sep 2010 01:09:16 +0000 (21:09 -0400)]
Add simple styling to bup web.
This adds a wrapper set to 960px to the bup web layout. It also
sets widths on the table and columns.
I added a wrapper div to list-directory.html but I think that must
have snuck in with a previous commit. I am not sure how to fix that
so I will leave it for now. Sorry!
David Roda [Wed, 22 Sep 2010 01:50:27 +0000 (18:50 -0700)]
If we are showing hidden files, continue to do so.
This appends ?hidden=1 to all url's outputted to the template
if we are currently showing hidden files. I added a variable
url_append which is appended to the urls outside of the escaping
David Roda [Tue, 14 Sep 2010 01:09:13 +0000 (21:09 -0400)]
Enable static resources. Move css to external file.
Add to the settings variable in web-cmd.py to set
/web/static to be servced as static resources. This is for
css, javascript, and images.
Move the current styles from the head to static/css/styles.css.
Remove a few unnecessary styles and change the tab stop
to 4 spaces to match the rest of the code.
David Roda [Wed, 8 Sep 2010 12:58:09 +0000 (08:58 -0400)]
--remote parameter requires a colon
This patch checks for the presence of a colon if the --remote option
is used in bup save, bup split, bup join, and bup init. Even though
specifying *only* a pathname without a hostname: is perfectly valid,
it's confusing to allow users to do so, because if they specify
"-r hostname" it will be treated as a path and thus give them a
confusing error message. Requiring a colon will avoid this.
It adds a few test cases to demonstrate that the code
works properly.
It also wraps the remote connection in a try except to prevent
a traceback if there is an error (so far I have only seen this
happen with an invalid bup dir parameter)
And I added the netbeans project folder to gitignore
cmd/restore: embarrassingly slow implementation of 'bup restore'
Well, that was easy, since vfs.py already existed and is doing most of the
hard work. Only 103 lines including all the log message handling and
whatnot.
Only one catch: the restoring code is definitely not optimized. Among other
things (like the probably-excessive-for-our-restoring-needs layering in
vfs.py), we're still calling into 'git cat-file --stdin' to retrieve our
objects. This involves lots and lots of context switches, plus it can't use
midx files for its lookups. The result is that restoring takes much more
CPU time and memory than it really should. But oh well, we have to start
somewhere.
cmd/save: always print a progress() message after a log() message.
An earlier commit (634df2f8b26a1439f22dc9f6a23d55a006bf0429) made 'bup save'
update the progress line much less frequently. Unfortunately, if you used
-v or -vv, this would mean that there was *no* progress bar for a short time
after every log() message (directory or filename). That made the progress
bar flicker annoyingly.
To fix it, make sure we reset the progress bar timer after every filename we
print with log(). It's subtle, but it makes a very visible difference.
options.py: get the real tty width for word wrapping purposes.
Previously we just assumed it was 70 chars, which was safe enough, but not
as elegant as actually reading the real value and adjusting the word wrap of
the usage string accordingly.
options.py: remove extra newlines in usage string.
If the first line after the "--" was a comment (started with whitespace),
then we'd end up printing a double newline instead of a single one after the
synopsis string.
It would also look weird if we had a multi-line comment; the lines would be
separated by blank lines.
options.py: handle optspecs that include inline square brackets.
We recently made it so if the last thing on an options line was [defval],
then the value in brackets became the default for that option. However, we
inadvertently matched *any* bracketed value on that line, not just the one
at the end of the line, which basically prevents us from using square
brackets anywhere on the line. That's no fun.
options.py: better support for explicit no-* options.
If a declared option name starts with no-xxx, then the 'xxx' option starts
off set to True by default, so that no-xxx is False by default, so that
passing --no-xxx as an option will have the desired effect of setting
--no-xxx=True (and thus --xxx=False).
Previously, trying to list a --no-xxx option in the argument list would
trigger an assertion failure.
client.py,git.py: run 'bup midx -a' automatically sometimes.
Now that 'bup midx -a' is smarter, we should run it automatically after
creating a new index file. This should remove the need for running it by
hand.
Thus, we also remove 'bup midx' from the lists of commonly-used subcommands.
(While we're here, let's take out 'split' and 'join' too; you should be
using 'index' and 'save' most of the time.)
They were generated by catting bunches of bup source code together, which,
as it turns out, makes 'git grep' super annoying. Let's rot13 them so
grepping doesn't do anything interesting but the other characteristics are
the same.
cmd/midx: --auto mode can combine existing midx files now.
Previously, --auto would *only* create a midx from not-already-midxed .idx
files. This wasn't optimal since you'd eventually end up with a tonne of
.midx files, which is just as bad as a tonne of .idx files.
Now we'll try to maintain a maximum number of midx files using a
highwater/lowwater mark. That means the number of active midx files should
now stay between 2 and 5, and you can run 'bup midx -a' as often as you
want.
'bup midx -f' will still make sure everything is in a single .midx file,
which is an efficient thing to run every now and then.
'bup midx -af' is the same, but uses existing midx files rather than forcing
bup to start from only .idx files. Theoretically this should always be
faster than, and never be worse than, 'bup midx -f'.
Bonus: 'bup midx -a' now works when there's a limited number of file
descriptors. The previous fix only worked properly with 'bup midx -f'.
(This was rarely a problem since 'bup midx -a' would only ever touch the
last few .idx files, so it didn't need many file descriptors.)
cmd/midx: use getrlimit() to find the max open files.
It turns out the default file limit on MacOS is 256, which is less than our
default of 500. I guess this means trouble after all, so let's auto-detect
it.
* maint:
index.py: handle uid/gid == -1 on cygwin
cmd/memtest: use getrusage() instead of /proc/self/stat.
cmd/index: catch exception for paths that don't exist.
Don't use $(wildcard) during 'make install'.
Don't forget to install _helpers.dll on cygwin.
On cygwin, the uid or gid might be -1 for some reason. struct.pack()
complains about a DeprecationWarning when packing a negative number into an
unsigned int, so fix it up first.
cmd/memtest: use getrusage() instead of /proc/self/stat.
Only Linux has /proc/self/stat, so 'bup memtest' didn't work on anything
except Linux. Unfortunately, getrusage() on *Linux* doesn't have a valid
RSS field (sigh), so we have to use /proc/self/stat as a fallback if it's
zero.
Now memtest works on MacOS as well, which means 'make test' passes again.
(It stopped passing because 'bup memtest' recently got added to one of the
tests.)
All our man pages end up in section 1 of man anyway, and it looks like that
will probably never change. So let's make our filenames simpler and easier
to understand.
Even if we do end up adding a page in (say) section 5 someday, it's no big
deal; we can just add an exception to the Makefile for it or something.
It seems the $(wildcard) is evaluated once at make's startup, so any changes
made *during* build don't get noticed.
That means 'make install' would fail if you ran it without first running
'make all', because $(wildcard cmd/bup-*) wouldn't match anything at startup
time; the files we were copying only got created during the build.
We were installing *.so, but not *$(SOEXT) like we should have. Now we do,
which should fix some cygwin install problems reported by David Roda.
Also, when installing *.so and *.dll files, make them 0755 instead of 0644,
also to prevent permissions problems on cygwin, also reported by David Roda.
* guesser:
_helpers.extract_bits(): rewrite git.extract_bits() in C.
_helpers.firstword(): a new function to extract the first 32 bits.
git.py: when seeking inside a midx, use statistical guessing.
git.py: recover more elegantly if a MIDX file has the wrong version.
Previously we'd throw an assertion for any too-new-format MIDX file, which
isn't so good. Let's recover more politely (and just ignore the file in
question) if that happens.
Noticed by Zoran Zaric who was testing my midx3 branch.
Zoran reported that 'bup midx -f' on his system tried to open 3000 files at
a time and wouldn't work. That's no good, so let's limit the maximum files
to open; the default is 500 for now, since that ought to be usable for
normal people. Arguably we could use getrlimit() or something to find out
the actual maximum, or just keep opening stuff until we get an error, but
maybe there's no point.
Unfortunately this patch isn't really perfect, because it limits the
usefulness of midx files. If you could merge midx files into other midx
files, then you could at least group them all together after multiple runs,
but that's not currently supported.
Avery Pennarun [Fri, 27 Aug 2010 03:19:49 +0000 (20:19 -0700)]
_helpers.extract_bits(): rewrite git.extract_bits() in C.
That makes our memtest run just slightly faster: 2.8 seconds instead of 3.0
seconds, which catches us back up with the pre-interpolation-search code.
Thus we should now be able to release this patch without feeling embarrassed
:)
Avery Pennarun [Fri, 27 Aug 2010 02:31:24 +0000 (19:31 -0700)]
git.py: when seeking inside a midx, use statistical guessing.
Instead of using a pure binary search (where we seek to the middle of the
area and do a greater/lesser comparison) we now use an "interpolation
search" (http://en.wikipedia.org/wiki/Interpolation_search), which means we
seek to where we statistically *expect* the desired value to be.
In my test data, this reduces the number of typical search steps in my test
midx from 8.7 steps/object to 4.8 steps/object.
This reduces memory churn when using a midx, since sometimes a given search
region spans two pages, and this technique allows us to more quickly
eliminate one of the two pages sometimes, allowing us to dirty one fewer
page.
Unfortunately the implementation requires some futzing, so this actually
makes memtest run about 35% *slower*. Will try to fix that next.
The original link to this algorithm came from this article:
http://sna-projects.com/blog/2010/06/beating-binary-search/
Avery Pennarun [Fri, 27 Aug 2010 02:16:34 +0000 (19:16 -0700)]
cmd/memtest: add a --existing option to test with existing objects.
This is useful for testing behaviour when we're looking for objects
that *do* exist. Of course, it just goes through the objects in order, so
it's not actually that realistic.
Avery Pennarun [Thu, 26 Aug 2010 04:06:46 +0000 (21:06 -0700)]
cmd/midx: fix SHA_PER_PAGE calculation.
For some reason we were dividing by 200 instead of by 20, which was way off.
Switch to 20 instead. Suspiciously, this makes memory usage slightly worse
in my current (smallish) set of test data, so we might need to revert it
later...? But if we're going to have an adjustment, we should at least make
it clear what for, rather than hiding it in something that looks
suspiciously like a typo.
Avery Pennarun [Thu, 26 Aug 2010 03:40:34 +0000 (20:40 -0700)]
cmd/margin: add a new --predict option.
When --predict is given, it tries to guess the offset in the indexfile of
each hash, based on assumption that the hashes are distributed evenly
throughout the file. Then it prints the maximum amount by which this guess
deviates from reality.
I was hoping the results would show that the maximum deviation in a typical
midx was less than a page's worth of hashes; that would mean the toplevel
lookup table could be redundant, which means fewer pages hit in the
common case. No such luck, unfortunately; with 1.6 million objects, my
maximum deviation was 913 hashes (about 18 kbytes, or 5 pages).
By comparison, midx files should hit about 2 pages in the common case (1
lookup table + 1 data page). Or 3 pages if we're unlucky and the search
spans two data pages.
Avery Pennarun [Mon, 23 Aug 2010 03:27:03 +0000 (20:27 -0700)]
Rename _faster.so to _helpers.so.
Okay, _faster.so wasn't a good choice of names. Partly because not
everything in there is just to make stuff faster, and partly because some
*proposed* changes to it don't just make stuff faster. So let's rename it
one more time. Hopefully the last time for a while!
Gabriel Filion [Mon, 16 Aug 2010 01:29:29 +0000 (21:29 -0400)]
import cleanup
Remove unused imported modules.
I started using the pyflakes.vim plugin and it automagically shows a
bunch of problems/uncleanliness in the code. It helped me pull this out
in 15mins.
This change shouldn't have any impact on performance or functionality
but it makes the code cleaner.
Avery Pennarun [Sun, 22 Aug 2010 06:44:49 +0000 (23:44 -0700)]
cmd/ftp: don't die if we can't import the ctypes module.
It's only needed on some rare broken versions of readline anyway. If we
can't find the module, chances are the system doesn't have that broken
version of readline.
Based on suggestions by Gabriel Filion and Aaron Ucko.
Gabriel Filion [Fri, 20 Aug 2010 06:24:57 +0000 (02:24 -0400)]
lib/bup/vfs: bring back Python 2.4 support
There is currently one test failure when running tests against Python
2.4: a try..except..finally block that's interpreted as a syntax error.
The commit introducing this incompatibility with 2.4 is f77a0829
This is a well known python 2.4 limitation and the workaround, although
ugly, is easy.
With this test passing, Python 2.4 support is back.
Gabriel Filion [Mon, 2 Aug 2010 06:20:06 +0000 (02:20 -0400)]
lib/bup/vfs: Add docstrings
Since the vfs module uses the function git._treeparse, it should not be
named as if it was a private function. Rename git._treeparse to
git.treeparse and document it (add a docstring to it).
Also, transform _ChunkReader, _FileReader and Node into new-style
classes.
Finally, remove trailing spaces from lib/bup/vfs.py .
If you ran 'bup save' in an ssh sessio, you could end up sending huge
amounts of data back over ssh *just* to update the progress meter after
every single block! Oops. Limit the updates to only about 5 per second,
which is much better.
Rename _hashsplit.so to _faster.so, and move bupsplit into its own source file.
A lot of stuff in _hashsplit.c wasn't actually about hashsplitting; it was
just a catch-all for all our C accelerator functions. Now the module name
reflects that.
Also move the bupsplit functions into their own non-python-dependent C
source file so they can be used as part of other projects.
_hashsplit.c: replace the stupidsum algorithm with rsync's adler32-based one.
I've been meaning to do this for a while, but a particular test dataset that
really caused problems with stupidsum() (ie. it split things into way more
chunks than it should have) finally screwed me over. Let's change over to a
"real" checksum algorithm.
Non-annoying datasets shouldn't be noticeably affected, but bad ones (such
as my test case from EQL Data) can be 10x more sensible. Typical backup
sets now have about 20% fewer chunks, although this has little affect on the
overall repository size.
WARNING: After this patch, all your chunk boundaries will be different from
before! That means your incremental backups won't be terribly incremental
and your backup repositories will jump in size. This should only happen
once.
_hashsplit.c: switch rollsum_roll() to a macro instead of an inline function.
gcc 4.3's optimizer manages to fail at optimizing the inline, but works okay
with the macro.
Mysteriously, if find_ofs() is *not* static (and therefore presumably
*harder* to optimize), the optimizer works either way. But removing the
static is just wrong, so use the macro instead.
The difference in speed is about 53 megs/sec vs 80 megs/sec on my machine
for this command:
bup random 100M 2>/dev/null | bup split -N --bench
Gabriel Filion [Tue, 27 Jul 2010 03:52:34 +0000 (23:52 -0400)]
cmd/ftp: Hide .dotfiles by default (-a shows them)
Normally in FTP sites, files beginning with a dot are hidden from a list
(ls) command by default. Also, using the argument '-a' makes the list
show hidden files.
The current 'bup ftp' implementation does not behave so. Make it hide
hidden files by default, as expected, and show hidden files when '-a' or
'--all' is specified to the 'ls' command.
All unknown switches will make bup ftp show the ls command usage.
Users can also give 'ls --help' to obtain the usage string.
Gabriel Filion [Tue, 27 Jul 2010 03:52:33 +0000 (23:52 -0400)]
lib/options: Add an onabort argument to Options()
Some times, we may want to parse a list of arguments and not have the
call to Options.parse() exit the program when it finds an unknown
argument.
Add an argument to the class' __init__ method that can be either a
function or a class (must be an exception class). If calling the
function or class constructor returns an object, this object will be
raised on abort.
Also add a convenience exception class named Fatal that can be
passed to Options() to exclusively catch situations in which
Options.parse() would have caused the program to exit.
Finally, set the default value to the onabort argument to call
sys.exit(97) as was previously the case.
Gabriel Filion [Tue, 27 Jul 2010 07:24:23 +0000 (03:24 -0400)]
cmd/ftp: if completion fails due to FileNotFound, just eat it.
Just as bash would do, if you're trying to complete a filename that doesn't
exist, just don't offer any completions. In this case, it only happens if
you try to complete through a broken symlink.
Now that we've fixed this case, enable the printing of exception tracebacks
in case of *other* kinds of completion errors, since we don't expect there
to be any.
[Committed by apenwarr based on an unofficial patch from Gabriel]
vfs: resolve absolute symlinks inside their particular backup set.
Let's say you back up a file "/etc/motd" that's a symlink to
"/var/run/motd". The file inside the backup repo is actually
/whatever/latest/etc/motd, so the symlink should *actually* point to
/whatever/latest/var/run/motd. Let's resolve it that way automatically in
Symlink.dereference().
vfs: try_lresolve() was a bad idea. Create try_resolve() instead.
Also add some comments to describe the actual differences between resolve()
and lresolve(), and clean things up a bit so that they actually work as
they're supposed to.
Basically, all of lresolve(), resolve(), and try_resolve() depend on
*intermediate* paths being resolvable; all of them will throw an exception
if not. They only differ in the very last node in the path, when that node
is a symlink:
resolve() will dereference it or throw an exception if it can't;
try_resolve() will try to dereference it, but return self if it can't;
lresolve() will not dereference it at all, like lstat() doesn't.
With that in mind, we can fix up cmd/ftp and cmd/web to use the right calls,
thus fixing an unexpected error in ftp's tab completion reported by Gabriel
Filion, which would happen if you tried to tab complete inside a directory
that contained a broken symlink. We only care what the symlink points to so
we can decide whether or not to append '/' to the tab completion, so we want
it to fail silently if it's going to fail.
Gabriel Filion [Sun, 25 Jul 2010 17:34:13 +0000 (13:34 -0400)]
fix helpers.columnate bug when list is empty
When the list given to the columnate function is empty, the function
raises an exception when determining the max(len of all elements), since
the list given to max is empty.
One indirect example of when this bug is apparent is in the 'bup ftp'
command when listing an empty directory:
bup> ls backupname/latest/etc/keys
error: max() arg is an empty sequence
Add a special condition at the beginning of the columnate function that
returns an empty string if the list of elements is empty.
Joe Beda [Fri, 23 Jul 2010 07:10:36 +0000 (00:10 -0700)]
Convert 'bup web' directory listing to use tornado templates.
This includes creating a new idea of a "resource path" that currently sits
under the lib dir. Getting resources is supported with a new helper
(resource_path).
I just took the tornado/tornado directory, along with the README.
I'm using tornado's git commit 7a30f9f6eac9aa0cf295b078695156776fd050ce,
since recent versions of Tornado have support for specifying which
address you want to listen to.
Signed-off-by: Peter McCurdy <petermccurdy@alumni.uwaterloo.ca>
git.py: use close_fds=True when starting git cat-file.
Otherwise git could inherit some other file descriptors we're using. This
is particularly relevant in cmd/web, and particularly when applying
pmccurdy's patches to use Tornado.
Because of changes to wvtest.py's chdir() handling, had to make some slight
changes to filenames used by the bup tests themselves - all changes for the
better.
options.py: differentiate unset and set-to-negative options.
Unset options will still be None, but options explicitly set to a negative
will now be 0. This doesn't change semantics for anything currently in bup,
but it could be useful later when applying defaults.
While we're here, clean up the option parsing code to make it
very slightly more efficient.
Apparently on some systems (Mandriva and Slackware at least), importing
the readline library can print some escape sequences to stdout, which screws
things up with the unit tests that run 'bup ftp "cat filename"' and expect
it to be the right data.
Thanks to Eduardo Kienetz for noticing and helping to track down the problem
since I couldn't reproduce it.
vfs: File.open() needs to do a seek(0) on the cached FileReader.
Otherwise if you open a file, read through it, and close it, then do it
again, you'll get zero bytes the second time.
To make this efficient, change seek() to not discard its _chunkiter every
single time; instead, keep the _chunkiter around until trying to read() from
a location that *isn't* the current offset. Now seeking around in the file
is cheap.
projects / bup.git / log