]> arthur.barton.de Git - bup.git/log
bup.git
2 years agobloom.do_bloom: always close filter; fix None vs 0 tests
Rob Browning [Mon, 22 Nov 2021 20:03:26 +0000 (14:03 -0600)]
bloom.do_bloom: always close filter; fix None vs 0 tests

Running a "split -n" for a larger file caused the assertion in
ShaBloom.__del__ to fail, i.e. the filter hadn't been explicitly
closed.  Fix the logic in do_bloom to be careful about 0 vs None since
ShaBloom has a __len__ method, and add one missing explicit close().

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoCheck that all context managed objects are properly closed
Rob Browning [Sun, 24 Oct 2021 17:24:16 +0000 (12:24 -0500)]
Check that all context managed objects are properly closed

Ensure all of our context managed objects have a __del__ that asserts
that the instance has been properly closed so that we'll be more
likely to notice related oversights.

This will only work in cases where __del__ is called before shutdown,
but that should normally be the case for cpython.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoFully (and explicitly) close PackIdxLists
Rob Browning [Sat, 23 Oct 2021 19:06:01 +0000 (14:06 -0500)]
Fully (and explicitly) close PackIdxLists

And stop checking _mpi_count in __del__ since there are no guarantees
about if/when it will run (and so could run after another has been
opened).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoPackMidx.__del__: replace with context management
Rob Browning [Fri, 1 Oct 2021 02:02:59 +0000 (21:02 -0500)]
PackMidx.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agocompat: add ExitStack
Rob Browning [Thu, 28 Oct 2021 17:11:05 +0000 (12:11 -0500)]
compat: add ExitStack

For now, just add enough py2 support (since it wasn't too hard) to
handle some forthcoming cases.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoCatPipe.__del__: replace with context management
Rob Browning [Tue, 28 Sep 2021 05:32:37 +0000 (00:32 -0500)]
CatPipe.__del__: replace with context management

Drop AbortableIter since CatFile is the only user, in favor of just
closing the catpipe (what the AbortableIter was doing) in the get()
exception handler.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoindex.Writer.__del__: replace with context management
Rob Browning [Tue, 28 Sep 2021 01:28:53 +0000 (20:28 -0500)]
index.Writer.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoReplace LocalRepo/RemoteRepo __del__ with context management
Rob Browning [Tue, 28 Sep 2021 00:58:06 +0000 (19:58 -0500)]
Replace LocalRepo/RemoteRepo __del__ with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosave: detect missing metastore via ENOENT not EACCES
Rob Browning [Fri, 8 Oct 2021 17:39:16 +0000 (12:39 -0500)]
save: detect missing metastore via ENOENT not EACCES

While we might also eventually want to handle EACCES, ENOENT should be
what we want for the current handler, i.e. "does it exist at all".

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoindex.Reader.__del__: replace with context management
Rob Browning [Sun, 26 Sep 2021 21:20:06 +0000 (16:20 -0500)]
index.Reader.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoMetaStoreWriter.__del__: replace with context management
Rob Browning [Sun, 26 Sep 2021 20:55:12 +0000 (15:55 -0500)]
MetaStoreWriter.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoMetaStoreReader.__del__: replace with context management
Rob Browning [Sun, 26 Sep 2021 20:43:34 +0000 (15:43 -0500)]
MetaStoreReader.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoHLinkDB.__del__: replace with context management
Rob Browning [Sun, 26 Sep 2021 19:53:18 +0000 (14:53 -0500)]
HLinkDB.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agodrecurse: replace OsFile (and __del__) with finalized fd
Rob Browning [Sun, 26 Sep 2021 19:40:02 +0000 (14:40 -0500)]
drecurse: replace OsFile (and __del__) with finalized fd

Add composable helpers.finalize(x, finalizer) context management
adapter and use it to replace OsFile.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoShaBloom.__del__: replace with context management
Rob Browning [Sun, 17 Oct 2021 17:17:36 +0000 (12:17 -0500)]
ShaBloom.__del__: replace with context management

These changes also just use finally in some cases, instead of the more
complex py2 compatible BaseException/with_pending_raise() catch becase
I'm leaning in favor of just dropping python 2 support soon.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agocompat.mmap: add py2 context management and use everywhere
Rob Browning [Sun, 3 Oct 2021 18:25:08 +0000 (13:25 -0500)]
compat.mmap: add py2 context management and use everywhere

It looks like py2 doesn't support mmap __enter__ and __exit__, so add
them, and switch all our mmap instances to be compat.mmap so that
we'll be able to context manage them.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agogit/packwriter: open(..) prohibited in __del__
Bas Stottelaar [Fri, 7 Aug 2020 11:04:19 +0000 (13:04 +0200)]
git/packwriter: open(..) prohibited in __del__

When an exception occurs, __del__ is invoked by the interpreter, to
perform cleanup. It seems that since Python 3.4, the behaviour has
changed, and also prohibits invocations of open(..) (source:
https://stackoverflow.com/a/29737870). Instead, contextmanager API
should be used (source: https://stackoverflow.com/a/26544629), which
seems to be in place already.

This should fix exception messages such as 'NameError: name 'open'
is not defined'.

Signed-off-by: Bas Stottelaar <basstottelaar@gmail.com>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agogit.packwriter: explicitly manage lifetimes
Rob Browning [Sat, 23 Oct 2021 17:59:06 +0000 (12:59 -0500)]
git.packwriter: explicitly manage lifetimes

Explicitly close all packwriters, instead of relying on __del__.  Use
"with ..." context management in most cases..  This is in preparation
for removing __del__.

Rework _end to handle *all* cleanup and to ensure that we always
release the relevant resources, whether closing normally, or aborting.
This means that we only have two places to consider with respect to
resource management: __init__ and _end.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoAdd any pending exception to context in all __exit__ methods
Rob Browning [Sun, 3 Oct 2021 17:47:17 +0000 (12:47 -0500)]
Add any pending exception to context in all __exit__ methods

Use pending_raise() in all __exit__ methods to make sure that any
pending exception is included in the context if the body throws.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoRemove Client __del__ in favor of context management
Rob Browning [Sun, 26 Sep 2021 00:55:13 +0000 (19:55 -0500)]
Remove Client __del__ in favor of context management

And drop EPIPE suppression for now.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agocompat.pending_raise: allow/ignore None ex; make rethrow optional
Rob Browning [Sun, 3 Oct 2021 17:44:25 +0000 (12:44 -0500)]
compat.pending_raise: allow/ignore None ex; make rethrow optional

This supports compact __exit__ handlers, e.g.

  def __exit__(self, type, value, ...):
      with pending_raise(value, rethrow=False):
          clean_up()

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosave: separate opt handling and work; isolate resource lifetimes
Rob Browning [Sun, 26 Sep 2021 00:17:13 +0000 (19:17 -0500)]
save: separate opt handling and work; isolate resource lifetimes

Rearrange save to disentangle the options parsing, saving, and
resource (e.g. packfile) lifetime management before we replace __del__
with context management.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosplit: separate opt handling and work; isolate packfile lifetime
Rob Browning [Sat, 25 Sep 2021 21:46:16 +0000 (16:46 -0500)]
split: separate opt handling and work; isolate packfile lifetime

Rearrange split to disentangle the options parsing, splitting, and
resource (packfile) lifetime management before we replace __del__ with
context management.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosplit: handle --noop/copy via do nothing packwriter
Rob Browning [Sat, 25 Sep 2021 19:02:43 +0000 (14:02 -0500)]
split: handle --noop/copy via do nothing packwriter

This allows us to remove the special-casing for those code paths, and
rearrange the structure in preparation for handling the packwriter via
a context manager, in preparation for removing __del__.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoconfigure: support optional cc arguments
Rob Browning [Sat, 30 Oct 2021 19:07:55 +0000 (14:07 -0500)]
configure: support optional cc arguments

I noticed in some unrelated reports that older compilers were
rejecting -Wno-unsed-command-line-argument, so add
bup-add-cflag-if-supported to configure and use it for that argument.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobitmatch: avoid signed/unsigned comparison
Rob Browning [Sat, 30 Oct 2021 18:23:45 +0000 (13:23 -0500)]
bitmatch: avoid signed/unsigned comparison

Thanks to Brian Minton for reporting the problem:

  I'm using gcc version 8.3.0 (Debian 8.3.0-6). The system is x86_64
  with 8 GB ram. When I try to build the source, with make, I get the
  following error:

    ...

    In file included from lib/bup/_helpers.c:69:
    lib/bup/_helpers.c: In function 'bitmatch':
    .../src/bup/src/bup/intprops.h:626:22: error: comparison of
    integer expressions of different signedness: 'long long unsigned int' and
    'Py_ssize_t' {aka 'long int'} [-Werror=sign-compare]
    : (tmax) / (b) < (a)))
    ^
    .../src/bup/src/bup/intprops.h:427:10: note: in expansion of
    macro '_GL_INT_MULTIPLY_RANGE_OVERFLOW'
    && _GL_INT_MULTIPLY_RANGE_OVERFLOW (a, b, 0, (__typeof__ (*(r))) -1)) \
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    .../src/bup/src/bup/intprops.h:655:36: note: in expansion of
    macro 'INT_MULTIPLY_WRAPV'
    #define INT_MULTIPLY_OK(a, b, r) ! INT_MULTIPLY_WRAPV (a, b, r)
    ^~~~~~~~~~~~~~~~~~
    lib/bup/_helpers.c:624:10: note: in expansion of macro 'INT_MULTIPLY_OK'
    if (!INT_MULTIPLY_OK(byte, 8, &result)
    ^~~~~~~~~~~~~~~

And to Mark J Hewitt for reporting the issue on a 32-bit Raspberry Pi
system.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobup_gethostname: always null terminate
Rob Browning [Sun, 10 Oct 2021 17:08:32 +0000 (12:08 -0500)]
bup_gethostname: always null terminate

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoweb list-directory: replace != with is not None
Rob Browning [Sat, 9 Oct 2021 15:25:57 +0000 (10:25 -0500)]
web list-directory: replace != with is not None

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoweb: do not show size for dirs and symlinks
Johannes Berg [Sun, 17 May 2020 19:55:28 +0000 (21:55 +0200)]
web: do not show size for dirs and symlinks

It's strange to see a lot of zeros for these items, just
don't show them by making the display_size None.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobitmatch: check for overflow via intprops
Rob Browning [Fri, 8 Oct 2021 15:47:28 +0000 (10:47 -0500)]
bitmatch: check for overflow via intprops

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoSupport signed commits (i.e. the gpgsig header)
Rob Browning [Tue, 5 Oct 2021 01:54:50 +0000 (20:54 -0500)]
Support signed commits (i.e. the gpgsig header)

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoCatPipe.get: report unexpected cat-file exit status
Rob Browning [Sun, 3 Oct 2021 19:57:55 +0000 (14:57 -0500)]
CatPipe.get: report unexpected cat-file exit status

Thanks to Mark Hewitt for reporting a problem that suggested we should
handle this case more carefully.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agocstr_from_bytes: fix null termination error
Rob Browning [Sun, 3 Oct 2021 19:35:15 +0000 (14:35 -0500)]
cstr_from_bytes: fix null termination error

Thanks to Johannes Berg for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoDrop builtin_mul_overflow in favor of INT_MULTIPLY_OK
Rob Browning [Sun, 3 Oct 2021 19:21:27 +0000 (14:21 -0500)]
Drop builtin_mul_overflow in favor of INT_MULTIPLY_OK

Now that we have intprops.h drop built_mul_overflow (since intpropos.h
should use it when available).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agomake: add check-py2 and check-py3 targets
Rob Browning [Sun, 3 Oct 2021 18:16:27 +0000 (13:16 -0500)]
make: add check-py2 and check-py3 targets

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoftp: use sys.exc_info(); import traceback unconditionally
Rob Browning [Sun, 3 Oct 2021 19:11:43 +0000 (14:11 -0500)]
ftp: use sys.exc_info(); import traceback unconditionally

Since it appears that works across all the versions we support.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoAllow PANDOC path to contain spaces
Rob Browning [Fri, 17 Sep 2021 18:16:58 +0000 (13:16 -0500)]
Allow PANDOC path to contain spaces

Thanks to Arthur Ward for reporting the problem and proposing an
alternate solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest-index-check-device: unmount "lazily" to avoid potential crash
Rob Browning [Fri, 17 Sep 2021 17:56:38 +0000 (12:56 -0500)]
test-index-check-device: unmount "lazily" to avoid potential crash

Apparently the umount was failing on some systems, and since we're
already using loop, mke2fs, etc. which aren't really portable, and we
use "umount -l" in cleanup-mounts-under, just use it here too for now.

Thanks to Abdel Said for reporting the problem:

  ! t/test-index-check-device.sh:38 mke2fs -F -j -m 0 testfs.img ok
  ! t/test-index-check-device.sh:39 mount -o loop testfs.img /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU ok
  ! t/test-index-check-device.sh:41 chown root:root /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU ok
  ! t/test-index-check-device.sh:42 chmod 0700 /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU ok
  ! t/test-index-check-device.sh:45 date ok
  umount: /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU: device is busy.
  (In some cases useful info about processes that use
  the device is found by lsof(8) or fuser(1))
  ! t/test-index-check-device.sh:46 umount /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU FAILED

Additional discussion: https://groups.google.com/d/msg/bup-list/GhVonW0XGVE/8kIkJEOwCQAJ

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: enable inconsistent-return-statements
Johannes Berg [Fri, 27 Aug 2021 23:07:16 +0000 (01:07 +0200)]
pylint: enable inconsistent-return-statements

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: return early for dry run in import_duplicity exo]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: enable catching-non-exception
Johannes Berg [Fri, 27 Aug 2021 22:54:50 +0000 (00:54 +0200)]
pylint: enable catching-non-exception

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: add undefined-variable
Johannes Berg [Fri, 27 Aug 2021 22:40:16 +0000 (00:40 +0200)]
pylint: add undefined-variable

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: add missing %s in validate_vfs_path changes]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: enable unidiomatic-typecheck
Johannes Berg [Fri, 27 Aug 2021 22:33:13 +0000 (00:33 +0200)]
pylint: enable unidiomatic-typecheck

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: change isinstance str checks to compat.str_type]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: enable consider-using-in
Johannes Berg [Fri, 27 Aug 2021 22:30:41 +0000 (00:30 +0200)]
pylint: enable consider-using-in

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: enable useless-return
Johannes Berg [Fri, 27 Aug 2021 22:27:53 +0000 (00:27 +0200)]
pylint: enable useless-return

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: enable return-in-init
Johannes Berg [Fri, 27 Aug 2021 22:25:32 +0000 (00:25 +0200)]
pylint: enable return-in-init

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agopylint: check for trailing whitespace
Johannes Berg [Fri, 27 Aug 2021 22:08:38 +0000 (00:08 +0200)]
pylint: check for trailing whitespace

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agometadata: accept EOPNOTSUPP for lchmod()
Johannes Berg [Sun, 12 Sep 2021 18:58:45 +0000 (20:58 +0200)]
metadata: accept EOPNOTSUPP for lchmod()

On some systems (e.g. mine, Fedora 33) python2 has
(started to have?) lchmod(), as glibc has it, but
that always only returns EOPNOTSUPP if you really
try to operate on a symlink. Allow for that and
handle it just like ENOSYS.

Note that the handling of ENOSYS is wrong, since
integers can't be thrown or caught - pylint found
this as well, but since I'm fixing EOPNOTSUPP here
I need to fix that at the same time.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
2 years agoREADME: fix and simplify cirrus badges
Rob Browning [Sun, 12 Sep 2021 17:28:11 +0000 (12:28 -0500)]
README: fix and simplify cirrus badges

Fix broken links (given the recent task name changes), and stop
reporting per-platform status since each one may have multiple tasks.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosystem-info: report freebsd cpu and mem info
Rob Browning [Sat, 11 Sep 2021 18:23:50 +0000 (13:23 -0500)]
system-info: report freebsd cpu and mem info

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agocirrus: run pylint for python 2 and python 3
Rob Browning [Sat, 11 Sep 2021 17:47:57 +0000 (12:47 -0500)]
cirrus: run pylint for python 2 and python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoconfigure: add --with-pylint=[yes|no|maybe] defaulting to maybe
Rob Browning [Sat, 11 Sep 2021 21:02:19 +0000 (16:02 -0500)]
configure: add --with-pylint=[yes|no|maybe] defaulting to maybe

When set to no, don't run pylint from ./pylint, just exit
successfully.  When set to maybe, use dev/have-pylint to figure out if
pylint is available, and if so, run it, otherwise exit successfully
after describing the situation.  When set to yes, always try to run
pylint.

This may be useful more generally, but in particular, it makes it
possible to run this:

  ./configure --with-pylint=maybe
  make check-both

in situations where pylint is available for say python 3, but not
python 2.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoGNUmakefile: run "bup features" during test
Rob Browning [Sat, 11 Sep 2021 16:52:30 +0000 (11:52 -0500)]
GNUmakefile: run "bup features" during test

Provide additional information about what's being tested, particularly
in CI.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest: add pylint and test imports
Johannes Berg [Fri, 27 Aug 2021 20:23:37 +0000 (22:23 +0200)]
test: add pylint and test imports

Add pylint and test for unnecessary imports as a first step, which
requires cleaning them all up.

[rlb@defaultvalue.org: add and use ./pylint]

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agocirrus: adjust for new python-config based builds
Rob Browning [Wed, 8 Sep 2021 05:50:15 +0000 (00:50 -0500)]
cirrus: adjust for new python-config based builds

Adjust to accommodate the switch from PYTHON to BUP_PYTHON_CONFIG
cf. 4d9dd65141326b0f3ffa1658a0535348940ed017

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoprep-for-freebsd-build: switch to py38
Rob Browning [Wed, 8 Sep 2021 05:47:49 +0000 (00:47 -0500)]
prep-for-freebsd-build: switch to py38

That's what's now in 12.2 (which we use in ci).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agocirrus: adjust test names (root vs non-root, etc.)
Rob Browning [Wed, 8 Sep 2021 05:43:48 +0000 (00:43 -0500)]
cirrus: adjust test names (root vs non-root, etc.)

The su tests run as non-root (bup), not root.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agodev/system-info: include df, mount, and git version
Rob Browning [Mon, 6 Sep 2021 16:50:37 +0000 (11:50 -0500)]
dev/system-info: include df, mount, and git version

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agobup: remove redundant allocation doubling
Rob Browning [Sat, 4 Sep 2021 19:08:02 +0000 (14:08 -0500)]
bup: remove redundant allocation doubling

Thanks to Johannes Berg for pointing out the issue.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest_git: be independent of user's git defaultBranch
Rob Browning [Sat, 4 Sep 2021 20:43:02 +0000 (15:43 -0500)]
test_git: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest-save-restore: be independent of user's git defaultBranch
Rob Browning [Sat, 4 Sep 2021 20:42:08 +0000 (15:42 -0500)]
test-save-restore: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest-misc: be independent of user's git defaultBranch
Rob Browning [Sat, 4 Sep 2021 20:40:57 +0000 (15:40 -0500)]
test-misc: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest_prune_older: be independent of user's git defaultBranch
Rob Browning [Sat, 4 Sep 2021 20:32:21 +0000 (15:32 -0500)]
test_prune_older: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agocmd/bup: add some missing safety checks
Rob Browning [Sun, 1 Aug 2021 19:45:26 +0000 (14:45 -0500)]
cmd/bup: add some missing safety checks

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoUse intprops for INTEGRAL_ASSIGNMENT_FITS INTEGER_TO_PY uadd
Rob Browning [Sat, 7 Aug 2021 17:41:55 +0000 (12:41 -0500)]
Use intprops for INTEGRAL_ASSIGNMENT_FITS INTEGER_TO_PY uadd

Rewrite INTEGRAL_ASSIGNMENT_FITS, INTEGER_TO_PY, and uadd using
intprops,  which also avoids needing all the custom (and historically
fragile) compiler option manipulations.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoMake intprops available as "bup/intprops.h"
Rob Browning [Sat, 7 Aug 2021 17:40:04 +0000 (12:40 -0500)]
Make intprops available as "bup/intprops.h"

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoAdd Gnulib intprops.h
Rob Browning [Sat, 7 Aug 2021 17:38:51 +0000 (12:38 -0500)]
Add Gnulib intprops.h

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoconfig.vars.in: add CC so a configure CC will be the default
Rob Browning [Sun, 1 Aug 2021 20:47:07 +0000 (15:47 -0500)]
config.vars.in: add CC so a configure CC will be the default

Add CC to config.vars.in (included by GNUMakefile) so that the CC
detected by ./configure will actually become the default.  Previously
it would still reflect GNU make's default, even after something like
this:

  CC=/some/where/gcc ./configure

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agogit: teach git_config_get() to read from a file
Johannes Berg [Wed, 18 Dec 2019 21:03:58 +0000 (22:03 +0100)]
git: teach git_config_get() to read from a file

We want to use git_config_get() to have a bup config file
in the future, so teach git_config_get() to read from a
file and add some tests for it.

Use this also to test the opttype conversions from the
previous patch.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agogit: use git's int parsing with pack.packSizeLimit config
Johannes Berg [Thu, 9 Jan 2020 12:27:35 +0000 (13:27 +0100)]
git: use git's int parsing with pack.packSizeLimit config

Our parse_num() understands a little more than git, in particular
 * T for terabytes
 * b suffix when you specify Kb, Mb, Gb or Tb.

Neither of those are understood by git, it only understands the
K, M and G suffixes (case-insensitive). However, a git repository
that actually states 'pack.packSizeLimit' as something that git
doesn't understand is broken for every single git command, and as
we call git, the added flexibility of parse_num() cannot be used.

Thus, simplify the code and just use opttype='int' for this.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agogit: allow config retrieval as bool or int
Johannes Berg [Wed, 8 Jan 2020 21:48:25 +0000 (22:48 +0100)]
git: allow config retrieval as bool or int

Allow retrieving configuration values as bool or int,
letting 'git config' normalize all the values instead
of trying to do it ourselves in bup.

Also use git config --null to avoid having to worry about
any whitespace issues, although .strip() on the output
would probably work just as well. Previously, the result
would include a terminating newline written out by the
git config command, which is not desirable.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agohashsplit: increase READ_SIZE to 8 MB
Johannes Berg [Tue, 28 Apr 2020 21:11:16 +0000 (23:11 +0200)]
hashsplit: increase READ_SIZE to 8 MB

It's not really possible to run bup with tiny amounts of memory,
so reading 1 MB or 8 MB doesn't make a significant difference
here.

However, python actually implements read() as mmap() (at least
on my Linux system), with the requested read size given to mmap
as the size. The kernel then doesn't appear to do any readahead
(which makes sense), which kills performance.

Even if this wasn't the case though, read() of 8MB isn't much
of an issue, so increase the size.

Note that 8 MB is also the size for the fadvise() code.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosave: refactor the code using helper classes
Johannes Berg [Tue, 28 Jan 2020 20:24:09 +0000 (21:24 +0100)]
save: refactor the code using helper classes

Instead of keeping three lists, refactor the code to use
a helper class for a folder being collected, and an item
in that folder being collected.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobup: make demux errors clearer
Johannes Berg [Mon, 1 Feb 2021 23:01:47 +0000 (00:01 +0100)]
bup: make demux errors clearer

If we hit the

    assert(n <= MAX_PACKET)

in the demux code then quite likely we've been reading
something that shouldn't even be demuxed, but got some
error message from ssh instead of a muxed connection
record.

Clarify this case a bit by printing out the data that
we got (under the assumption it's an error message) and
raising Exception("Connection broken") instead of just
asserting.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoget_argv: declare loop variable outside for()
Rob Browning [Sat, 5 Jun 2021 17:18:02 +0000 (12:18 -0500)]
get_argv: declare loop variable outside for()

Don't require a newer compiler/standard.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agomain: remove vestigial debug statement
Rob Browning [Sat, 5 Jun 2021 17:17:49 +0000 (12:17 -0500)]
main: remove vestigial debug statement

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoSend *all* output before the BUPMUX header to stderr
Rob Browning [Sat, 29 May 2021 20:18:08 +0000 (15:18 -0500)]
Send *all* output before the BUPMUX header to stderr

This is just an incremental fix.  We may need to follow up with
additional improvements.

Thanks to Johannes Berg for diagnosing the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agols: read metadata only if needed
Johannes Berg [Wed, 5 Feb 2020 21:24:01 +0000 (22:24 +0100)]
ls: read metadata only if needed

If we just want to get a list of files, there's no point in
reading metadata, since it simply isn't needed. Pass want_meta
to the appropriate functions only when we need the metadata,
and similarly call vfs.augment_item_meta() only then.

Note that the previous vfs change really made this effective,
we'd otherwise lose the information.

Note also that unfortunately this is necessary even for the
--file-type command line option because otherwise we cannot
identify FIFOs; for everything else the git mode appears to
be sufficient.

Together, this reduces the number of blob reads from 62 to 8
for 'bup ls' on a trivial folder (the bup git's root folder).

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agovfs: improve cache behaviour
Johannes Berg [Sun, 15 Mar 2020 21:38:15 +0000 (22:38 +0100)]
vfs: improve cache behaviour

If _commit_item_from_oid() is called first with require_meta=False,
and then again with require_meta=True, the second and further calls
will not use the cache, as the cached entry is without metadata.
Improve this by overwriting the cache entry if it couldn't be used.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agovfs: read metadata only if needed
Johannes Berg [Wed, 5 Feb 2020 21:21:18 +0000 (22:21 +0100)]
vfs: read metadata only if needed

If we get into contents(), we have an indication of whether or
not metadata is needed (the want_meta argument), but it doesn't
get passed down to revlist_items() and further down, so that in
cache_commit() we eventually call _revlist_item_from_oid() with
metadata always. This is wasteful.

Fix this by passing the information all the way down.

To make the caching work properly in this case, store a special
entry in the revlist dict (_HAS_META_ENTRY) indicating whether
or not metadata was cached, and if that's not set on a later
lookup, then don't return it if it doesn't have metadata but we
need the metadata.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoTest symlink target changes between stat and readlink
Johannes Berg [Thu, 31 Dec 2020 22:10:05 +0000 (23:10 +0100)]
Test symlink target changes between stat and readlink

Test the changes in "metadata: fix symlink stat() vs. readlink() race".

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message; rework to use injection]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agometadata: fix symlink stat() vs. readlink() race
Johannes Berg [Thu, 31 Dec 2020 22:29:58 +0000 (23:29 +0100)]
metadata: fix symlink stat() vs. readlink() race

We might stat() the file and get some size, but by the time
we readlink() the link has changed and the size is something
else.

Arguably, we can't really avoid races here if we don't have
a consistent snapshot of the filesystem to save, however, in
this case we later get an assertion failure when the data is
read back from the index (or repo), and it's easy to avoid.

Set the size from the actually recorded symlink target.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoTest file size changes during save
Johannes Berg [Thu, 31 Dec 2020 22:10:05 +0000 (23:10 +0100)]
Test file size changes during save

Test the changes in "save: fix race in data vs. metadata size".

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message; rework to use injection]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosave: fix race in data vs. metadata size
Johannes Berg [Thu, 31 Dec 2020 22:20:53 +0000 (23:20 +0100)]
save: fix race in data vs. metadata size

Fix the race in data vs. metadata size by overriding the metadata
size with what we actually read and stored into the repository.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobup-midx.md: don't include the index-cache in the --dir default
Rob Browning [Sat, 29 May 2021 18:57:56 +0000 (13:57 -0500)]
bup-midx.md: don't include the index-cache in the --dir default

We stopped including it in 6819a53d7c033b5b4e6b6efb87a4f9d4023be7a2

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agomidx: make passing idx files along with --dir work
Johannes Berg [Wed, 30 Dec 2020 22:10:19 +0000 (23:10 +0100)]
midx: make passing idx files along with --dir work

If you do something like

bup midx --dir /some/dir/ ...

with some idx files ("..."), then things don't actually work
properly because --dir gets ignored in this case. Fix that.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobup.main: restore support for --x=y style arguments
Rob Browning [Sat, 22 May 2021 17:25:01 +0000 (12:25 -0500)]
bup.main: restore support for --x=y style arguments

Previously removed in 3a01e9021f7e7c5eafec996f4fd5d5613bb776ce.

Reported-by: Mark J Hewitt <mjh@idnet.com>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoREADME: don't use a markdown link for irc at all
Rob Browning [Fri, 28 May 2021 19:16:13 +0000 (14:16 -0500)]
README: don't use a markdown link for irc at all

...since it's currently rendered in a way that hides all the useful
information.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoREADME: use normal markdown link for ircs
Rob Browning [Fri, 28 May 2021 19:13:50 +0000 (14:13 -0500)]
README: use normal markdown link for ircs

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoREADME: fix irc web interface link typo
Rob Browning [Fri, 28 May 2021 19:11:40 +0000 (14:11 -0500)]
README: fix irc web interface link typo

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoREADME: fix IRC links and add a list mailto
Rob Browning [Fri, 28 May 2021 18:56:34 +0000 (13:56 -0500)]
README: fix IRC links and add a list mailto

2 years agoREADME: mention #bup
Rob Browning [Fri, 28 May 2021 17:36:02 +0000 (12:36 -0500)]
README: mention #bup

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agodev/checksum: don't rely on bash /dev/stdin
Rob Browning [Sun, 16 May 2021 17:09:45 +0000 (12:09 -0500)]
dev/checksum: don't rely on bash /dev/stdin

It's only supported by bash in redirections, so we can't use it as the
src path.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agodev/checksum: paper over platform differences
Rob Browning [Sun, 16 May 2021 02:14:24 +0000 (21:14 -0500)]
dev/checksum: paper over platform differences

Replace use of sha1sum with dev/checksum in order to accommodate
platform differences, i.e. coreutils sha1sum vs freebsd sha1.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agobup_limited_vint_pack: revert to malloc/free
Rob Browning [Sun, 16 May 2021 16:14:41 +0000 (11:14 -0500)]
bup_limited_vint_pack: revert to malloc/free

Replace PyMem_RawMalloc and PyMemRawFree with malloc and free,
reverting my adjustment in 3b98812aa98a1590e2bebbc83c113c07e4c002bc.

Python 2 doesn't support them, and at least right now, regardless of
what the current python docs suggest, at the moment, they're just
trivial wrappers around malloc and free.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agocirrus: upgrade to freebsd-12.2-release-amd64
Rob Browning [Sun, 16 May 2021 06:19:33 +0000 (01:19 -0500)]
cirrus: upgrade to freebsd-12.2-release-amd64

The previous version (12.1) had been failing like this:

  ld-elf.so.1: /usr/local/lib/libpython3.7m.so.1.0: Undefined symbol "close_range@FBSD_1.6"

and there was some indication it might be caused by an "incomplete
upgrade".

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
2 years agoweb: make leaf path component a link too
Rob Browning [Sat, 1 May 2021 19:35:47 +0000 (14:35 -0500)]
web: make leaf path component a link too

This allows it to serve the same purpose as the removed "." link,
without any additional clutter.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agoweb: omit '.' link
Johannes Berg [Sun, 17 May 2020 20:03:24 +0000 (22:03 +0200)]
web: omit '.' link

There's no value in offering a link to go to the same place
you're already at, it's just clutter. Should it really be
necessary there's always a browser refresh instead.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosave: fix symlink target race
Johannes Berg [Thu, 31 Dec 2020 22:33:03 +0000 (23:33 +0100)]
save: fix symlink target race

If the symlink target changes while save is running, we can end up
reading one target and recording it in the metadata, but then
recording a different one (with perhaps a different size) in the
blob.

Fix this by not reading the symlink again, but just using the one
we already have in the metadata.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agotest: test index vs. save file type change
Johannes Berg [Fri, 1 Jan 2021 10:04:59 +0000 (11:04 +0100)]
test: test index vs. save file type change

In this case, the item shouldn't be recorded at all.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 years agosave: skip/balk if an entry changed type since indexing
Johannes Berg [Sat, 2 Jan 2021 19:58:44 +0000 (20:58 +0100)]
save: skip/balk if an entry changed type since indexing

If an entry changed file type (link, regular file, ...) since
indexing, then all kinds of weird things can happen. Skip the
item in such cases and record an error.

This also requires adjusting the test that actually provokes
a failure in metadata read - now we don't store the file at
all, so modify the test to account for that behaviour in the
check.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>