Handle external commands directly rather than via renamed symlinks

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

fuse: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

fuse-cmd: copy to bup.cmd.fuse

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

fsck: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

fsck-cmd: copy to bup.cmd.fsck

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

get: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

get-cmd: copy to bup.cmd.get

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

restore: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

restore-cmd: copy to bup.cmd.restore

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

save: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

save-cmd: copy to bup.cmd.save

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

on--server: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

on--server-cmd: copy to bup.cmd.on__server

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

on: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

on-cmd: copy to bup.cmd.on

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

import-duplicity: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

import-duplicity-cmd: copy to bup.cmd.import_duplicity

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

server: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

server-cmd: copy to bup.cmd.server

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

prune-older: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

prune-older-cmd: copy to bup.cmd.prune_older

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

mux: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

mux-cmd: copy to bup.cmd.mux

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

list-idx: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

list-idx-cmd: copy to bup.cmd.list_idx

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

drecurse: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

drecurse-cmd: copy to bup.cmd.drecurse

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

damage: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

damage-cmd: copy to bup.cmd.damage

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

ftp: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

ftp-cmd: copy to bup.cmd.ftp

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

web: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

web-cmd: copy to bup.cmd.web

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

daemon: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

daemon-cmd: copy to bup.cmd.daemon

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

tick: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

tick-cmd: copy to bup.cmd.tick

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

tag: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

tag-cmd: copy to bup.cmd.tag

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

midx: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

midx-cmd: copy to bup.cmd.midx

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

random: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

random-cmd: copy to bup.cmd.random

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

memtest: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

memtest-cmd: copy to bup.cmd.memtest

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

margin: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

margin-cmd: copy to bup.cmd.margin

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

features: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

features-cmd: copy to bup.cmd.features

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

split: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

split-cmd: copy to bup.cmd.split

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

init: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

init-cmd: copy to bup.cmd.init

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

gc: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

gc-cmd: copy to bup.cmd.gc

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

rm: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

rm-cmd: copy to bup.cmd.rm

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

index: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

index-cmd: copy to bup.cmd.index

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

join: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

join-cmd: copy to bup.cmd.join

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

help: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

help-cmd: copy to bup.cmd.help

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bloom: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bloom-cmd: copy to bup.cmd.bloom

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Makefile: check for bup version failures

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

version: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

version-cmd: copy to bup.cmd.version

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

xstat: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

xstat-cmd: copy to bup.cmd.xstat

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

meta: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

meta-cmd: copy to bup.cmd.meta

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

ls: convert to internal command

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

ls-cmd: copy to bup.cmd.ls

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

cat-file-cmd: migrate to internal bup.cmd.cat_file

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

cat-file-cmd: copy to bup.cmd.cat_file

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Remove $(dirname "$0") from sys.path

Python adds $(dirname "$0") to the sys.path, which we definitely don't
want (why would we?), so remove it early in the startup process.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bup: include built-in subcommands in help

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bup: filter stdout/stderr via thread/pipe for internal subcommands

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

filter_output: handle multiple srcs/dests as lists

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

configure-sampledata: cp lib/bup/*.py, not cmd/*.py

So the tests won't depend on the location of the subcommands, in
preparation for their move to lib/bup/cmd.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Make it possible to close all catpipes

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

options: add parse_bytes

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

git: explicitly request that Popen close_fds

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Add --import-py-module

This can be used by tests (and will be used by test-save-errors) to
override functions for error injection, etc.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Add handle_ctrl_c to bup itself

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Enable --profile for built-in subcommands

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bup: don't convert bup_dir to bytes twice

Thanks to Johannes Berg for reporting the issue.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

bup: add support for "internal" subcommands

Look for the existing something-cmd.py commands as both
lib/cmd/bup-something and lib/bup/cmd/something.py, and don't load the
latter as a module unless specifically allowed, so that we can create
separate rename commits as we convert each relevant external command
to an internal command.  This should make it easier to see what
actually changed in the git history, e.g. with --ignore-space-change.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Update base_version for 0.32 release

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Update README, HACKING, and release notes for 0.32

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

prep-for-macos-build: don't run brew update

The update was causing tests to fail, and the previous attempt to
address the issue e244aa8ec2ea4b754d6242ee81dd0d3c6dace543 didn't.
Thanks to Johannes Berg proposing the fix.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

README: add markdown mode indicator

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

index: add newline to 'corrupt index' message

This otherwise prints the traceback afterwards starting
on the same line, which looks odd.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

index: remove pointless git.repo() invocation

Since the only way we can get to clear_index() is with
indexfile having previously been assigned

    indexfile = git.repo(b'bupindex')

since opt.clear and opt.indexfile are explicitly mutually
exclusive, there's no point in calling git.repo() again,
it has no effect.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

get: remove pointless git.repo() call

There's no point in doing "opt.source or git.repo()" since
we just pass the result to LocalRepo(), which already does
exactly the same thing.

Remove the src_dir variable entirely.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>

prune-older: fix output with --pretend

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>

bup-web: handle 4 item getsockname() result for IPv6 sockets

So bup-web will create a new network socket to listen on for HTTP
connections and it may use either IPv4 or IPv6 (or both) addresses,
depending on the configuration of the system. This is done using the
tornado.netutil.bind_sockets().

Currently, bup-web does not restrict the address family to either
AF_INET/AF_INET6 and according to the documentation of
[tornado](https://www.tornadoweb.org/en/stable/netutil.html?highlight=netutil#tornado.netutil.bind_sockets)
it will use both address families and listen for connections on IPv4 and
IPv6 addresses.

> Family may be set to either socket.AF_INET or socket.AF_INET6 to
> restrict to IPv4 or IPv6 addresses, otherwise both will be used if
> available.

However, when your system has only IPv6 enabled, the print() statement
may fail with:

Traceback (most recent call last):
  File "/opt/bup/lib/bup/cmd/bup-web", line 310, in <module>
    print('Serving HTTP on %s:%d...' % sockets[0].getsockname())
TypeError: not all arguments converted during string formatting

For systems that have IPv4 as well as IPv6 enabled, it may or may not
fail, depending on whether the first returned socket by getsockname() is
for AF_INET6 or AF_INET4

The reason is, getsockname() for a AF_INET6 connections returns a tuple with
4 items: `(host, port, flowinfo, scope_id)`, while for AF_INET4
addresses, it contains only `(host, port)`.

Since we are only interested in the first 2 items, make sure to return
only the first 2 items, which are `host` and `port` for IPv4 (AF_INET)
and IPv6 (AF_INET6).

Signed-off-by: Christian Brabandt <cb@256bit.org>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit summary]

restore: fix error message for inaccessible items

This actually shows up during tests, but it's hidden as it's
an error where we _expect_ bup to fail (and don't check why).
Fix the message in this case.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>

Replace atoi with explicit checks; rm unused atof

Replace atoi with explicit checks, making the intention a bit clearer
at each call site, and avoiding trouble we've had in the past with the
default to 0 for any unrecognized input.

This does make us stricter about the values (and the environment),
i.e. BUP_TTY_WIDTH=random-noise will now cause a crash.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>