Compare commits

..

No commits in common. "master" and "v0.23.0" have entirely different histories.

30 changed files with 4125 additions and 8926 deletions

3
.gitattributes vendored
View File

@ -1,3 +0,0 @@
# remove files from deployment using `git archive`
.gitattributes export-ignore
.gitignore export-ignore

3
.github/FUNDING.yml vendored
View File

@ -1,3 +0,0 @@
# These are supported funding model platforms
custom: ['https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S', 'https://btc.com/19DYtoEepxBmn9ZPspJGZrhCtySKCxPcP1']

View File

@ -1,26 +0,0 @@
---
name: Codespell
on:
push:
branches: [master]
pull_request:
branches: [master]
permissions:
contents: read
jobs:
codespell:
name: Check for spelling errors
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Codespell
uses: codespell-project/actions-codespell@v2
with:
check_filenames: true
skip: ".git,*.pdf,*.svg"
ignore_words_list: uptodate

364
ChangeLog
View File

@ -1,367 +1,3 @@
btrbk-0.32.6
* Fix backup of unrelated (by parent_uuid) snapshots (close #339).
* Remove echo -e for portability (close #506).
* Support btrfs send protocol v2 (send_protocol and
send_compressed_data config options).
* Add bzip3 support.
* Convert ssh_filter_btrbk.sh to POSIX sh, and harden it.
* Slight change in ssh_filter_btrbk.sh logging output.
* Minor bugfixes, stability and documentation improvements.
btrbk-0.32.5
* Correct handling of zero-size raw info file (close #491).
btrbk-0.32.4
* Fix regression: wrong deprecation warnings in some cases.
btrbk-0.32.3
* Fix deletion of many subvolumes at once (close #476).
* Allow disabling ssh_identity and ssh_user options.
* Minor bugfixes and documentation improvements.
btrbk-0.32.2
* Fix regression: archive on missing target directories.
* Fix action "config print".
* Print version and help message to stdout (close #452).
* Check results of filesystem usage (close #469).
* Add "compat ignore_receive_errors" option (close #383).
* Support multiple gpg recipients (close #471).
* Fix changelog: remove "compat missing_otime" item in v0.32.1.
* Fix changelog: correct "warn_unknown_targets" item in v0.31.3.
* Minor bugfixes and framework improvements.
btrbk-0.32.1
* Fix regression: correctly read mountinfo (close #445).
* Fix regression: filter paths for "lsbtr" (action "ls").
* Add backend btrfs-progs-doas (close #444).
* Allow directory traversal for local command line arguments.
* Display all source subvolumes in "usage" action.
* Handle errors from btrfs filesystem usage.
btrbk-0.32.0
* MIGRATION
- If timestamp_format is not configured, explicitly set
"timestamp_format short" to revert old behavior.
- Update ssh_filter_btrbk.sh on remote hosts.
* Change default for timestamp_format to "long".
* Optional "volume" config section.
* Use "volume" section only for grouping and relative paths.
* Allow absolute path for "subvolume" and "snapshot_dir"
(close #407, #121).
* Support subvolume names with UTF-8 characters (close #392, #213,
#120). Add "safe_commands" option for paranoid people.
* Add "incremental_prefs" configuration option (close #387).
* Change incremental prefs policy, giving snapshots created by btrbk
higher preference than the global ones resolved by parent-uuid.
* Change "incremental_clones" option to boolean.
* Drop support of deprecated raw file format (btrbk < 0.26.0).
* Drop support of deprecated "-r, --resume-only" cmdline option.
* Fix ambiguous naming in --format=raw output of "list" actions.
* Accept quoted values in config.
* Use single quotes for raw table output.
btrbk-0.31.3
* Add "warn_unknown_targets" configuration option (close #393).
* Add -1,--single-column command-line option for listing actions.
* Allow relative path command line argument for all actions.
* Add support for zstd adaptive compression (close #397).
* Add contrib/tools/btrbk_restore_raw.py (close #401).
* Minor bugfixes and framework improvements.
btrbk-0.31.2
* MIGRATION
- Update ssh_filter_btrbk.sh on remote hosts.
* ssh_filter_btrbk.sh: Fix security vulnerability.
Specially crafted commands may be executed without being properly
checked. Applies to remote hosts filtering ssh commands using
ssh_filter_btrbk.sh in authorized_keys.
* Warn if no subvolume defined in config (close #378).
btrbk-0.31.1
* Bugfix: duplicate snapshot location check (close #360).
* Bugfix: action "list all" (close #367).
* btrbk-mail: optionally prefix command output lines.
btrbk-0.31.0
* Add action "extents [diff]": print accurate disk space usage based
on block regions (extent data, FIEMAP ioctl).
* Improve action "list" and "stats" table output (close #334).
* Add subcommand "list all", make default for "list" action.
* Change semantics on output of action "list": show only btrbk
snapshots/backups (close #333).
* Print human readable units in action "diff" (close #345).
* Allow custom table format definitions from command line.
* Add Bash completion (close #343).
* Add support for zstd compression (close #358).
* Bugfix: fs_path fallback for action "origin" and logging.
* Bugfix: makefile bin-links race (close #341).
btrbk-0.30.0
* MIGRATION
- In your scripts (e.g. cron jobs, systemd units), replace
"/usr/sbin/btrbk" with "/usr/bin/btrbk".
* Install to "/usr/bin" instead of "/usr/sbin".
* Support IPv6 addresses (close #332).
* Add "compat busybox" configuration option (close #327).
* Create "lsbtr" symlink (alias for "btrbk ls").
* Improve action "ls" (allow url, bugfixes).
* Add "backend_local_user" configuration option.
* btrbk-mail: add more elaborated email and rsync options.
* Minor bugfixes and framework improvements.
btrbk-0.29.1
* Allow multiple path arguments for action "ls".
* Change default output to format=short for action "ls".
* ssh_filter_btrbk.sh: exclude "btrfs subvolume show|list" from
restrict-path (close #309).
* Bugfix: fix filter statement match on wildcards (close #311).
* Fix regression: print ssh errors only if needed.
btrbk-0.29.0
* MIGRATION
- If stream_buffer or rate_limit is configured, please install
"mbuffer" (version >= 20180505). Note that mbuffer is only
required on remote hosts if stream_buffer_remote or
rate_limit_remote is set.
- In order to mimic old behavior, replace rate_limit with
rate_limit_remote, and read btrbk.conf(5).
- If you are using table output other than --format=raw in your
scripts, note that the column headings changed from pretty
two-line to (uppercase) one-line format. Either add --pretty
option to mimic old behavior, or adapt your scripts.
* Dropped run-time dependency on "pv" (in favor of "mbuffer").
* Combined stream_buffer and rate_limit: for rate_limit, use
"mbuffer" (on local host) instead of "pv" (on source host).
* Add stream_buffer_remote and rate_limit_remote options.
* Use "mbuffer" instead of "pv" for --progress display.
* Add stream_buffer functionality for raw targets.
* Add action "ls": list all btrfs subvolumes below given path.
* Change table output: print single-line, uppercase headings.
* ssh_filter_btrbk.sh: whitelist mkdir if -t,--target option is set,
used by "btrbk archive" action (close #297).
* Bugfix: handle subvolumes with special characters (close #288).
* Bugfix: don't display "<no_action>" in backup summary if volume is
skipped by --exclude or noauto (close #291).
* Bugfix: systemd: Use WantedBy=timers.target instead of
WantedBy=multi-user.target in btrbk.timer (close #293).
btrbk-0.28.3
* Bugfix: return exitcode=10 if action skipped by stray subvolumes
(close: #285).
* Bugfix: correctly handle root subvolumes having uuid.
* Bugfix: fix mount point resolving.
btrbk-0.28.2
* Fix regression: crash if raw target dir is empty (close #281).
* Bugfix: honor incremental_resolve on targets: resolve from
mountpoint instead of directory.
* Bugfix: handle directories named "0" correctly.
btrbk-0.28.1
* Fix regression: crash if deep parent/child relations exceed
depth=100 (close #279).
* Remove pre-generated man pages, requires "asciidoctor" for builds.
btrbk-0.28.0
* MIGRATION
- Replace "ssh_port" options with "ssh://hostname[:port]" notation
on "volume" or "target" declarations.
- Check filter statements in your scripts ("btrbk run <filter>"),
especially when using groups having the same name as subvolumes.
* Fix table format "resolved" (btrbk list snapshots|backups).
* Add btrbk-verify: tool for automated backup integrity check based
on rsync (close #256).
* Use "ssh://hostname[:port]" notation instead of ambiguous
"ssh_port" option (now deprecated).
* Bugfix: correctly handle multiple volume sections with same host,
but distinct port numbers (virtual machines setups, close #233).
* Remove selected rows ("*host", "*port") from table output if all
values are empty.
* Consider all parent/child relations (close #255).
* Disallow unrelated parents for btrfs-send if incremental=strict.
* Add clone sources to btrfs-send if necessary.
* Add "incremental_resolve" configuration option.
* Add "incremental_clones" configuration option.
* Add "noauto" configuration option.
* Add "--exclude" command line option.
* Make target_type optional, defaults to "send-receive".
* Use more sophisticated filter arguments.
* Allow whitespace separated "group" option.
btrbk-0.27.2
* Bump dependency >= btrfs-progs-4.12 (as of btrbk-0.27.0).
* Trigger autofs mount while resolving mountpoints (close #259).
* Bugfix: fix scheduler when overriding "target_preserve_min" in
combination with global "target" section.
btrbk-0.27.1
* MIGRATION
- Update ssh_filter_btrbk.sh on remote hosts if using
backend=btrfs-progs-sudo.
* Bugfix (regression): call "sudo -n readlink" for
backend=btrfs-progs-sudo (close #257).
* Bugfix (regression): fix failing "config print" command.
* Don't print title and blank lines for --format=raw, syslog and
transaction log.
btrbk-0.27.0
* MIGRATION
- Run "btrbk prune --dry-run --print-schedule" and check if any
snapshots/backups would get deleted [1] [2]. If you want to
forcibly preserve a snapshot/backup forever, rename it (btrbk
ignores subvolumes with unknown naming scheme), e.g.:
"mv mysubvol.YYYYMMDD mysubvol.YYYYMMDD.keep_forever"
- Update ssh_filter_btrbk.sh on remote hosts (btrbk always calls
"readlink" and "cat /proc/self/mountinfo").
* Add "preserve_hour_of_day" configuration option (close #202).
* Allow backup of filesystem root using "subvolume ." (close #240).
* Add "-S" command line option (shortcut for --print-schedule).
* Bugfix: correct scheduling of "first weekly backup in month/year"
(close #217) [1] [2].
* Bugfix: add "btrfs filesystem usage" backend mapping (close #222).
* Bugfix: do not fail (exitcode=10) if targets are not accessible on
"btrfs snapshot --preserve".
* Bugfix: if "incremental no" is configured on target, do not keep
latest common snapshot.
* Enhance internal data structures:
- Allow snapshot_dir to be a mountpoint.
- Search complete target tree for correlated subvolumes.
- Include snapshots from all mountpoints as candidates (disabled
due to upstream bug: github.com/kdave/btrfs-progs/issues/96).
- Read /proc/self/mountinfo instead of /proc/self/mounts.
- Always read /proc/self/mountinfo.
- Resolve realpath using readlink(1).
* Fallback to "asciidoctor" for manpage generation (close #219).
[1] https://github.com/digint/btrbk/issues/217
[2] https://github.com/digint/btrbk/commit/719fb5f
btrbk-0.26.1
* Add "archive_exclude" configuration option.
* Add warning on redefined configuration option.
* Bugfix: fix parsing of "openssl_iv_size" configuration option.
* Bugfix: fix filter statement matching for volume=/ (close #209).
btrbk-0.26.0
* MIGRATION
- If you are using raw targets, make sure to run the
"raw_suffix2sidecar" utility in each target directory.
* Support for btrfs-progs v4.13.2: adapt parsing of "btrfs sub list"
output (close #192).
* Add "resume" command, replacement for "-r, --resume-only" command
line option (which is now deprecated).
* Add "snapshot" command (close #150).
* Add "prune" command.
* Add "--preserve-snapshots" and "--preserve-backups" options.
* Add "--wipe" command line option (close #173).
* Change raw backup format (sidecar file instead of uuid in file).
* Honor target_preserve for raw targets (delete raw targets).
* Add symmetric encryption for raw targets (close #157).
* Add "{snapshot,target,archive}_qgroup_destroy" configuration
options (close #49, #189).
* Do not run in "perl taint mode" by default: remove "perl -T" in
hashbang; hardcode $PATH only if taint mode is enabled.
* Remove "duration" column from transaction_log/transaction_syslog.
* Resolve ancestors (recursive on parent_uuid chain) when searching
for latest common subvolume.
* Generate man pages from asciidoc (remove raw groff sources).
* Bugfix: ssh_filter_btrbk: accept mbuffer command (stream_buffer).
* Bugfix: print correct (end-)time in transaction_log.
* Bugfix: check path when expanding wildcards (close #181).
* Bugfix: never show failed deletes in summary (close #183).
btrbk-0.25.1
* Support for btrfs-progs v4.12: fix parsing of "btrfs sub show"
output, which now prints relative paths (close #171).
* Add "stream_buffer" configuration option (close #154).
* Bugfix: accept "no" for "transaction_log", "transaction_syslog"
and "lockfile" configuration options.
* Show "up-to-date" status for backups in "stats" command.
* Show "correlated" status instead of "orphaned" in "stats" command.
* Check source subvolumes for readonly and received_uuid flags, and
abort if one of them is set.
btrbk-0.25.0
* MIGRATION
- If you call ssh_filter_btrbk.sh with "--sudo" option, make sure
to set "backend btrfs-progs-sudo" in btrbk.conf for this host.
- If "rate_limit" is enabled, update ssh_filter_btrbk.sh on remote
source hosts, and make sure the "pv" command is available there.
* Allow converting backup disks to source disks (close #114).
* Add "backend btrfs-progs-sudo" configuration option (close #115).
* Show aggregate "size" and "used" for "usage" action (close #119).
* Add "raw_target_split" configuration option (close #125).
* Allow trailing comments in btrbk.conf (close #129).
* Bugfix: rate limiting must be done after compression (close #134).
* raw_target_encrypt: Always set "gpg --no-random-seed-file":
prevents creation of "~/.gnupg/random_seed" with slight performance
penalty.
btrbk-0.24.0
* MIGRATION
- update ssh_filter_btrbk.sh on remote hosts if "stream_compress"
is enabled. Also add "--compress" option to ssh_filter_btrbk.sh
invocation if "stream_compress" is enabled.
* Add "stream_compress" configuration option.
* Perform extra metadata check on target subvolume after "btrfs
receive" (adds an additional call to "btrfs subvolume show").
* Bugfix: Replace "realpath" with "readlink" in ssh_filter_btrbk.sh
* Add "raw_target_block_size" configuration option (close #105).
* Add "backend" configuration option (experimental).
* Bugfix: fix "list latest" with no snapshots (close #111).
* Support for btrfs-progs v4.8.3: fix parsing of "btrfs sub show"
output, which has changed for toplevel subvolume.
btrbk-0.23.3
* Replace "realpath -e" with "readlink -e" for mountpoint discovery
(close #92).
* Dynamically set BINDIR in btrbk.service by Makefile.
* Add ChangeLog to DOCDIR in Makefile.
btrbk-0.23.2
* Added "lockfile" configuration option and --lockfile command line
option (close: #81).
* Bugfix: raw targets: correctly handle multiple backups in same
target directory (close: #87).
* Use relative instead of absolute binary calls in btrbk-mail.
btrbk-0.23.1
* Bugfix: set correct parent section when propagating targets
(close: #85).
* Add syslog output of transaction log (close #82).
* Do not print headers to transaction log anymore.
* Explain "orphaned" status in "stats" command, and suppress it on
"list backups" command (close: #76).
btrbk-0.23.0
* INCOMPATIBLE CONFIGURATION:

134
Makefile
View File

@ -1,113 +1,49 @@
#
# Btrbk is a single perl script, and does not require any special
# installation procedures or libraries. There is no need to run the
# "all" build target if you don't want to build the man pages (see
# doc/Makefile).
#
# Note: systemd units (file names) are hardcoded in "install-systemd"
# build target for simplicity.
#
PN = btrbk
BIN = btrbk
BIN_LINKS = lsbtr
CONFIGS = btrbk.conf.example
DOCS = ChangeLog \
README.md
SCRIPTS = ssh_filter_btrbk.sh \
contrib/cron/btrbk-mail \
contrib/cron/btrbk-verify \
contrib/migration/raw_suffix2sidecar \
contrib/crypt/kdf_pbkdf2.py \
contrib/tools/btrbk_restore_raw.py
PREFIX ?= /usr
CONFDIR = /etc
CRONDIR = /etc/cron.daily
BINDIR = $(PREFIX)/sbin
DOCDIR = $(PREFIX)/share/doc/$(PN)
SCRIPTDIR = $(PREFIX)/share/$(PN)/scripts
SYSTEMDDIR = $(PREFIX)/lib/systemd/system
MAN1DIR = $(PREFIX)/share/man/man1
MAN5DIR = $(PREFIX)/share/man/man5
PN = btrbk
PREFIX ?= /usr
CONFDIR = /etc
CRONDIR = /etc/cron.daily
BINDIR = $(PREFIX)/bin
DOCDIR = $(PREFIX)/share/doc/$(PN)
SCRIPTDIR = $(PREFIX)/share/$(PN)/scripts
SYSTEMDDIR = $(PREFIX)/lib/systemd/system
BASHCOMPDIR = $(PREFIX)/share/bash-completion/completions
MAN1DIR = $(PREFIX)/share/man/man1
MAN5DIR = $(PREFIX)/share/man/man5
ifeq ($(COMPRESS), yes)
DOCS := $(addsuffix .gz,$(DOCS))
endif
replace_vars = sed \
-e "s|@PN@|$(PN)|g" \
-e "s|@CONFDIR@|$(CONFDIR)|g" \
-e "s|@CRONDIR@|$(CRONDIR)|g" \
-e "s|@BINDIR@|$(BINDIR)|g" \
-e "s|@DOCDIR@|$(DOCDIR)|g" \
-e "s|@SCRIPTDIR@|$(SCRIPTDIR)|g" \
-e "s|@SYSTEMDDIR@|$(SYSTEMDDIR)|g" \
-e "s|@BASHCOMPDIR@|$(BASHCOMPDIR)|g" \
-e "s|@MAN1DIR@|$(MAN1DIR)|g" \
-e "s|@MAN5DIR@|$(MAN5DIR)|g"
all: man
install: install-bin install-bin-links install-etc install-completion install-systemd install-share install-man install-doc
all:
@echo 'nothing to do for "all"'
install-bin:
@echo 'installing binary...'
install -d -m 755 "$(DESTDIR)$(BINDIR)"
install -p -m 755 $(BIN) "$(DESTDIR)$(BINDIR)"
install-bin-links: install-bin
@echo 'installing symlinks...'
for name in $(BIN_LINKS); do \
ln -s -n -f $(BIN) "$(DESTDIR)$(BINDIR)/$$name"; \
done
install-etc:
@echo 'installing example configs...'
install -d -m 755 "$(DESTDIR)$(CONFDIR)/btrbk"
install -p -m 644 $(CONFIGS) "$(DESTDIR)$(CONFDIR)/btrbk"
install-completion:
@echo 'installing bash completion...'
install -d -m 755 "$(DESTDIR)$(BASHCOMPDIR)"
install -p -m 644 contrib/bash/completion.bash "$(DESTDIR)$(BASHCOMPDIR)/$(BIN)"
for name in $(BIN_LINKS); do \
ln -s -n -f $(BIN) "$(DESTDIR)$(BASHCOMPDIR)/$$name"; \
done
@echo 'installing main script and config...'
install -Dm644 btrbk.conf.example "$(DESTDIR)$(CONFDIR)/btrbk/btrbk.conf.example"
install -Dm755 $(PN) "$(DESTDIR)$(BINDIR)/$(PN)"
install-systemd:
@echo 'installing systemd service units...'
install -d -m 755 "$(DESTDIR)$(SYSTEMDDIR)"
$(replace_vars) contrib/systemd/btrbk.service.in > contrib/systemd/btrbk.service.tmp
$(replace_vars) contrib/systemd/btrbk.timer.in > contrib/systemd/btrbk.timer.tmp
install -p -m 644 contrib/systemd/btrbk.service.tmp "$(DESTDIR)$(SYSTEMDDIR)/btrbk.service"
install -p -m 644 contrib/systemd/btrbk.timer.tmp "$(DESTDIR)$(SYSTEMDDIR)/btrbk.timer"
rm contrib/systemd/btrbk.service.tmp
rm contrib/systemd/btrbk.timer.tmp
install -Dm644 contrib/systemd/btrbk.service "$(DESTDIR)$(SYSTEMDDIR)/btrbk.service"
install -Dm644 contrib/systemd/btrbk.timer "$(DESTDIR)$(SYSTEMDDIR)/btrbk.timer"
install-share:
@echo 'installing auxiliary scripts...'
install -d -m 755 "$(DESTDIR)$(SCRIPTDIR)"
install -p -m 755 $(SCRIPTS) "$(DESTDIR)$(SCRIPTDIR)"
install -Dm755 ssh_filter_btrbk.sh "$(DESTDIR)$(SCRIPTDIR)/ssh_filter_btrbk.sh"
install -Dm755 contrib/cron/btrbk-mail "$(DESTDIR)$(SCRIPTDIR)/btrbk-mail"
install-man: man
@echo 'installing man pages...'
@$(MAKE) -C doc install-man
install-man:
@echo 'installing manpages...'
install -Dm644 doc/btrbk.1 "$(DESTDIR)$(MAN1DIR)/btrbk.1"
install -Dm644 doc/ssh_filter_btrbk.1 "$(DESTDIR)$(MAN1DIR)/ssh_filter_btrbk.1"
install -Dm644 doc/btrbk.conf.5 "$(DESTDIR)$(MAN5DIR)/btrbk.conf.5"
gzip -9 "$(DESTDIR)$(MAN1DIR)/btrbk.1"
gzip -9 "$(DESTDIR)$(MAN1DIR)/ssh_filter_btrbk.1"
gzip -9 "$(DESTDIR)$(MAN5DIR)/btrbk.conf.5"
install-doc: $(DOCS)
install-doc:
@echo 'installing documentation...'
install -d -m 755 "$(DESTDIR)$(DOCDIR)"
install -p -m 644 $(DOCS) "$(DESTDIR)$(DOCDIR)"
@$(MAKE) -C doc install-doc
install -Dm644 README.md "$(DESTDIR)$(DOCDIR)/README.md"
install -Dm644 doc/FAQ.md "$(DESTDIR)$(DOCDIR)/FAQ.md"
install -Dm644 doc/upgrade_to_v0.23.0.md "$(DESTDIR)$(DOCDIR)/upgrade_to_v0.23.0.md"
gzip -9 "$(DESTDIR)$(DOCDIR)/README.md"
gzip -9 "$(DESTDIR)$(DOCDIR)/FAQ.md"
gzip -9 "$(DESTDIR)$(DOCDIR)/upgrade_to_v0.23.0.md"
man:
@echo 'generating manpages...'
@$(MAKE) -C doc man
clean:
rm -f *.gz
@$(MAKE) -C doc clean
%.gz : %
gzip -9 -n -c $< > $@
install: install-bin install-systemd install-share install-man install-doc

703
README.md
View File

@ -1,7 +1,7 @@
Introduction
============
Btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs
btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs
specific capabilities to create atomic snapshots and transfer them
incrementally to your backup locations.
@ -9,195 +9,143 @@ The source and target locations are specified in a config file, which
allows to easily configure simple scenarios like "laptop with locally
attached backup disks", as well as more complex ones, e.g. "server
receiving backups from several hosts via ssh, with different retention
policies".
policy".
Key Features:
* Atomic snapshots
* Incremental backups
* Flexible retention policy
* Configurable retention policy
* Backups to multiple destinations
* Transfer via ssh
* Robust recovery from interrupted backups (for removable and mobile
devices)
* Archive to offline storage
* Encrypted backups to non-btrfs storage
* Resume of backups (if backup target was not reachable for a while)
* Encrypted backups to non-btrfs destinations
* Wildcard subvolumes (useful for docker and lxc containers)
* Transaction log
* Comprehensive list and statistics output
* Resolve and trace btrfs parent-child and received-from relationships
* List file changes between backups
* Calculate accurate disk space usage based on block regions
* Display file changes between two backups
Btrbk is designed to run as a cron job for triggering periodic
btrbk is designed to run as a cron job for triggering periodic
snapshots and backups, as well as from the command line (e.g. for
instantly creating additional snapshots).
###Upgrading from v0.22.2
Please read the [upgrade guide](doc/upgrade_to_v0.23.0.md) if you are updating from btrbk <= v0.22.2.
Installation
============
Btrbk is a single perl script, and does not require any special
installation procedures or libraries. Download the latest [btrbk
source tarball], or try latest master:
wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
chmod +x btrbk
sudo ./btrbk ls /
For more information, read the [installation documentation].
[btrbk source tarball]: https://digint.ch/download/btrbk/releases/
[installation documentation]: doc/install.md
btrbk comes as a single executable file (perl script), without the
need of any installation procedures. If you want the package and
man-pages properly installed, follow the instructions below.
### Prerequisites
Prerequisites
-------------
* [btrfs-progs]: Btrfs filesystem utilities >= v4.12
* [btrfs-progs]: Btrfs filesystem utilities >= v3.18.2
* [Perl interpreter]: Probably already installed on your system
* [OpenSSH]: If you want to transfer backups from/to remote locations
* [mbuffer]: If you want rate limiting and progress bars
* [OpenSSH]: If you want to tranfer backups from/to remote locations
[btrfs-progs]: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
[btrfs-progs]: http://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
[Perl interpreter]: https://www.perl.org
[OpenSSH]: https://www.openssh.com
[mbuffer]: https://www.maier-komor.de/mbuffer.html
[OpenSSH]: http://www.openssh.org
Instructions
------------
In order to install the btrbk executable along with the man-pages and
an example configuration file, choose one of the following methods:
### Generic Linux System
Download and unpack the newest stable [btrbk source tarball] and type:
sudo make install
### Gentoo Linux
Grab the digint portage overlay from:
`git://dev.tty0.ch/portage/digint-overlay.git`
emerge app-backup/btrbk
### Debian Based Distros
btrbk is in `stretch (testing) (utils)`: https://packages.debian.org/stretch/btrbk
Packages are also available via NeuroDebian: http://neuro.debian.net/pkgs/btrbk.html
### Arch Linux
btrbk is in AUR: https://aur.archlinux.org/packages/btrbk/
### Alpine Linux
btrbk is in `testing`, install with:
apk add btrbk
[btrbk source tarball]: http://digint.ch/download/btrbk/releases/
Synopsis
========
Please consult the [btrbk(1)] man-page provided with this package for
a full description of the command line options.
Please consult the [btrbk(1)] man-page provided with this package for a
full description of the command line options.
[btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
[btrbk(1)]: http://digint.ch/btrbk/doc/btrbk.html
Configuration
=============
Configuration File
==================
Before running `btrbk`, you will need to create a configuration
file. You might want to take a look at `btrbk.conf.example` provided
with this package. For a detailed description, please consult the
[btrbk.conf(5)] man-page.
After a configuration change, it is highly recommended to check it by
running btrbk with the `-n,--dry-run` option:
When playing around with config-files, it is highly recommended to
check the output using the `dryrun` command before executing the
backups:
# btrbk -c /path/to/myconfig -v -n run
btrbk -c myconfig -v dryrun
This will read all btrfs information on the source/target filesystems
and show what actions would be performed (without writing anything to
the disks).
The examples below assume that the btrfs subvolume containing `home`
and `rootfs` is mounted at `/mnt/btr_pool`. This is usually the btrfs
root subvolume, which always has `subvolid=5`.
Mounting `subvolid=5` is *recommended* (mandatory for btrbk < v0.32.0)
if you want to backup your root filesystem `/`.
/etc/fstab:
/dev/sda1 /mnt/btr_pool btrfs subvolid=5,noatime 0 0
Note that some default btrfs installations (e.g. Ubuntu) use subvolume
names `@` for rootfs (mounted at `/`) and `@home` for `/home`, as a
naming convention. If this is the case on your file system, replace
the `subvolume` declarations in the examples accordingly.
[btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html
[btrbk.conf(5)]: http://digint.ch/btrbk/doc/btrbk.conf.html
Example: Local Regular Snapshots (time-machine)
-----------------------------------------------
The simplest use case is to only create snapshots of your data. This
will obviously not protect it against hardware failure, but can be
useful for:
* protection against inadvertent changes or deletions
* keeping past states of copies from rsync or similar tools
Let's assume you need regular snapshots of your home directory, which
is located in the subvolume `home` of the volume `/mnt/btr_pool`. The
snapshots are to be stored in `btrbk_snapshots` (on the same volume).
/etc/btrbk/btrbk.conf:
timestamp_format long
snapshot_preserve_min 18h
snapshot_preserve 48h
volume /mnt/btr_pool
snapshot_dir btrbk_snapshots
subvolume home
Notice that the `target` option is not provided, and btrbk will only
manage snapshots located on the same volume in `snapshot_dir`. Btrbk
does not create subdirs by default, the snapshot directory must first
be created manually:
# mkdir /mnt/btr_pool/btrbk_snapshots
The "volume" section is merely used as a specifier for a base
directory, and can be skipped if you prefer to configure everything
using absolute paths. The above configuration can also be written as:
snapshot_dir /mnt/btr_pool/btrbk_snapshots
subvolume /mnt/btr_pool/home
If you don't want to mount the btrfs root filesystem to
`/mnt/btr_pool`, you might as well configure it like this:
snapshot_dir /btrbk_snapshots
subvolume /home
Start a dry run (-n, --dry-run):
# btrbk run -n
Create the first snapshot:
# btrbk run
Print schedule (-S, --print-schedule):
# btrbk run -n -S
If it works as expected, configure a cron job to run btrbk hourly:
/etc/cron.hourly/btrbk:
#!/bin/sh
exec /usr/bin/btrbk -q run
Snapshots will now be created every hour. All snapshots are preserved for at
least 18 hours (`snapshot_preserve_min`), whether they are created by the cron
job or manually by calling `sudo btrbk run` on the command line. Additionally,
48 hourly snapshots are preserved (`snapshot_preserve`).
Example: Backups to USB Disk
----------------------------
Example: laptop with usb-disk for backups
-----------------------------------------
In this example, we assume you have a laptop with:
* a disk having a btrfs root subvolume (subvolid=5) mounted on
`/mnt/btr_pool`, containing a subvolume `rootfs` for the root
filesystem (i.e. mounted on `/`) and a subvolume `home` for the
user data,
* a directory or subvolume `/mnt/btr_pool/btrbk_snapshots` which
will hold the btrbk snapshots,
* a disk having a btrfs volume mounted as `/mnt/btr_pool`,
containing a subvolume `rootfs` for the root filesystem and a
subvolume `home` for the user data.
* a backup disk having a btrfs volume mounted as `/mnt/btr_backup`,
containing a subvolume or directory `mylaptop` for the incremental
backups.
containing a subvolume `mylaptop` for the incremental backups.
Retention policy:
* keep all snapshots for 2 days, no matter how frequently you (or
your cron job) run btrbk
* keep daily snapshots for 14 days (very handy if you are on
your cron-job) run btrbk
* keep latest daily snapshots for 14 days (very handy if you are on
the road and the backup disk is not attached)
* keep monthly backups forever
* keep weekly backups for 10 weeks
@ -205,28 +153,26 @@ Retention policy:
/etc/btrbk/btrbk-mylaptop.conf:
snapshot_preserve_min 2d
snapshot_preserve 14d
snapshot_preserve_min 2d
snapshot_preserve 14d
# Create snapshots only if the backup disk is attached
#snapshot_create ondemand
target_min no
target_preserve 20d 10w *m
target_preserve_min no
target_preserve 20d 10w *m
snapshot_dir btrbk_snapshots
snapshot_dir btrbk_snapshots
volume /mnt/btr_pool
target /mnt/btr_backup/mylaptop
subvolume rootfs
target send-receive /mnt/btr_backup/mylaptop
subvolume home
[...]
target send-receive /mnt/btr_backup/mylaptop
/etc/cron.daily/btrbk:
#!/bin/sh
exec /usr/bin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run
exec /usr/sbin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run
* This will create snapshots on a daily basis:
@ -236,56 +182,54 @@ Retention policy:
* `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD`
* `/mnt/btr_backup/mylaptop/home.YYYYMMDD`
If you prefer triggering the backups manually, change the cron command
to run the `snapshot` action instead of `run`. Start the backups
manually by running:
If you want the snapshots to be created only if the backup disk is
attached, simply add the following line to the config:
# btrbk resume
For a quick additional snapshot of your home, run:
# btrbk snapshot home
snapshot_create ondemand
Example: Host-initiated Backup on Fileserver
Example: host-initiated backup on fileserver
--------------------------------------------
Let's say you have a fileserver at "myserver.example.org" where you
want to create backups of your laptop disk. The config could look like
Let's say you have a fileserver at "myserver.mydomain.com" where you
want to create backups of your laptop disk, the config would look like
this:
ssh_identity /etc/btrbk/ssh/id_rsa
ssh_identity /etc/btrbk/ssh/id_rsa
volume /mnt/btr_pool
subvolume rootfs
target /mnt/btr_backup/mylaptop
target ssh://myserver.example.org/mnt/btr_backup/mylaptop
target send-receive /mnt/btr_backup/mylaptop
target send-receive ssh://myserver.mydomain.com/mnt/btr_backup/mylaptop
In addition to the backups on your local usb-disk mounted at
`/mnt/btr_backup/mylaptop`, incremental backups would also be pushed
to `myserver.example.org`.
to `myserver.mydomain.com`.
Example: Fileserver-initiated Backups from Several Hosts
Example: fileserver-initiated backups from several hosts
--------------------------------------------------------
If you're a sysadmin and want to trigger backups directly from your
fileserver, the config would be something like:
ssh_identity /etc/btrbk/ssh/id_rsa
ssh_identity /etc/btrbk/ssh/id_rsa
volume ssh://alpha.example.org/mnt/btr_pool
target /mnt/btr_backup/alpha
volume ssh://alpha.mydomain.com/mnt/btr_pool
subvolume rootfs
target send-receive /mnt/btr_backup/alpha
subvolume home
target send-receive /mnt/btr_backup/alpha
volume ssh://beta.example.org/mnt/btr_pool
target /mnt/btr_backup/beta
volume ssh://beta.mydomain.com/mnt/btr_pool
subvolume rootfs
subvolume dbdata
target send-receive /mnt/btr_backup/beta
This will pull backups from alpha/beta.example.org and locally
create:
subvolume dbdata
target send-receive /mnt/btr_backup/beta
This will pull backups from alpha/beta.mydomain.com and locally create:
* `/mnt/btr_backup/alpha/rootfs.YYYYMMDD`
* `/mnt/btr_backup/alpha/home.YYYYMMDD`
@ -293,7 +237,33 @@ create:
* `/mnt/btr_backup/beta/dbdata.YYYYMMDD`
Example: Multiple Btrbk Instances
Example: local time-machine (hourly snapshots)
----------------------------------------------
If all you want is to create snapshots of your home directory on a
regular basis:
/etc/btrbk/btrbk.conf:
timestamp_format long
snapshot_preserve_min 18h
snapshot_preserve 48h 20d 6m
volume /mnt/btr_pool
snapshot_dir btrbk_snapshots
subvolume home
/etc/cron.hourly/btrbk:
#!/bin/sh
exec /usr/sbin/btrbk -q run
Note that you can run btrbk more than once an hour, e.g. by by calling
`sudo btrbk run` from the command line. With this setup, all those
extra snapshots will be kept for 18 hours.
Example: multiple btrbk instances
---------------------------------
Let's say we have a host (at 192.168.0.42) running btrbk with the
@ -306,65 +276,32 @@ to only fetch the snapshots.
target_preserve 0d 10w *m
volume ssh://192.168.0.42/mnt/btr_pool
target /mnt/btr_backup/my-laptop
subvolume home
snapshot_dir btrbk_snapshots
snapshot_preserve_min all
snapshot_create no
target send-receive /mnt/btr_backup/my-laptop.com
If the server runs btrbk with this config, 10 weeklies and all
monthlies are received from 192.168.0.42. The source filesystem is
never altered because of `snapshot_preserve_min all`.
Example: Virtual Machine Setup
------------------------------
Common virtual machine setups have multiple volume sections with same
host, but distinct port numbers for each machine.
/etc/btrbk/btrbk.conf:
# This propagates to all subvolume sections:
target /mnt/btr_backup/
volume ssh://localhost:2201/mnt/btr_pool
group vm vm01
subvolume home
snapshot_name vm01-home
subvolume data
snapshot_name vm01-data
volume ssh://localhost:2202/mnt/btr_pool
group vm vm02
subvolume home
snapshot_name vm02-home
volume ssh://localhost:2203/mnt/btr_pool
[...]
This will create `/mnt/btr_backup/vm[NN]-home`, `vm[NN]-data`, ...
Note that btrbk holds a single reference to every btrfs filesystem
tree, regarding UUID's as "globally unique". If the configured
subvolumes point to the same filesystem on different machines (ports),
you will see log lines like this when running `btrbk -v`:
```
Assuming same filesystem: "ssh://localhost:2201/dev/sda1", "ssh://localhost:2202/dev/sda1"
```
Example: Backup from non-btrfs Source
Example: backup from non-btrfs source
-------------------------------------
If you want to make backups from a filesystem other than btrfs
(e.g. ext4 or reiserfs), you need to create a *synchronization
subvolume* on the backup disk:
First create a btrfs subvolume on the backup server:
# btrfs subvolume create /mnt/btr_backup/myhost_sync
Configure btrbk to use `myhost_sync` as source subvolume:
In your daily cron script, prior to running btrbk, sync your source to
`myhost_sync`, something like:
rsync -a --delete -e ssh myhost.mydomain.com://data/ /mnt/btr_backup/myhost_sync/
Then run btrbk, with myhost_sync configured *without any targets* as
follows:
volume /mnt/btr_backup
subvolume myhost_sync
@ -373,28 +310,14 @@ Configure btrbk to use `myhost_sync` as source subvolume:
snapshot_preserve_min latest
snapshot_preserve 14d 20w *m
The btrbk package provides the "btrbk-mail" script, which automates
the synchronization using rsync, and can be run as cron job or systemd
timer unit. For configuration details, see the config section in
"/contrib/cron/btrbk-mail".
This will produce daily snapshots `/mnt/btr_backup/myhost.20150101`,
with retention as defined with the snapshot_preserve option.
Alternatively, you can run any synchronization software prior to
running btrbk. Something like:
#!/bin/sh
rsync -az --delete \
--inplace --numeric-ids --acls --xattrs \
-e 'ssh -i /etc/btrbk/ssh/id_rsa' \
myhost.example.org:/data/ \
/mnt/btr_backup/myhost_sync/
exec /usr/bin/btrbk -q run
This will produce snapshots `/mnt/btr_backup/myhost.20150101`, with
retention as defined with the snapshot_preserve option.
Note that the provided script: "contrib/cron/btrbk-mail" has support
for this!
Example: Encrypted Backup to non-btrfs Target
Example: encrypted backup to non-btrfs target
---------------------------------------------
If your backup server does not support btrfs, you can send your
@ -412,7 +335,7 @@ compressed and piped through GnuPG.
raw_target_compress xz
raw_target_encrypt gpg
gpg_keyring /etc/btrbk/gpg/pubring.gpg
gpg_recipient btrbk@example.org
gpg_recipient btrbk@mydomain.com
volume /mnt/btr_pool
subvolume home
@ -421,251 +344,139 @@ compressed and piped through GnuPG.
# incremental no
This will create a GnuPG encrypted, compressed files on the target
host. For each backup, two files are created:
host:
* `/backup/home.YYYYMMDD.btrfs.xz.gpg`: main data file containing
the btrfs send-stream,
* `/backup/home.YYYYMMDD.btrfs.xz.gpg.info`: sidecar file containing
metadata used by btrbk.
* `/backup/home.YYYYMMDD.btrfs_<received_uuid>.xz.gpg` for
non-incremental images,
* `/backup/home.YYYYMMDD.btrfs_<received_uuid>@<parent_uuid>.xz.gpg`
for subsequent incremenal images.
If you are using raw _incremental_ backups, please make sure you
I you are using raw _incremental_ backups, please make sure you
understand the implications (see [btrbk.conf(5)], TARGET TYPES).
Setting up SSH
==============
Since btrbk needs root access, it is *very advisable* to take all the
security precautions you can. In most cases backups are generated
periodically without user interaction, so it is not possible to
protect your ssh key with a password. The steps below will give you
hints on how to secure your ssh server for a backup scenario. Note
that the btrbk package is not required on the remote side, but you
will need the `btrfs` executable from the [btrfs-progs] package.
Since btrbk needs root access on the remote side, it is *very
advisable* to take all the security precautions you can. Usually
backups are generated periodically without user interaction, so it is
not possible to protect your ssh key with a password. The steps below
will give you hints on how to secure your ssh server for a backup
scenario. Note that the btrbk executable is not needed on the remote
side, but you will need "/sbin/btrfs" from the btrfs-progs package.
btrbk comes with a shell script "ssh_filter_btrbk.sh", which restricts
ssh access to sane calls to the /sbin/btrfs command needed for
snapshot creation and send/receive operations (see
[ssh_filter_btrbk(1)]). Here is an example on how it can be used with
ssh:
### Create SSH Key Pair
**Step 1** (client): Create a ssh key dedicated to btrbk, without
password protection:
On the client side, create a ssh key dedicated to btrbk, without
password protection:
ssh-keygen -t rsa -b 2048 -f /etc/btrbk/ssh/id_rsa -C btrbk@mydomain.com -N ""
# ssh-keygen -t rsa -b 4096 -f /etc/btrbk/ssh/id_rsa -C btrbk@example.org -N ""
**Step 2** (server): Copy the "ssh_filter_btrbk.sh" from the btrbk
project to "/backup/scripts/".
The content of the public key (/etc/btrbk/ssh/id_rsa.pub) is used for
authentication in "authorized_keys" on the server side (see [sshd(8)]
for details).
### Allow Root Login
The most straight forward setup is to allow root login on the remote
host. If this is not an option for you, refer to the more complex
"Dedicated Btrbk User Login" section below.
/etc/ssh/sshd_config:
PermitRootLogin prohibit-password
Add your btrbk public key to "/root/.ssh/authorized_keys" on the
server, and you are good to go.
### Restrict Access
Restrict ssh access to a static IP address within your network. On the
remote host, either add a "Match" block in:
/etc/ssh/sshd_config:
Match Address 192.168.0.42
Or restrict in authorized_keys:
from="192.168.0.42" <pubkey>...
Consult the [sshd_config(5)] man-page for a detailed explanation and
more options.
Dedicated Btrbk User Login (optional)
-------------------------------------
If allowing root login is not an option for you, there are several
ways to restrict SSH access to a regular user.
### Option 1: Use sudo
On the client side, configure btrbk use the sudo backend. This changes
the ssh calls to btrfs commands to `sudo btrfs <subcommand>
<options>`.
/etc/btrbk/btrbk.conf:
backend_remote btrfs-progs-sudo
On the remote host, grant root permissions for the "btrfs" command
groups (subcommands) in "/etc/sudoers". If you are using
[ssh_filter_btrbk(1)], also add the `ssh_filter_btrbk.sh --sudo`
option in "authorized_keys" (see below).
### Option 2: Use btrfs-progs-btrbk
Instead of using the all-inclusive `btrfs` command,
"btrfs-progs-btrbk" allows you to restrict privileges to its
subcommands using linux capabilities(7) or setuid.
Note that the "btrfs-progs-btrbk" package is not available on all
linux distributions, you might need to build and install it on your
own (refer to [btrfs-progs-btrbk] on GitHub for more details).
/etc/btrbk/btrbk.conf:
backend_remote btrfs-progs-btrbk
Make sure that only the required binaries with elevated privileges can
be called by the btrbk user. For example, on a server acting as "btrbk
source", allow only the following binaries for the "btrbk" group:
# getcap /usr/bin/btrfs-*
/usr/bin/btrfs-send cap_dac_read_search,cap_fowner,cap_sys_admin=ep
/usr/bin/btrfs-subvolume-delete cap_dac_override,cap_sys_admin=ep
/usr/bin/btrfs-subvolume-list cap_dac_read_search,cap_fowner,cap_sys_admin=ep
/usr/bin/btrfs-subvolume-show cap_dac_read_search,cap_fowner,cap_sys_admin=ep
/usr/bin/btrfs-subvolume-snapshot cap_dac_override,cap_dac_read_search,cap_fowner,cap_sys_admin=ep
# ls -l /usr/bin/btrfs-*
-rwx--x--- 1 root btrbk /usr/bin/btrfs-send
-rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-delete
-rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-list
-rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-show
-rwx--x--- 1 root btrbk /usr/bin/btrfs-subvolume-snapshot
Restrict Commands with "ssh_filter_btrbk.sh" (optional)
-------------------------------------------------------
Btrbk comes with a shell script "ssh_filter_btrbk.sh", which restricts
ssh access to sane calls to the "btrfs" command needed for snapshot
creation and send/receive operations (see [ssh_filter_btrbk(1)]).
Copy "ssh_filter_btrbk.sh" to "/backup/scripts/", and configure sshd
to run it whenever the key is used for authentication. Example
"/root/.ssh/authorized_keys":
**Step 3** (server): Add contents of the public key
(/etc/btrbk/ssh/id_rsa.pub) to "/root/.ssh/authorized_keys", and
configure "ssh_filter_btrbk.sh" to be executed whenever this key is
used for authentication. Example lines:
# example backup source (also allowing deletion of old snapshots)
command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete",restrict <pubkey>...
command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete" <pubkey>...
# example backup target (also allowing deletion of old snapshots)
command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete",restrict <pubkey>...
command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete" <pubkey>...
# example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no),
# restricted to subvolumes within /home or /data
command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data",restrict <pubkey>...
command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data" <pubkey>...
You might also want to restrict ssh access to a static IP address
within your network:
from="192.168.0.42",command="/backup/scripts/ssh_filter_btrbk.sh [...]" <pubkey>...
Please refer to [ssh_filter_btrbk(1)] for a description of the
"ssh_filter_btrbk.sh" options, as well as [sshd(8)] for a description
of the "authorized_keys" file format.
Also consider setting up ssh access for a user dedicated to btrbk and
either set suid root on ssh_filter_btrbk.sh or use the "--sudo" option
and configure /etc/sudoers accordingly. For even more security, you
can setup a chroot environment in /etc/ssh/sshd_config (see
[sshd_config(5)]).
[ssh_filter_btrbk(1)]: https://digint.ch/btrbk/doc/ssh_filter_btrbk.1.html
[sshd(8)]: https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8
[sshd_config(5)]: https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config
[btrfs-progs-btrbk]: https://github.com/digint/btrfs-progs-btrbk
[ssh_filter_btrbk(1)]: http://digint.ch/btrbk/doc/ssh_filter_btrbk.html
[sshd(8)]: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8
[sshd_config(5)]: http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config.5
Restoring Backups
=================
Btrbk does not provide any mechanism to restore your backups, this has
to be done manually. In the instructions below, we assume that you
have a btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you
want to restore is at `/mnt/btr_pool/data`.
**Important**: don't use `btrfs property set` to make a subvolume
read-write after restoring. This is a low-level command, and leaves
"Received UUID" in a false state which causes btrbk to fail on
subsequent incremental backups. Instead, use `btrfs subvolume
snapshot` (without `-r` flag) as described below.
btrbk does not provide any mechanism to restore your backups, this has
to be done manually. In the examples below, we assume that you have a
btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you want to
have restored is at `/mnt/btr_pool/data`.
### Step 0: Identify Subvolume
Example: Restore a Snapshot
-----------------------------
First, pick a snapshot to be restored:
# list snapshots managed by btrbk
btrbk list snapshots
# alternative: list all subvolumes
btrbk ls /
btrbk ls -L /
From the list, identify the snapshot you want to restore. Let's say it's
From the list, pick the snapshot you want to restore. Let's say it's
`/mnt/btr_pool/_btrbk_snap/data.20150101`.
If the broken subvolume is still present, move it away:
### Step 1: Restore Backup
(skip this step if you restore from a snapshot)
# locally mounted backup disk
btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
# from / to remote host
ssh root@remote btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
btrfs send /mnt/btr_backup/data.20150101 | ssh root@remote btrfs receive /mnt/btr_pool/
**Hint**: Try to send-receive backups incrementally if possible. In
case you still have common snapshot / backup pairs (i.e. both
"snapshot_subvol" and "target_subvol" are listed above), use `btrfs
send -p <parent>`.
From this point on, `data.20150101` on both disks can be used as
parents for subsequent send-receive operations, and a *received_uuid*
relationship is established (see below).
### Step 2: Create read-write Subvolume
# if still present, move broken subvolume away
mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
# create read-write subvolume
btrfs subvolume snapshot /mnt/btr_pool/data.20150101 /mnt/btr_pool/data
Now restore the snapshot:
Your `data` subvolume is restored, you can carry on with incremental
backups to `/mnt/btr_backup`.
btrfs subvolume snapshot /mnt/btr_pool/_btrbk_snap/data.20150101 /mnt/btr_pool/data
That's it; your `data` subvolume is restored. If everything went fine,
it's time to nuke the broken subvolume:
### Step 3: Cleanup
# if everything went fine, delete the broken subvolume
btrfs subvolume delete /mnt/btr_pool/data.BROKEN
Make sure to keep `data.20150101` subvolumes on both disks at least
until you created a new backup using btrbk, in order to keep the
incremental chain alive.
Example: Restore a Backup
-------------------------
Btrfs Relationship (technical note)
-----------------------------------
First, pick a backup to be restored:
btrbk origin -t /mnt/btr_backup/data.20150101
btrbk ls -L /mnt/btr_pool /mnt/btr_backup
btrbk list backups
* **received_uuid** relationship: *correlated*, *identical*
read-only subvolumes, cross-filesystem.
From the list, pick the backup you want to restore. Let's say it's
`/mnt/btr_backup/data.20150101`.
a.received_uuid = b.received_uuid
a.received_uuid = b.uuid
If the broken subvolume is still present, move it away:
* Required for subvolumes used as parent (or clone-src) of
send-receive operations.
* Present on subvolumes created by `btrfs send | btrfs receive`.
* `/mnt/btr_pool/data.20150101 === /mnt/btr_backup/data.20150101`
mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
* **parent_uuid** relationship: "is-snapshot-of"
Now restore the backup:
a.parent_uuid = b.uuid
btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
btrfs subvolume snapshot /mnt/btr_pool/data.20150101 /mnt/btr_pool/data
btrfs subvolume delete /mnt/btr_pool/data.20150101
* Present on subvolumes created by `btrfs subvolume snapshot` or
`btrfs send -p | btrfs receive`.
* Used by btrbk to determine best parent.
* `/mnt/btr_pool/data.20150101 <-- /mnt/btr_pool/data`
Alternatively, if you're restoring data on a remote host, do something
like this:
btrfs send /mnt/btr_backup/data.20150101 | ssh root@my-remote-host.com btrfs receive /mnt/btr_pool/
If everything went fine, nuke the broken subvolume:
btrfs subvolume delete /mnt/btr_pool/data.BROKEN
FAQ
@ -675,32 +486,16 @@ Make sure to also read the [btrbk FAQ page](doc/FAQ.md).
Help improve it by asking!
Donate
======
So btrbk saved your day?
I will definitively continue to develop btrbk for free. If you want to
support my hard work with a donation, you are welcome to do so!
[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S)
Development
===========
Source Code Repository
----------------------
The source code for btrbk is managed using Git.
The source code for btrbk is managed using Git. Check out the source
like this:
Official repository:
git clone https://dev.tty0.ch/btrbk.git
Mirror on GitHub:
git clone https://github.com/digint/btrbk.git
git clone git://dev.tty0.ch/btrbk.git
How to Contribute
@ -712,21 +507,21 @@ If you would like to contribute or have found bugs:
* Visit the [btrbk project page on GitHub] and use the
[issues tracker] there.
* Talk to us on [Libera.Chat] in `#btrbk`.
* Talk to us on Freenode in `#btrbk`.
* Contact the author via email (the email address can be found in
the sources).
Any feedback is appreciated!
[btrbk project page on GitHub]: https://github.com/digint/btrbk
[issues tracker]: https://github.com/digint/btrbk/issues
[Libera.Chat]: https://libera.chat
[btrbk project page on GitHub]: http://github.com/digint/btrbk
[issues tracker]: http://github.com/digint/btrbk/issues
License
=======
btrbk is free software, available under the [GNU General Public
License, Version 3 or later][GPL-3.0-or-later].
License, Version 3][GPLv3].
[GPLv3]: http://www.gnu.org/licenses/gpl.html
[GPL-3.0-or-later]: https://www.gnu.org/licenses/gpl.html

7389
btrbk

File diff suppressed because it is too large Load Diff

View File

@ -2,33 +2,16 @@
# Example btrbk configuration file
#
#
# Please refer to the btrbk.conf(5) man-page for a complete
# description of all configuration options.
# For more examples, see README.md included with this package.
# Please refer to the btrbk.conf(5) man-page for more details.
#
# btrbk.conf(5): <https://digint.ch/btrbk/doc/btrbk.conf.5.html>
# README.md: <https://digint.ch/btrbk/doc/readme.html>
#
# Note that the options can be overridden per volume/subvolume/target
# in the corresponding sections.
# Note that the options can be overridden in the
# volume/subvolume/target sections.
#
# Enable transaction log
transaction_log /var/log/btrbk.log
# Specify SSH private key for remote connections
ssh_identity /etc/btrbk/ssh/id_ed25519
ssh_user root
# Use sudo if btrbk or lsbtr is run by regular user
backend_local_user btrfs-progs-sudo
# Enable stream buffer. Adding a buffer between the sending and
# receiving side is generally a good idea.
# NOTE: If enabled, make sure to install the "mbuffer" package!
stream_buffer 256m
# Directory in which the btrfs snapshots are created. Relative to
# <volume-directory> of the volume section.
# If not set, the snapshots are created in <volume-directory>.
@ -36,7 +19,7 @@ stream_buffer 256m
# If you want to set a custom name for the snapshot (and backups),
# use the "snapshot_name" option within the subvolume section.
#
# NOTE: btrbk does not automatically create this directory, and the
# NOTE: btrbk does not autmatically create this directory, and the
# snapshot creation will fail if it is not present.
#
snapshot_dir _btrbk_snap
@ -50,10 +33,6 @@ snapshot_dir _btrbk_snap
# creation of non-incremental backups if no parent is found).
#incremental yes
# Specify after what time (in full hours after midnight) backups/
# snapshots are considered as a daily backup/snapshot
#preserve_hour_of_day 0
# Specify on which day of week weekly/monthly backups are to be
# preserved.
#preserve_day_of_week sunday
@ -67,57 +46,51 @@ snapshot_dir _btrbk_snap
# Preserve all backup targets for a minimum period of time.
#target_preserve_min no
# Retention policy for backup targets:
# Preserve matrix for backup targets:
#target_preserve <NN>h <NN>d <NN>w <NN>m <NN>y
# Retention policy for archives ("btrbk archive" command):
#archive_preserve_min no
#archive_preserve <NN>h <NN>d <NN>w <NN>m <NN>y
# Specify SSH private key for "ssh://" volumes / targets:
#ssh_identity /etc/btrbk/ssh/id_ed25519
#ssh_user root
#ssh_port default
#ssh_compression no
#ssh_cipher_spec default
# Enable compression for remote btrfs send/receive operations:
#stream_compress no
#stream_compress_level default
#stream_compress_threads default
# Enable lock file support: Ensures that only one instance of btrbk
# can be run at a time.
#lockfile /var/lock/btrbk.lock
# Don't wait for transaction commit on deletion. Enable this to make
# sure the deletion of subvolumes is committed to disk when btrbk
# terminates.
#btrfs_commit_delete no
# Don't wait for transaction commit on deletion. Set this to "after"
# or "each" to make sure the deletion of subvolumes is committed to
# disk when btrbk terminates.
#btrfs_commit_delete no
#
# Volume section (optional): "volume <volume-directory>"
# Volume section: "volume <volume-directory>"
#
# <volume-directory> Base path within a btrfs filesystem
# containing the subvolumes to be backuped
# <volume-directory> Directory of a btrfs volume (or subvolume)
# containing the subvolume to be backuped
# (usually the mount-point of a btrfs filesystem
# mounted with subvolid=5 option).
# mounted with subvolid=0 option)
#
# Subvolume section: "subvolume <subvolume-name>"
# Subvolume section: "subvolume <subvolume-name>
#
# <subvolume-name> Subvolume to be backuped, relative to
# <volume-directory> in volume section.
#
# Target section: "target <type> <volume-directory>"
#
# <type> (optional) type, defaults to "send-receive".
# <volume-directory> Directory within a btrfs filesystem
# <type> Backup type, currently only "send-receive".
# <volume-directory> Directory of a btrfs volume (or subvolume)
# receiving the backups.
#
# NOTE: The parser does not care about indentation, this is only for
# human readability. All options apply to the last section
# human readability. The options always apply to the last section
# encountered, overriding the corresponding option of the upper
# section. This means that the global options must be set on top,
# before any "volume", "subvolume" or "target section.
# section. This means that the global options must be set before any
# "volume" section.
#
#
# Example retention policy:
# Example configuration:
#
snapshot_preserve_min 2d
snapshot_preserve 14d
@ -125,80 +98,42 @@ snapshot_preserve 14d
target_preserve_min no
target_preserve 20d 10w *m
#
# Simple setup: Backup root and home to external disk
#
snapshot_dir /btrbk_snapshots
target /mnt/btr_backup
subvolume /
subvolume /home
#
# Complex setup
#
# In order to keep things organized, it is recommended to use "volume"
# sections and mount the top-level subvolume (subvolid=5):
#
# $ mount -o subvolid=5 /dev/sda1 /mnt/btr_pool
#
# Backup to external disk mounted on /mnt/btr_backup
volume /mnt/btr_pool
# Create snapshots in /mnt/btr_pool/btrbk_snapshots
snapshot_dir btrbk_snapshots
# no action if external disk is not attached
snapshot_create ondemand
# Target for all subvolume sections:
target /mnt/btr_backup
subvolume root_gentoo
target send-receive /mnt/btr_backup/_btrbk
# Some default btrfs installations (e.g. Ubuntu) use "@" for rootfs
# (mounted at "/") and "@home" (mounted at "/home"). Note that this
# is only a naming convention.
#subvolume @
subvolume root
subvolume home
subvolume kvm
# Use different retention policy for kvm backups:
target_preserve 7d 4w
subvolume kvm
# use different preserve matrix for kvm backups
target_preserve 7d 4w
target send-receive /mnt/btr_backup/_btrbk
# Backup data to external disk as well as remote host
# Backup to external disk as well as some remote host
volume /mnt/btr_data
subvolume data
# Always create snapshot, even if targets are unreachable
snapshot_create always
target /mnt/btr_backup
target ssh://backup.my-remote-host.com/mnt/btr_backup
subvolume home
# always create snapshot, even if targets are unreachable
snapshot_create always
target send-receive /mnt/btr_backup/_btrbk
target send-receive ssh://backup.my-remote-host.com/mnt/btr_backup
# Backup from remote host, with different naming
volume ssh://my-remote-host.com/mnt/btr_pool
subvolume data_0
snapshot_dir snapshots/btrbk
snapshot_name data_main
target /mnt/btr_backup/my-remote-host.com
# Backup on demand (noauto) to remote host running busybox, login as
# regular user using ssh-agent with current user name (ssh_user no)
# and default credentials (ssh_identity no).
volume /home
noauto yes
compat busybox
backend_remote btrfs-progs-sudo
ssh_user no
ssh_identity no
target ssh://my-user-host.com/mnt/btr_backup/home
subvolume alice
subvolume bob
snapshot_dir snapshots/btrbk
snapshot_name data_main
target send-receive /mnt/btr_backup/_btrbk/my-remote-host.com
# Resume backups from remote host which runs its own btrbk instance
# creating snapshots for "home" in "/mnt/btr_pool/btrbk_snapshots".
volume ssh://my-remote-host.com/mnt/btr_pool
snapshot_dir btrbk_snapshots
snapshot_create no
snapshot_preserve_min all
subvolume home
target /mnt/btr_backup/my-remote-host.com
snapshot_dir btrbk_snapshots
snapshot_preserve_min all
snapshot_create no
target send-receive /mnt/btr_backup/_btrbk/my-remote-host.com

View File

@ -1,133 +0,0 @@
_btrbk_init_cmds()
{
# set $cmds to an array of the commands so far
#
# for example, for this command:
#
# btrbk -v --override warn_unknown_targets=yes list config --long
#
# then $cmds is:
#
# cmds=(list config)
#
cmds=()
local i
for ((i = 1; i < cword; i++)); do
case "${words[i-1]}" in
'-c' | '--config' | '--exclude' | '-l' | '--loglevel' | '--format' | '--lockfile' | '--override')
continue
;;
esac
[[ ${words[i]} != -* ]] && cmds+=(${words[i]})
done
return 0
}
_btrbk()
{
local cur prev words cword split cmds
_init_completion -s || return
_btrbk_init_cmds || return
case "$prev" in
'-c' | '--config')
_filedir
return
;;
'--exclude')
return
;;
'-l' | '--loglevel')
COMPREPLY=($(compgen -W 'error warn info debug trace' -- "$cur"))
return
;;
'--format')
COMPREPLY=($(compgen -W 'table long raw' -- "$cur"))
return
;;
'--lockfile')
_filedir
return
;;
'--override')
return
;;
esac
$split && return
if [[ $cur == -* ]]; then
COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur"))
[[ $COMPREPLY == *= ]] && compopt -o nospace
else
if [[ ! -v 'cmds[0]' ]]; then
COMPREPLY=($(compgen -W 'run dryrun snapshot resume prune archive clean stats list usage origin diff extents ls' -- "$cur"))
fi
fi
case "${cmds[0]}" in
'archive')
# <source>
if [[ ! -v 'cmds[1]' ]]; then
_filedir -d
# <target>
elif [[ ! -v 'cmds[2]' ]]; then
_filedir -d
# [--raw]
elif [[ $cur == -* ]]; then
COMPREPLY+=($(compgen -W '--raw' -- "$cur"))
fi
;;
'list')
if [[ ! -v 'cmds[1]' ]]; then
COMPREPLY=($(compgen -W 'all snapshots backups latest config source volume target' -- "$cur"))
fi
;;
'origin')
# <subvolume>
if [[ ! -v 'cmds[1]' ]]; then
_filedir -d
fi
;;
'ls')
# <path>|<url>...
_filedir -d
;;
'extents')
# [diff] <path>... [exclusive] <path>...
if [[ ! -v 'cmds[1]' ]]; then
COMPREPLY+=($(compgen -W 'diff' -- "$cur"))
elif [[ ! ${cmds[*]} =~ (^|[[:space:]])"exclusive"($|[[:space:]]) ]]; then
COMPREPLY+=($(compgen -W 'exclusive' -- "$cur"))
fi
_filedir -d
;;
esac
} && complete -F _btrbk btrbk
_lsbtr()
{
local cur prev words cword split
_init_completion -s || return
case "$prev" in
'-c' | '--config')
_filedir
;;
'--override')
;;
esac
$split && return
if [[ $cur == -* ]]; then
COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur"))
[[ $COMPREPLY == *= ]] && compopt -o nospace
else
# <path>|<url>...
_filedir -d
fi
} && complete -F _lsbtr lsbtr
# ex: filetype=bash

View File

@ -2,277 +2,152 @@
## Wrapper script running "btrbk" and sending email with results
now=$(date +%Y%m%d)
set -uf
declare -A rsync_src rsync_dst rsync_log rsync_key rsync_opt
now=$(/bin/date +%Y%m%d)
declare -A rsync_src rsync_dst rsync_log rsync_rsh rsync_opt
declare -A sync_fs_onchange
##### start config section #####
# Email recipients, separated by whitespace:
mailto=${MAILTO:-root}
# Email subject:
mail_subject_prefix="btrbk <${HOSTNAME:-localhost}>"
# Add summary and/or detail (rsync/btrbk command output) to mail body.
# If both are not set, a mail is only sent on errors.
mail_summary=yes
mail_detail=no
mailto=$MAILTO
# List of mountpoints to be mounted/unmounted (whitespace-separated)
# mount_targets="/mnt/btr_pool /mnt/backup"
mount_targets=
umount_targets=$mount_targets
# btrbk configuration file:
config="/etc/btrbk/btrbk.conf"
# Uncomment this if you only want to receive error messages:
#btrbk_opts="-q"
#skip_empty_mail=yes
# Email subject:
mail_subject_prefix="btrbk <${HOSTNAME:-localhost}>"
# rsync declarations (repeat complete block for more declarations):
rsync_src[example_data]="user@example.com:/data/"
rsync_dst[example_data]="/mnt/backup/example.com/data/"
rsync_log[example_data]="/mnt/backup/example.com/data-${now}.log"
rsync_rsh[example_data]="ssh -i /mnt/backup/ssh_keys/id_rsa"
rsync_opt[example_data]="-az --delete --inplace --numeric-ids --acls --xattrs"
# If set, add "rsync_dst" to "sync_fs" (see below) if rsync reports files transferred
#sync_fs_onchange[example_data]=yes
rsync_src[example_data]=user@example.com:/data/
rsync_dst[example_data]=/mnt/backup/example.com/data/
rsync_log[example_data]=/mnt/backup/example.com/data-${now}.log
rsync_key[example_data]=/mnt/backup/ssh_keys/id_rsa
rsync_opt[example_data]="-az --delete"
# Enable all rsync declarations (all indices of rsync_src array)
#rsync_enable=${!rsync_src[@]}
# Explicitly enable rsync declarations (whitespace-separated list)
# Enabled rsync declarations (space separated list)
#rsync_enable="example_data"
rsync_enable=
# If set, do not run btrbk if rsync reports no changes.
# If set to "quiet", do not send mail.
#skip_btrbk_if_unchanged=quiet
# Log level (1=error, 2=warn, 3=info)
loglevel=2
# Array of directories to sync(1) prior to running btrbk. This is
# useful for source subvolumes having "snapshot_create ondemand"
# configured in btrbk.conf.
#sync_fs=("/mnt/btr_data" "/mnt/btr_pool")
# btrbk command / options:
btrbk_command="run"
btrbk_opts="-c /etc/btrbk/btrbk.conf"
### Layout options:
# Prefix command output: useful when using mail clients displaying
# btrbk summary lines starting with ">>>" as quotations.
#mail_cmd_block_prefix='\\u200B' # zero-width whitespace
#mail_cmd_block_prefix=". "
# Newline character
BR=$'\n'
##### end config section #####
check_options()
{
[[ -n "$btrbk_command" ]] || die "btrbk_command is not set"
for key in $rsync_enable; do
[[ -n "${rsync_src[$key]}" ]] || die "rsync_src is not set for \"$key\""
[[ -n "${rsync_dst[$key]}" ]] || die "rsync_dst is not set for \"$key\""
[[ -n "${rsync_opt[$key]}" ]] || die "rsync_opt is not set for \"$key\""
done
}
send_mail()
{
# assemble mail subject
local subject="$mail_subject_prefix"
[[ -n "$has_errors" ]] && subject+=" ERROR";
[[ -n "$status" ]] && subject+=" - $status";
[[ -n "$xstatus" ]] && subject+=" (${xstatus:2})";
# assemble mail body
local body=
if [[ -n "$info" ]] && [[ -n "$has_errors" ]] || [[ "${mail_summary:-no}" = "yes" ]]; then
body+="$info"
fi
if [[ -n "$detail" ]] && [[ -n "$has_errors" ]] || [[ "${mail_detail:-no}" = "yes" ]]; then
[[ -n "$body" ]] && body+="${BR}${BR}DETAIL:${BR}"
body+="$detail"
fi
# skip sending mail on empty body
if [[ -z "$body" ]] && [[ -n "$has_errors" ]]; then
body+="FATAL: something went wrong (errors present but empty mail body)${BR}"
fi
[[ -z "$body" ]] && exit 0
# send mail
echo "$body" | mail -s "$subject" $mailto
if [[ $? -ne 0 ]]; then
echo "$0: Failed to send btrbk mail to \"$mailto\", dumping mail:${BR}" 1>&2
echo "<mail_subject>$subject</mail_subject>${BR}<mail_body>${BR}$body</mail_body>" 1>&2
fi
}
einfo()
{
info+="$1${BR}"
}
ebegin()
{
ebtext=$1
detail+="${BR}### $1${BR}"
}
eend()
{
if [[ $1 -eq 0 ]]; then
eetext=${3-success}
detail+="${BR}"
else
has_errors=1
eetext="ERROR (code=$1)"
[[ -n "$2" ]] && eetext+=": $2"
detail+="${BR}### $eetext${BR}"
fi
info+="$ebtext: $eetext${BR}"
return $1
}
mail_body=""
die()
{
einfo "FATAL: ${1}, exiting"
has_errors=1
send_mail
/bin/echo "$0 FATAL: $1" 1>&2
/bin/echo "$0 FATAL: exiting" 1>&2
exit 1
}
log_error() { [ $loglevel -ge 1 ] && /bin/echo "$0 ERROR: $1" 1>&2 ; }
log_warning() { [ $loglevel -ge 2 ] && /bin/echo "$0 WARNING: $1" 1>&2 ; }
log_info() { [ $loglevel -ge 3 ] && /bin/echo "$0 INFO: $1" 1>&2 ; }
run_cmd()
{
cmd_out=$("$@" 2>&1)
local ret=$?
detail+="++ ${@@Q}${BR}"
if [[ -n "${mail_cmd_block_prefix:-}" ]] && [[ -n "$cmd_out" ]]; then
detail+=$(echo -n "$cmd_out" | sed "s/^/${mail_cmd_block_prefix}/")
detail+="${BR}"
#
# mount all mountpoints listed in $mount_targets
#
for mountpoint in $mount_targets; do
$(/bin/findmnt -r -n -t btrfs $mountpoint 1>&2)
if [ $? = 0 ]; then
log_warning "btrfs filesystem already mounted: $mountpoint"
else
detail+=$cmd_out
log_info "mount $mountpoint"
$(/bin/mount --target $mountpoint 1>&2)
[ $? = 0 ] || log_error "mount failed: $mountpoint"
fi
return $ret
}
mount_all()
{
# mount all mountpoints listed in $mount_targets
mounted=""
for mountpoint in $mount_targets; do
ebegin "Mounting $mountpoint"
run_cmd findmnt -n $mountpoint
if [[ $? -eq 0 ]]; then
eend -1 "already mounted"
else
detail+="${BR}"
run_cmd mount --target $mountpoint
eend $? && mounted+=" $mountpoint"
fi
done
}
umount_mounted()
{
for mountpoint in $mounted; do
ebegin "Unmounting $mountpoint"
run_cmd umount $mountpoint
eend $?
done
}
check_options
mount_all
done
#
# run rsync for all $rsync_enable
#
for key in $rsync_enable; do
ebegin "Running rsync[$key]"
if [[ -d "${rsync_dst[$key]}" ]]; then
# There is no proper way to get a proper machine readable
# output of "rsync did not touch anything at destination", so
# we add "--info=stats2" and parse the output.
# NOTE: This also appends the stats to the log file (rsync_log).
# Another approach to count the files would be something like:
# "rsync --out-format='' | wc -l"
run_cmd rsync ${rsync_opt[$key]} \
--info=stats2 \
${rsync_log[$key]:+--log-file="${rsync_log[$key]}"} \
${rsync_rsh[$key]:+-e "${rsync_rsh[$key]}"} \
"${rsync_src[$key]}" \
"${rsync_dst[$key]}"
exitcode=$?
log_info "starting rsync: $key"
# parse stats2 (count created/deleted/transferred files)
REGEXP=$'\n''Number of created files: ([0-9]+)'
REGEXP+='.*'$'\n''Number of deleted files: ([0-9]+)'
REGEXP+='.*'$'\n''Number of regular files transferred: ([0-9]+)'
if [[ $cmd_out =~ $REGEXP ]]; then
rsync_stats="${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}"
rsync_stats_long="${BASH_REMATCH[1]} created, ${BASH_REMATCH[2]} deleted, ${BASH_REMATCH[3]} transferred"
nfiles=$(( ${BASH_REMATCH[1]} + ${BASH_REMATCH[2]} + ${BASH_REMATCH[3]} ))
else
rsync_stats_long="failed to parse stats, assuming files transferred"
rsync_stats="-1/-1/-1"
nfiles=-1
fi
[ -n "${rsync_src[$key]}" ] || die "rsync_src is not set for \"$key\""
[ -n "${rsync_dst[$key]}" ] || die "rsync_dst is not set for \"$key\""
[ -n "${rsync_log[$key]}" ] || die "rsync_log is not set for \"$key\""
[ -n "${rsync_key[$key]}" ] || die "rsync_key is not set for \"$key\""
[ -n "${rsync_opt[$key]}" ] || die "rsync_opt is not set for \"$key\""
eend $exitcode "$rsync_stats_long" "$rsync_stats_long"
xstatus+=", rsync[$key]=$rsync_stats"
rsync_header="### rsync ${rsync_opt[$key]} ${rsync_src[$key]} ${rsync_dst[$key]}"
if [[ $nfiles -ne 0 ]]; then
# NOTE: on error, we assume files are transferred
rsync_files_transferred=1
[[ -n "${sync_fs_onchange[$key]}" ]] && sync_fs+=("${rsync_dst[$key]}")
fi
if [ -d ${rsync_dst[$key]} ]; then
/bin/echo "$rsync_header" >> ${rsync_log[$key]}
ret=$(/usr/bin/rsync ${rsync_opt[$key]} --info=STATS --log-file=${rsync_log[$key]} -e "/usr/bin/ssh -i ${rsync_key[$key]}" ${rsync_src[$key]} ${rsync_dst[$key]})
if [ $? != 0 ]; then
log_error "rsync failed: $key"
ret+="\nERROR: rsync failed with exit code $?\n"
fi
mail_body+="$rsync_header$ret\n\n"
else
eend -1 "Destination directory not found, skipping: ${rsync_dst[$key]}"
ret="rsync destination directory not found for \"$key\", skipping: ${rsync_dst[$key]}"
mail_body+="$rsync_header\n$ret\n\n"
log_error "$ret"
fi
done
# honor skip_btrbk_if_unchanged (only if rsync is enabled and no files were transferred)
if [[ -n "$rsync_enable" ]] && [[ -n "$skip_btrbk_if_unchanged" ]] && [[ -z "$rsync_files_transferred" ]]; then
einfo "No files transferred, exiting"
status="No files transferred"
umount_mounted
if [[ "$skip_btrbk_if_unchanged" != "quiet" ]] || [[ -n "$has_errors" ]]; then
send_mail
fi
exit 0
fi
#
# sync filesystems in sync_fs
#
if [[ ${#sync_fs[@]} -gt 0 ]]; then
ebegin "Syncing filesystems at ${sync_fs[@]}"
run_cmd sync -f "${sync_fs[@]}"
eend $?
fi
#
# run btrbk
#
ebegin "Running btrbk"
run_cmd btrbk ${btrbk_opts:-} ${btrbk_command}
log_info "running btrbk"
ret=$(/usr/sbin/btrbk -c "$config" ${btrbk_opts:-} run 2>&1)
exitcode=$?
case $exitcode in
0) status="All backups successful"
;;
3) status="Another instance of btrbk is running, no backup tasks performed!"
;;
10) status="At least one backup task aborted!"
;;
*) status="btrbk failed with error code $exitcode"
;;
;;
10) status="ERROR: At least one backup task aborted!"
;;
*) status="ERROR: btrbk failed with error code $exitcode"
;;
esac
eend $exitcode "$status"
umount_mounted
send_mail
mail_body+=$ret
if [ "${skip_empty_mail:-no}" = "yes" ] && [ -z "$mail_body" ] && [ $exitcode -eq 0 ]; then
: # skip email sending if skip_empty_mail=yes
else
# send email
/bin/echo -e "$mail_body" | /bin/mail -s "$mail_subject_prefix - $status" $mailto
if [ $? != 0 ]; then
log_error "failed to send btrbk mail to \"$mailto\", dumping mail body:"
/bin/echo -e "$mail_body" 1>&2
fi
fi
#
# sync all mountpoints listed in $umount_targets
#
# exit on failure!
#for mountpoint in $umount_targets; do
# log_info "btrfs filesystem sync $mountpoint"
# $(/sbin/btrfs filesystem sync $mountpoint 1>&2)
# [ $? = 0 ] || die "btrfs filesystem sync failed: $mountpoint"
# sleep 1
#done
#
# unmount all mountpoints listed in $umount_targets
#
for mountpoint in $umount_targets; do
log_info "umount $mountpoint"
$(/bin/umount $mountpoint 1>&2)
[ $? = 0 ] || log_error "umount failed: $mountpoint"
done

View File

@ -1,424 +0,0 @@
#!/bin/bash
#
# NAME
#
# btrbk-verify - check latest btrbk snapshot/backup pairs
#
#
# SYNOPSIS
#
# btrbk-verify [options] <command> [filter...]
#
#
# DESCRIPTION
#
# Compare btrbk backups. Reads all files and attributes, and
# compares checksums of source and target. Uses rsync(1) as backend,
# in dry-run mode with all preserve options enabled.
#
# Resolves snapshot/backup pairs by evaluating the output of
# "btrbk list latest [filter...]". The filter argument is passed
# directly to btrbk, see btrbk(1) FILTER STATEMENTS.
#
# Restrictions:
# - ".d..t...... ./" lines are ignored by default:
# Root folder timestamp always differ.
# - "cd+++++++++ .*" lines are ignored by default:
# Nested subvolumes appear as new empty directories.
# - btrbk raw targets are skipped
# - rsync needs root in most cases (see --ssh-* options)
#
# NOTE: Depending on your setup (hardware, btrfs mount options),
# btrbk-verify may eat all your CPU power and use high bandwidth!
# Consider nice(1), ionice(1).
#
# Incomplete resource eater list:
# - rsync: checksums, heavy disk I/O
# - btrfs: decompression, encryption
# - ssh: compression, encryption
#
#
# EXAMPLES
#
# btrbk-verify latest /mnt/btr_pool
#
# Verify latest backups from targets configured in
# /etc/btrbk/btrbk.conf, matching the "/mnt/btr_pool" filter.
#
# btrbk-verify all
#
# Verify ALL backups from targets in /etc/btrbk/btrbk.conf.
# NOTE: This really re-checksums ALL files FOR EACH BACKUP,
# even if they were not touched between backups!
#
# btrbk-verify latest -n -v -v
#
# Print detailed log as well as command executed by this script,
# without actually executing rsync commands (-n, --dry-run).
#
# btrbk-verify --ssh-agent --ssh-user root --ssh-identity /etc/btrbk/ssh/id_ed25519
#
# Use "ssh -i /etc/btrbk/ssh/id_ed25519 -l root" for rsync rsh
# (override settings from btrbk.conf), start an ssh-agent(1) for
# this session and verify all latest snapshot / backups.
#
#
# SEE ALSO
#
# btrbk(1), btrbk.conf(5), rsync(1), nice(1), ionice(1)
#
#
# AUTHOR
#
# Axel Burri <axel@tty0.ch>
#
set -u
set -e
set -o pipefail
btrbk_version_min='0.32.0'
# defaults: ignore subvol dirs and root folder timestamp change
ignore_nested_subvolume_dir=1
ignore_root_folder_timestamp=1
ssh_identity=
ssh_user=
ssh_start_agent=
verbose=0
stats_enabled=
dryrun=
print_usage()
{
#80-----------------------------------------------------------------------------
cat 1>&2 <<EOF
usage: btrbk-verify [options] <command> [btrbk-list-options...] [filter...]
options:
-h, --help display this help message
-c, --config FILE specify btrbk configuration file
-n, --dry-run perform a trial run without verifying subvolumes
-v, --verbose be verbose (set twice for debug loglevel)
--stats print rsync stats to stderr (--info=stats2)
--strict treat all rsync diffs as errors
--ignore-acls ignore acls when verifying subvolumes
--ignore-xattrs ignore xattrs when verifying subvolumes
--ssh-identity FILE override ssh_identity from btrbk.conf(5) with FILE,
and clear all other ssh_* options (use with --ssh-user)
--ssh-user USER override ssh_user from btrbk.conf(5) with USER, and
clear all other ssh_* options(use with --ssh-identity)
--ssh-agent start ssh-agent(1) and add identity
commands:
latest verify most recent snapshots and backups (btrbk list latest)
all verify all snapshots and backups (btrbk list backups)
For additional information, see <https://digint.ch/btrbk/>
EOF
#80-----------------------------------------------------------------------------
exit ${1:-0}
}
list_subcommand=
btrbk_args=()
rsync_args=(-n --itemize-changes --checksum -a --delete --numeric-ids --hard-links --acls --xattrs --devices --specials)
while [[ "$#" -ge 1 ]]; do
key="$1"
case $key in
latest)
[[ -n "$list_subcommand" ]] && print_usage 2;
list_subcommand="latest"
;;
all)
[[ -n "$list_subcommand" ]] && print_usage 2;
list_subcommand="backups"
;;
-n|--dry-run)
dryrun=1
;;
--stats)
# enable rsync stats2 (transfer statistics)
rsync_args+=(--info=stats2)
stats_enabled=1
;;
--strict)
# treat all rsync diffs as errors:
# - empty directories (nested subvolumes)
# - root folder timestamp mismatch
ignore_nested_subvolume_dir=
ignore_root_folder_timestamp=
;;
--ignore-*) # --ignore-acls, --ignore-xattrs, --ignore-device, ...
# remove "--xxx" flag from rsync_args for --ignore-xxx
rsync_args=(${rsync_args[@]/"--"${key#"--ignore-"}})
;;
--ssh-identity)
# use different ssh identity (-i option) for rsync rsh.
# NOTE: this overrides all btrbk ssh_* options
ssh_identity="$2"
shift
;;
--ssh-user)
# use different ssh user (-l option) for rsync rsh
# NOTE: this overrides all btrbk ssh_* options
ssh_user="$2"
shift
;;
--ssh-agent)
ssh_start_agent=1
;;
-v|--verbose)
verbose=$((verbose+1))
btrbk_args+=("-v")
;;
-h|--help)
print_usage 0
;;
*)
# all other args are passed to btrbk (filter, -c,--config=FILE)
btrbk_args+=("$key")
;;
esac
shift
done
BR=$'\n'
log_line()
{
echo "$@" 1>&2
}
log_stats () { [[ -n "$stats_enabled" ]] && log_line "$@" ; return 0; }
log_verbose() { [[ $verbose -ge 1 ]] && log_line "$@" ; return 0; }
log_debug() { [[ $verbose -ge 2 ]] && log_line "$@" ; return 0; }
log_cmd()
{
local prefix=""
[[ -n "$dryrun" ]] && prefix="(dryrun) "
log_debug "### ${prefix}$@"
}
tlog()
{
# same output as btrbk transaction log
local status=$1
local comment=${2:-}
[[ -n "$dryrun" ]] && [[ "$status" == "starting" ]] && status="dryrun_starting"
local line="$(date --iso-8601=seconds) verify-rsync ${status} ${target} ${source} - -"
[[ -n "$comment" ]] && line="$line # $comment";
tlog_text+="$line${BR}"
log_debug "$line"
}
tlog_print()
{
# tlog goes to stdout
echo "${BR}TRANSACTION LOG${BR}---------------${BR}${tlog_text:-}"
}
# parse "rsync -i,--itemize-changes" output.
# prints ndiffs to stdout, and detailed log messages to stderr
count_rsync_diffs()
{
local nn=0
local rsync_line_match='^(...........) (.*)$'
local dump_stats_mode=
# unset IFS: no word splitting, trimming (read literal line)
while IFS= read -r rsync_line; do
local postfix_txt=""
if [[ -n "$dump_stats_mode" ]]; then
# dump_stats_mode enabled, echo to stderr
log_stats "${rsync_line}"
elif [[ "$rsync_line" == "" ]]; then
# empty line denotes start of --info=stats, enable dump_stats_mode
dump_stats_mode=1
log_stats "--- BEGIN rsync stats2 dump ---"
elif [[ "$rsync_line" =~ $rsync_line_match ]]; then
rl_flags="${BASH_REMATCH[1]}"
rl_path="${BASH_REMATCH[2]}"
if [[ -n "$ignore_root_folder_timestamp" ]] && [[ "$rsync_line" == ".d..t...... ./" ]]; then
# ignore timestamp on root folder, for some reason this does not match
postfix_txt=" # IGNORE reason=ignore_root_folder_timestamp"
elif [[ -n "$ignore_nested_subvolume_dir" ]] && [[ "$rl_flags" == "cd+++++++++" ]]; then
# nested subvolumes appear as new empty directories ("cd+++++++++") in rsync (btrfs bug?)
postfix_txt=" # IGNORE reason=ignore_nested_subvolume_dir"
else
nn=$((nn+1))
postfix_txt=" # FAIL ndiffs=$nn"
fi
log_verbose "[rsync] ${rsync_line}${postfix_txt}"
else
nn=$((nn+1))
log_line "btrbk-verify: ERROR: failed to parse rsync line: ${rsync_line}"
fi
done
[[ -n "$dump_stats_mode" ]] && log_stats "--- END rsync stats2 dump ---"
echo $nn
return 0
}
rsync_rsh()
{
# btrbk v0.27.0 sets source_rsh="ssh [flags...] ssh_user@ssh_host"
# this returns "ssh [flags...] -l ssh_user"
local rsh=$1
local rsh_match="(.*) ([a-z0-9_-]+)@([a-zA-Z0-9.-]+)$"
if [[ -z "$rsh" ]]; then
return
elif [[ -n "$ssh_user" ]] || [[ -n "$ssh_identity" ]]; then
# override btrbk.conf from command line arguments
log_debug "Overriding all ssh_* options from btrbk.conf"
local cmd="ssh -q"
[[ -n "$ssh_identity" ]] && cmd="$cmd -i '$ssh_identity'"
[[ -n "$ssh_user" ]] && cmd="$cmd -l '$ssh_user'"
echo "$cmd"
elif [[ $rsh =~ $rsh_match ]]; then
echo "${BASH_REMATCH[1]} -l ${BASH_REMATCH[2]}"
else
log_line "btrbk-verify: ERROR: failed to parse source_rsh: $rsh"
exit 1
fi
}
kill_ssh_agent()
{
echo "Stopping SSH agent"
eval `ssh-agent -k`
}
start_ssh_agent()
{
if [[ -z "$ssh_identity" ]]; then
log_line "btrbk-verify: ERROR: no SSH identity specified for agent"
print_usage 2
fi
echo "Starting SSH agent"
eval `ssh-agent -s`
ssh_agent_running=1
trap 'exit_trap_action' EXIT
ssh-add "$ssh_identity"
}
eval_btrbk_resolved_line()
{
local line=" $1"
local prefix=$2
local required_keys=$3
# reset all variables first
for vv in $required_keys; do
eval "${prefix}${vv}="
done
for vv in $required_keys; do
# basic input validation, set prefixed variable (eval)
local match=" ${vv}='([^']*('\\\\''[^']*)*)'"
if [[ $line =~ $match ]] ; then
eval "${prefix}${vv}='${BASH_REMATCH[1]}'" || return 1
else
log_line "btrbk-verify: ERROR: Missing variable \"${vv}\""
return 1
fi
done
}
exit_trap_action()
{
[[ -n "${ssh_agent_running:-}" ]] && kill_ssh_agent
[[ $verbose -gt 0 ]] && tlog_print
}
# start ssh-agent(1)
[[ -n "$ssh_start_agent" ]] && start_ssh_agent
# run "btrbk list"
[[ -z "$list_subcommand" ]] && print_usage 2
log_verbose "Resolving btrbk $list_subcommand"
btrbk_cmd=("btrbk" "list" "$list_subcommand" "--format=raw" "-q" "${btrbk_args[@]}")
log_debug "### ${btrbk_cmd[@]}"
btrbk_list=$("${btrbk_cmd[@]}")
btrbk_list_exitstatus=$?
if [[ $btrbk_list_exitstatus -ne 0 ]]; then
log_line "btrbk-verify: ERROR: Command execution failed (status=$btrbk_list_exitstatus): ${btrbk_cmd[@]}"
exit 1
fi
log_debug "--- BEGIN btrbk list $list_subcommand ---"
log_debug "$btrbk_list"
log_debug "--- END btrbk list $list_subcommand ---"
tlog_text=""
exitstatus=0
# trap on EXIT (includes all signals)
trap 'exit_trap_action' EXIT
while read -r btrbk_list_line; do
# set R_xxx variables from format=raw line (table format "resolved")
log_debug "Evaluating [btrbk list] line: $btrbk_list_line"
[[ -z "$btrbk_list_line" ]] && continue
if ! eval_btrbk_resolved_line "$btrbk_list_line" \
"R_" "snapshot_subvolume target_subvolume source_host target_host target_type source_rsh target_rsh"
then
log_line "btrbk-verify: ERROR: Parse error of command output: ${btrbk_cmd[@]}"
log_line "Make sure to have >=btrbk-${btrbk_version_min} installed!"
exitstatus=1
break
fi
source="${R_snapshot_subvolume}/"
target="${R_target_subvolume}/"
[[ -n "$R_source_host" ]] && source="${R_source_host}:${source}"
[[ -n "$R_target_host" ]] && target="${R_target_host}:${target}"
if [[ -z "$R_snapshot_subvolume" ]]; then
log_line "WARNING: Skipping task (missing snapshot): target=$target"
elif [[ -z "$R_target_subvolume" ]]; then
log_line "Skipping task (no target): source=$source"
elif [[ "$R_target_type" != "send-receive" ]]; then
log_line "Skipping task (target_type=$R_target_type): source=$source, target=$target"
elif [[ -n "$R_source_rsh" ]] && [[ -n "$R_target_rsh" ]]; then
log_line "WARNING: Skipping task (SSH for both source and target is not supported): target=$target"
else
log_line "Comparing [rsync] $source $target"
# rsync rsh is either source_rsh or target_rsh or empty
eff_rsh="$R_source_rsh"
[[ -z "$eff_rsh" ]] && eff_rsh="$R_target_rsh"
eff_rsh=$(rsync_rsh "$eff_rsh")
rsync_cmd=("rsync" "${rsync_args[@]}")
[[ -n "$eff_rsh" ]] && rsync_cmd+=("-e" "$eff_rsh")
rsync_cmd+=("${source}" "${target}")
log_cmd "${rsync_cmd[@]}"
[[ -n "$dryrun" ]] && rsync_cmd=("cat" "/dev/null")
#rsync_cmd=("echo" '........... SHOULD/FAIL/'); # simulate failure
#rsync_cmd=("echo" 'cd+++++++++ SHOULD/IGNORE/'); # simulate ignored
# execute rsync
tlog "starting"
set +e
ndiffs=$("${rsync_cmd[@]}" | count_rsync_diffs)
rsync_exitstatus=$?
set -e
if [[ $rsync_exitstatus -ne 0 ]] || [[ -z "$ndiffs" ]]; then
log_line "btrbk-verify: ERROR: Command execution failed (status=$rsync_exitstatus): ${rsync_cmd[@]}"
tlog "ERROR"
exitstatus=10
elif [[ $ndiffs -gt 0 ]]; then
log_line "VERIFY FAIL (ndiffs=$ndiffs): ${source} ${target}"
tlog "fail" "ndiffs=$ndiffs"
exitstatus=10
else
log_verbose "Compare success (ndiffs=$ndiffs)"
tlog "success"
fi
fi
done <<< "$btrbk_list"
#done < <(echo "$btrbk_list") # more posix'ish
# NOTE: this triggers exit_trap_action()
exit $exitstatus

View File

@ -1,60 +0,0 @@
#!/usr/bin/env python3
#
# kdf_pbkdf2.py - (kdf_backend for btrbk)
#
# Copyright (c) 2017 Axel Burri
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# ---------------------------------------------------------------------
# The official btrbk website is located at:
# https://digint.ch/btrbk/
#
# Author:
# Axel Burri <axel@tty0.ch>
# ---------------------------------------------------------------------
import sys
import os
import getpass
import hashlib
def passprompt():
pprompt = lambda: (getpass.getpass("Passphrase: "), getpass.getpass("Retype passphrase: "))
p1, p2 = pprompt()
while p1 != p2:
print("No match, please try again", file=sys.stderr)
p1, p2 = pprompt()
return p1
if len(sys.argv) <= 1:
print("Usage: {} <dklen>".format(sys.argv[0]), file=sys.stderr)
sys.exit(1)
hash_name = "sha256"
iterations = 300000
dklen = int(sys.argv[1])
salt = os.urandom(16)
password = passprompt().encode("utf-8")
dk = hashlib.pbkdf2_hmac(hash_name=hash_name, password=password, salt=salt, iterations=iterations, dklen=dklen)
salt_hex = "".join(["{:02x}".format(x) for x in salt])
dk_hex = "".join(["{:02x}".format(x) for x in dk])
print("KEY=" + dk_hex);
print("algorithm=pbkdf2_hmac");
print("hash_name=" + hash_name);
print("salt=" + salt_hex);
print("iterations=" + str(iterations));

View File

@ -1,181 +0,0 @@
#!/usr/bin/perl
#
# raw_suffix2sidecar - migrate to btrbk raw target sidecar files
#
# Copyright (C) 2017 Axel Burri
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# ---------------------------------------------------------------------
# The official btrbk website is located at:
# https://digint.ch/btrbk/
#
# Author:
# Axel Burri <axel@tty0.ch>
# ---------------------------------------------------------------------
# Create raw sidecar ".info" files from uuid-suffixed raw backup files
# generated by btrbk < v0.26.0.
use strict;
use warnings FATAL => qw( all );
use Getopt::Long qw(GetOptions);
our $VERSION = '0.26.0'; # match btrbk version
our $AUTHOR = 'Axel Burri <axel@tty0.ch>';
our $PROJECT_HOME = '<https://digint.ch/btrbk/>';
my $VERSION_INFO = "raw_suffix2sidecar (btrbk migration script), version $VERSION";
my $compress_format_alt = 'gz|bz2|xz|lzo|lz4';
my $file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/; # note: ubuntu uses '@' in the subvolume layout: <https://help.ubuntu.com/community/btrfs>
my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/;
my $timestamp_postfix_match = qr/\.(?<YYYY>[0-9]{4})(?<MM>[0-9]{2})(?<DD>[0-9]{2})(T(?<hh>[0-9]{2})(?<mm>[0-9]{2})((?<ss>[0-9]{2})(?<zz>(Z|[+-][0-9]{4})))?)?(_(?<NN>[0-9]+))?/; # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]"
my $raw_postfix_match = qr/--(?<received_uuid>$uuid_match)(\@(?<parent_uuid>$uuid_match))?\.btrfs?(\.(?<compress>($compress_format_alt)))?(\.(?<encrypt>gpg))?(\.(?<split>split_aa))?(\.(?<incomplete>part))?/; # matches ".btrfs_<received_uuid>[@<parent_uuid>][.gz|.bz2|.xz|...][.gpg][.split_aa][.part]"
my $dryrun;
my %raw_info_sort = (
TYPE => 1,
FILE => 2,
RECEIVED_UUID => 3,
RECEIVED_PARENT_UUID => 4,
INCOMPLETE => 5,
compress => 9,
split => 10,
encrypt => 11,
);
sub VERSION_MESSAGE
{
print STDERR $VERSION_INFO . "\n\n";
}
sub HELP_MESSAGE
{
print STDERR "usage: raw_suffix2sidecar <dir>...\n";
print STDERR "\n";
print STDERR "options:\n";
# "--------------------------------------------------------------------------------"; # 80
print STDERR " -h, --help display this help message\n";
print STDERR " --version display version information\n";
print STDERR " -n, --dry-run perform a trial run with no changes made\n";
print STDERR "\n";
print STDERR "For additional information, see $PROJECT_HOME\n";
}
sub write_raw_info($$)
{
my $file = shift // die;
my $raw_info = shift // die;
my $info_file = $file . '.info';
my @line;
push @line, "#raw_suffix2sidecar-v$VERSION";
push @line, "# Do not edit this file";
# sort by %raw_info_sort, then by key
foreach(sort { (($raw_info_sort{$a} || 99) <=> ($raw_info_sort{$b} || 99)) || ($a cmp $b) } keys %$raw_info) {
push @line, ($_ . '=' . $raw_info->{$_}) if($raw_info->{$_});
}
print "Creating info file: $info_file\n";
unless($dryrun) {
open (INFOFILE, ">> $info_file") || die "Failed to open $info_file";
print INFOFILE join("\n", @line) . "\n";
close(INFOFILE);
}
return $info_file;
}
MAIN:
{
Getopt::Long::Configure qw(gnu_getopt);
unless(GetOptions(
'help|h' => sub { VERSION_MESSAGE(); HELP_MESSAGE(0); exit 0; },
'version' => sub { VERSION_MESSAGE(); exit 0; },
'dry-run|n' => \$dryrun,
))
{
VERSION_MESSAGE();
HELP_MESSAGE(0);
exit 2;
}
unless(@ARGV) {
VERSION_MESSAGE();
HELP_MESSAGE();
exit 1;
}
foreach my $target_dir (@ARGV) {
$target_dir =~ s/\/+$//;
print "Processing directory: $target_dir/\n";
opendir(my($dh), $target_dir) || die "Failed to open directory '$target_dir': $!";
my @files = readdir($dh);
closedir $dh;
my @splitfiles = @files;
foreach my $file (@files) {
if($file =~ /^(?<basename>$file_match$timestamp_postfix_match)$raw_postfix_match$/) {
print "\nProcessing raw backup: $file\n";
my $newname = $+{basename} || die;
my %raw_info = (
TYPE => 'raw',
RECEIVED_UUID => $+{received_uuid},
RECEIVED_PARENT_UUID => $+{parent_uuid},
INCOMPLETE => $+{incomplete} ? 1 : 0,
compress => $+{compress},
split => ($+{split} ? (-s $file) : undef), # file size
encrypt => $+{encrypt},
);
die "Missing received uuid in file: $file" unless $raw_info{RECEIVED_UUID};
$newname .= '.btrfs';
$newname .= '.' . $raw_info{compress} if($raw_info{compress});
$newname .= '.' . $raw_info{encrypt} if($raw_info{encrypt});
$raw_info{FILE} = $newname;
write_raw_info("$target_dir/$newname", \%raw_info);
if($raw_info{split}) {
my $sfile = $file;
$sfile =~ s/_aa$//; # we match on ".split_aa" above
foreach my $splitfile (@splitfiles) {
if($splitfile =~ /^${sfile}(_[a-z]+)$/) {
my $suffix = $1 // die;
print "Renaming file: $target_dir/$splitfile -> $target_dir/$newname.split$suffix\n";
unless($dryrun) {
rename("$target_dir/$splitfile", "$target_dir/$newname.split$suffix") || die "Failed to rename file: $target_dir/$splitfile -> $target_dir/${newname}.split$suffix: $!";
}
}
}
}
else {
print "Renaming file: $target_dir/$file -> $target_dir/$newname\n";
unless($dryrun) {
rename("$target_dir/$file", "$target_dir/$newname") || die "Failed to rename file: $target_dir/$file -> $target_dir/$newname";
}
}
}
}
}
if($dryrun) {
print "\nNOTE: Dryrun was active, none of the operations above were actually executed!\n";
}
}
1;

View File

@ -1,7 +1,6 @@
[Unit]
Description=btrbk backup
Documentation=man:btrbk(1)
[Service]
Type=oneshot
ExecStart=@BINDIR@/btrbk run
ExecStart=/usr/sbin/btrbk run

View File

@ -7,4 +7,4 @@ AccuracySec=10min
Persistent=true
[Install]
WantedBy=timers.target
WantedBy=multi-user.target

View File

@ -1,217 +0,0 @@
#!/usr/bin/env python3
import os
import logging
import subprocess
import argparse
logger = logging.getLogger(__name__)
class TransformProcess:
def run(self, bfile, options, **kw):
return subprocess.Popen(self.get_cmd(bfile, options), **kw)
def get_cmd(self, bfile, options):
raise NotImplementedError()
@classmethod
def add_parser_options(cls, parser):
pass
class TransformOpensslDecrypt(TransformProcess):
@staticmethod
def get_cmd(bfile, options):
return [
'openssl', 'enc', '-d', '-' + bfile.info['cipher'], '-K',
open(options.openssl_keyfile, 'r').read(), '-iv', bfile.info['iv']
]
@staticmethod
def add_parser_options(parser):
parser.add_argument('--openssl-keyfile', help="path to private encryption key file")
class TransformDecompress(TransformProcess):
def __init__(self, program):
self.p = program
def get_cmd(self, bfile, options):
return [self.p, '-d']
class TransformBtrfsReceive(TransformProcess):
@classmethod
def run(cls, bfile, options, **kw):
return subprocess.Popen(cls.get_cmd(bfile, options), **kw)
@staticmethod
def get_cmd(bfile, options):
return ['btrfs', 'receive', options.restore_dir]
TRANSFORMERS = (
TransformOpensslDecrypt, TransformDecompress, TransformBtrfsReceive
)
class BtrfsPipeline:
def __init__(self, bfile):
self.bfile = bfile
self.processors = []
def append(self, transformer):
self.processors.append(transformer)
def run(self, options):
processes = []
with open(self.bfile.data_file, 'rb') as next_input:
for transformer in self.processors:
process = transformer.run(
self.bfile, options,
stdin=next_input, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
next_input = process.stdout
processes.append(process)
btrfs_process = TransformBtrfsReceive.run(
self.bfile, options, stdin=next_input,
stderr=subprocess.PIPE, stdout=subprocess.DEVNULL)
processes.append(btrfs_process)
# warning: the code below is pretty ugly and hacky
terminated = 0
while terminated < len(processes):
for p in processes:
if p.returncode is not None:
continue
msg = None
try:
p.wait(timeout=1)
except subprocess.TimeoutExpired as e:
pass
except Exception as e:
msg = e
else:
msg = p.stderr.read().decode('utf-8').strip()
finally:
if p.returncode is not None:
terminated += 1
if p.returncode != 0:
for p_other in processes:
p_other.terminate()
terminated += 1
if msg:
logger.error(f"error running {p.args}: {msg}")
def get_cmd(self, options):
command_pipe = [['cat', self.bfile.data_file]]
for transformer in self.processors:
command_pipe.append(transformer.get_cmd(self.bfile, options))
command_pipe.append(TransformBtrfsReceive.get_cmd(self.bfile, options))
return ' | '.join(' '.join(x) for x in command_pipe)
class BackupFile:
def __init__(self, path):
assert path.endswith('.info')
self.info_file = path
self.info = self._parse_info()
self.uuid = self.info['RECEIVED_UUID']
self.data_file = os.path.join(os.path.dirname(path), self.info['FILE'])
self.parent = self.info.get('RECEIVED_PARENT_UUID')
self.is_restored = False
def _parse_info(self):
config = {}
with open(self.info_file, 'r') as fh:
# skip command option line
for line in fh.readlines():
if '=' not in line:
continue
key, val = line.strip().split('=', maxsplit=1)
config[key] = val
return config
def get_transformers(self):
if 'encrypt' in self.info:
if self.info['encrypt'] == 'gpg':
raise NotImplementedError('gpg encryption')
elif self.info['encrypt'] == 'openssl_enc':
yield TransformOpensslDecrypt()
else:
raise Exception(f'unknown encryption type: "{self.info["encrypt"]}"')
if 'compress' in self.info:
yield TransformDecompress(self.info['compress'])
def restore_file(self, options):
assert self.info.get('TYPE') == 'raw'
assert not self.info.get('INCOMPLETE')
pipeline = BtrfsPipeline(self)
for transformer in self.get_transformers():
pipeline.append(transformer)
if options.dry_run:
print(pipeline.get_cmd(options))
else:
logger.info(f"restoring backup {os.path.basename(self.data_file)}")
pipeline.run(options)
self.is_restored = True
def restore_from_path(backup, options):
path = os.path.dirname(backup)
info_files = {}
backup_file = BackupFile(backup + '.info')
restored_files = set()
for entry in os.scandir(path):
if entry.is_file() and entry.name.endswith('.info'):
info = BackupFile(entry.path)
info_files[info.uuid] = info
restored_files.update(restore_backup(backup_file, info_files, options))
logger.info(f"finished; restored {len(restored_files)} backup files")
def restore_backup(bfile, parents, options):
if bfile.is_restored:
return
if bfile.parent:
parent = parents.get(bfile.parent)
if not parent:
msg = (f"missing parent {bfile.parent} for"
f"'{os.path.basename(bfile.info_file)}'")
if options.ignore_missing:
logger.warning(msg)
else:
raise Exception(msg)
else:
yield from restore_backup(parent, parents, options)
bfile.restore_file(options)
yield bfile.uuid
def main():
parser = argparse.ArgumentParser(description="restore btrbk raw backup")
parser.add_argument('backup', help="backup file to restore; for incremental"
" backups the parent files must be in the same directory")
parser.add_argument('restore_dir', help="target directory for restored subvolumes"
" (path argument for \"btrfs receive\")")
parser.add_argument('-n', '--dry-run', action='store_true',
help="print commands that would be executed")
parser.add_argument('--ignore-missing', action='store_true',
help="do not fail on missing parent snapshots")
for transformer in TRANSFORMERS:
transformer.add_parser_options(parser)
args = parser.parse_args()
if args.dry_run:
logger.setLevel('ERROR')
restore_from_path(args.backup, args)
if __name__ == '__main__':
logger.setLevel('INFO')
logging.basicConfig(format='%(asctime)s %(levelname)s - %(message)s')
main()

2
doc/.gitignore vendored
View File

@ -1,2 +0,0 @@
*.[1-8]
*.[1-8].gz

View File

@ -19,38 +19,38 @@ FILTER STATEMENTS), which means you can e.g. add "group automount"
tags in your configuration and dump only the volumes of this group:
`btrbk list volume automount`.
[btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
[btrbk(1)]: http://digint.ch/btrbk/doc/btrbk.html
How can I setup a debian pre-install hook?
------------------------------------------
Create a file `/etc/apt/apt.conf.d/70btrbk`, e.g.:
// create a btrfs snapshot before (un)installing packages
Dpkg::Pre-Invoke {"/usr/bin/btrbk run /mnt/btr_pool/rootfs";};
In order to make sure that the snapshots are always generated and
nothing is deleted, add the btrbk command line options `--preserve
--override=snapshot_create=always`.
Why is "subvolume ." configuration not recommended?
Why is it not possible to backup '/' (btrfs root) ?
---------------------------------------------------
Referring to a btrbk configuration like this:
or in other words: why does this config not work:
/etc/btrbk/btrbk.conf:
volume /
subvolume .
subvolume /
snapshot_name rootfs
Btrbk is designed to operate on the subvolumes *within* a root
subvolume. In the config above, the btrbk snapshots would be created
*inside* the source subvolume, altering it (from user perspective).
From btrfs perspective this is not a problem, as the snapshots are
separate subvolumes referring to the source subvolume and mapped into
the file system tree below the source subvolume.
*ERROR: Only relative files allowed for option "subvolume"*.
### Answer
btrbk is designed to never alter your source subvolume. In the config
above, the btrbk snapshots would be created *inside* the source
subvolume, altering it.
The same applies to **any "btrfs root" mount point** (subvolid=0). In
the example below, you will **not be able to backup** `/mnt/data`
using btrbk:
/etc/fstab:
/dev/sda1 /mnt/data btrfs subvolid=0 [...]
btrbk is designed to operate on the subvolumes *within* `/mnt/data`.
The recommended way is to split your data into subvolumes, e.g.:
# btrfs subvolume create /mnt/data/www
@ -73,6 +73,23 @@ The btrbk configuration for this would be:
[...]
### Tech Answer
While *btrfs root* (subvolid=0) is a regular subvolume, it is still
special: being the root node, it does not have a "name" inside the
subvolume tree.
Here, `/mnt/btr_pool` is mounted with `subvolid=0`:
# btrfs sub show /mnt/btr_pool/
/mnt/btr_pool is btrfs root
# btrfs sub show /mnt/btr_pool/rootfs
/mnt/btr_pool/rootfs
Name: rootfs
uuid: [...]
How should I organize my btrfs filesystem?
------------------------------------------
@ -88,7 +105,7 @@ a good entry point.
If your linux root filesystem is btrfs, I recommend booting linux from
a btrfs subvolume, and use the btrfs root only as a container for
subvolumes (i.e. NOT booting from "subvolid=5"). This has the big
subvolumes (i.e. NOT booting from "subvolid=0"). This has the big
advantage that you can choose the subvolume in which to boot by simply
switching the `rootflags=subvol=<subvolume>` kernel boot option.
@ -107,15 +124,15 @@ have to create a run-time (rw) snapshot before booting into it:
# btrfs subvolume snapshot /mnt/btr_pool/backup/btrbk/rootfs-20150101 /mnt/btr_pool/rootfs_testing
How do I convert '/' (subvolid=5) into a subvolume?
How do I convert '/' (subvolid=0) into a subvolume?
---------------------------------------------------
There's several ways to achieve this, the solution described below
guarantees not to create new files (extents) on disk.
There's several ways to achieve this, the solution described below is
that it guarantees not to create new files (extents) on disk.
### Step 1: make a snapshot of your root filesystem
Assuming that '/' is mounted with `subvolid=5`:
Assuming that '/' is mounted with `subvolid=0`:
# btrfs subvolume snapshot / /rootfs
@ -123,17 +140,13 @@ Note that this command does NOT make any physical copy of the files of
your subvolumes within "/", it will only add some metadata.
### Step 2: (optional) add the toplevel subvolume to fstab
### Step 2: make sure that "/rootfs/etc/fstab" is ok.
Add mount point for subvolid=5 to fstab, something like this:
Add mount point for subvolid=0 to fstab, something like this:
/rootfs/etc/fstab:
/dev/sda1 /mnt/btr_pool btrfs subvolid=5,noatime 0 0
> This step is not critical for a proper root change, but will save
> your time by preventing further configurations/reboots and manually
> mounting the toplevel subvolume.
/dev/sda1 /mnt/btr_pool btrfs subvolid=0,noatime 0 0
### Step 3: boot from the new subvolume "rootfs".
@ -143,20 +156,10 @@ Either add `rootflags=subvol=rootfs` to grub.cfg, or set subvolume
# btrfs subvolume set-default <subvolid> /
You can obtain `<subvolid>` via
`btrfs subvolume show /rootfs | grep "Subvolume ID"`
> Editing grub.cfg manually may lead you some troubles if you perform
> some actions that will fire `grub-mkconfig`.
### Step 4: after reboot, check if everything went fine:
First check your **system log** for btrfs errors:
cat /var/log/messages | grep -i btrfs | grep -i error
then check if current `/` is our new subvolume:
First check your **system log** for btrfs errors, then:
# btrfs subvolume show /
Name: rootfs
@ -164,34 +167,34 @@ then check if current `/` is our new subvolume:
Great, this tells us that we just booted into our new snapshot!
# mount /mnt/btr_pool
# btrfs subvolume show /mnt/btr_pool
/mnt/btr_pool is toplevel subvolume
/mnt/btr_pool is btrfs root
This means that the root volume (subvolid=5) is correctly mounted.
This means that the root volume (subvolid=0) is correctly mounted.
### Step 5: delete old (duplicate) files
Carefully delete all old files from `/mnt/btr_pool`, except "rootfs"
and any other subvolumes within "/mnt/btr_pool". In other words,
delete any folders that are NOT LISTED by `btrfs subvolume list -a
/mnt/btr_pool`:
and all other subvolumes within "/". You can list all these by typing:
# btrfs subvolume list -a /mnt/btr_pool
Make sure you do NOT delete anything within the directories listed
here!
something like:
# cd /mnt/btr_pool
# mkdir TO_BE_REMOVED
# mv bin sbin usr lib var ... TO_BE_REMOVED
Then reboot. If everything went fine, remove the directory:
# cd /mnt/btr_pool
# rm -rf TO_BE_REMOVED
# rm -rf bin sbin usr lib var ...
What is the most efficient way to clone btrfs storage?
------------------------------------------------------
It is very common (and avisable!) to keep backups on a separate
location. In some situations, it is also required to transport the
location. In some situations, is is also required to transport the
data physically, either to the datacenter or to your safe in the
basement.
@ -205,6 +208,12 @@ effect, this also detects possible read-errors on your backup targets
See **btrbk archive** command in [btrbk(1)] for more details.
**Note that kernels >=4.1 and <4.4 have a bug when re-sending
subvolumes**, make sure you run a recent/patched kernel or step 3 will
fail. Read
[this thread on gmane](http://thread.gmane.org/gmane.comp.file-systems.btrfs/48798)
(the patch provided is confirmed working on kernels 4.2.x and 4.3.x).
### Answer 2: Use external storage as "stream-fifo"
@ -224,83 +233,3 @@ This approach has the advantage that you don't need to reformat your
USB disk. This works fine, but be aware that you may run into trouble
if a single stream gets corrupted, making all subsequent streams
unusable.
### Warning: Avoid using "dd" on btrfs filesystems!
If you use `dd` (e.g. in order to clone a partition), make sure you
don't mount the cloned filesystem at the same time as the original
one. You will end up having multiple filesystems **sharing identical
UUID**, which will break things. If you _really_ want to do this, make
sure to run:
btrfstune -u /dev/sdaX
which changes the UUID of the given device. Note that the btrfs
subvolumes still share identical UUID's, but at least the kernel can
cope with it (see
[this post on stackexchange](https://unix.stackexchange.com/questions/246976/btrfs-subvolume-uuid-clash)
).
Btrbk on the other hand relies on subvolume UUID's being *universally
unique*, and uses them as hash keys for identifying and caching
filesystem and subvolume trees, which leads to undefined behavior if
multiple identical UUID's are processed.
I'm getting an error: Aborted: "Received UUID" is set
-----------------------------------------------------
You probably restored a backup with send-receive, and made it
read/write using `btrfs property set`. This is bad, as all snapshots
and backups will inherit this identical "Received UUID", which results
in all these subvolumes will be treated as "containing same data".
To fix this, create a "proper" snapshot:
# cd /mnt/btr_pool
# mv mysubvolume mysubvolume.broken
# btrfs subvolume snapshot mysubvolume.broken mysubvolume
Now, `mysubvolume` should have an empty "Received UUID". Note that in
order to have a clean environment, you also need to fix all subvolumes
(snapshots as well as backups) that you created with the broken
subvolume.
Check if there are more broken subvolumes:
# btrfs subvolume show mysubvolume.broken
# btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
# btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>
Either delete them (they won't be used for incremental send-receive
anyways), or clean them as follows:
# btrfs subvolume snapshot listed_ro_subvol listed_ro_subvol.rw
# btrfs subvolume delete listed_ro_subvol
# btrfs subvolume snapshot -r listed_ro_subvol.rw listed_ro_subvol
# btrfs subvolume delete listed_ro_subvol.rw
Finally, don't forget to delete the broken source subvolume:
# btrfs subvolume delete mysubvolume.broken
You should now have a clean environment, and btrbk will not complain
any more.
I'm getting an error: Aborted: subvolume has no UUID
----------------------------------------------------
If your file system was created with btrfs-progs < 4.16, the btrfs
root subvolume (id=5) has no UUID. You can check this by calling:
# btrfs subvolume show /mnt/btr_pool
/
Name: <FS_TREE>
UUID: -
[...]
Without a UUID, the snapshots would get no parent_uuid, leaving btrbk
unable to track parent/child relationships. In this case, btrbk
refuses to create snapshots and backups.

View File

@ -1,69 +0,0 @@
DOCS = FAQ.md
MAN_MAN1 = btrbk.1 \
lsbtr.1 \
ssh_filter_btrbk.1
MAN_MAN5 = btrbk.conf.5
PN = btrbk
PREFIX ?= /usr
DOCDIR = $(PREFIX)/share/doc/$(PN)
MAN1DIR = $(PREFIX)/share/man/man1
MAN5DIR = $(PREFIX)/share/man/man5
ifeq ($(COMPRESS), yes)
DOCS := $(addsuffix .gz,$(DOCS))
MAN_MAN1 := $(addsuffix .gz,$(MAN_MAN1))
MAN_MAN5 := $(addsuffix .gz,$(MAN_MAN5))
endif
# convert using "asciidoctor": <https://asciidoctor.org>
# fallback to "a2x" from asciidoc package: <http://asciidoc.org>
ifneq (, $(shell command -v asciidoctor 2> /dev/null))
ASCIIDOC_MANPAGE = asciidoctor -d manpage -b manpage
ASCIIDOC_HTML = asciidoctor -b html5 -d article
else ifneq (, $(shell command -v a2x 2> /dev/null))
# NOTE: using -L (--no-xmllint), as xmllint is a separate package on many distros.
ASCIIDOC_MANPAGE = a2x -L -d manpage -f manpage
ASCIIDOC_HTML = asciidoc -b html -d article
else
ASCIIDOC_ERR = $(error "please install either asciidoc or asciidoctor")
ASCIIDOC_MANPAGE = $(ASCIIDOC_ERR)
ASCIIDOC_HTML = $(ASCIIDOC_ERR)
endif
# reproducible builds: reference date is ":date:" attribute from asciidoc source
date_attr = $(shell sed -rn 's/:date:\s*//p' $(1))
source_date_epoch = $(shell date +%s -u -d $(call date_attr,$(1)))
all: man
man: man1 man5
man1: $(MAN_MAN1)
man5: $(MAN_MAN5)
install: install-man install-doc
install-man: man
install -d -m 755 "$(DESTDIR)$(MAN1DIR)"
install -d -m 755 "$(DESTDIR)$(MAN5DIR)"
install -p -m 644 $(MAN_MAN1) "$(DESTDIR)$(MAN1DIR)"
install -p -m 644 $(MAN_MAN5) "$(DESTDIR)$(MAN5DIR)"
install-doc: $(DOCS)
install -d -m 755 "$(DESTDIR)$(DOCDIR)"
install -p -m 644 $(DOCS) "$(DESTDIR)$(DOCDIR)"
clean:
rm -f *.md.gz *.[15] *.[15].{gz,html}
%.gz : %
gzip -9 -n -c $< > $@
%.1 : %.1.asciidoc
SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $<
%.5 : %.5.asciidoc
SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $<
%.html : %.asciidoc
SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_HTML) -o $@ $<

341
doc/btrbk.1 Normal file
View File

@ -0,0 +1,341 @@
.TH "btrbk" "1" "2016-04-23" "btrbk v0.23.0" ""
.\" disable hyphenation
.nh
.\" disable justification (adjust text to left margin only)
.ad l
.SH NAME
btrbk \- backup tool for btrfs volumes
.SH SYNOPSIS
.nf
\fBbtrbk\fR [\-h|\-\-help] [\-\-version] [\-c|\-\-config <file>]
[\-n|\-\-dry\-run] [\-p|\-\-preserve] [\-r|\-\-resume\-only]
[\-v|\-\-verbose] [\-q|\-\-quiet] [\-l|\-\-loglevel <level>]
[\-t|\-\-table] [\-\-format <output\-format>]
[\-\-progress] [\-\-print\-schedule]
<command> [<args>]
.fi
.SH DESCRIPTION
\fBbtrbk\fR is a backup tool for btrfs subvolumes, taking advantage of
btrfs specific capabilities to create atomic snapshots and transfer
them incrementally to a target btrfs filesystem. It is able to perform
backups from one source to multiple destinations.
.PP
Snapshots as well as backup subvolume names are created in form:
.PP
.RS 4
<snapshot_name>.<timestamp>[_N]
.RE
.PP
Where <snapshot_name> is identical to the source subvolume name,
unless the configuration option \fIsnapshot_name\fR is set. The
<timestamp> is either "YYYYMMDD" or "YYYYMMDDThhmm" (dependent of the
\fItimestamp_format\fR configuration option), where "YYYY" is the
year, "MM" is the month, "DD" is the day, "hh" is the hour and "mm" is
the minute of the creation time (local time of the host running
btrbk). If multiple snapshots/backups are created on the same
date/time, N will be incremented on each snapshot, starting at 1.
.SH OPTIONS
.PP
\-h, \-\-help
.RS 4
Prints the synopsis and a list of the commands.
.RE
.PP
\-\-version
.RS 4
Prints the btrbk version.
.RE
.PP
\-n, \-\-dry\-run
.RS 4
Don't run anything that would alter the filesystem, just show the
snapshots and backup subvolumes that would be created/deleted by the
\fBrun\fR and \fBclean\fR commands. Use in conjunction with \fI\-l
debug\fR to see the btrfs commands that would be executed.
.RE
.PP
\-c, \-\-config <file>
.RS 4
Read the configuration from <file>.
.RE
.PP
\-p, \-\-preserve
.RS 4
Preserve all snapshots and backups. Skips deletion of any snapshots
and backups, even if specified in the configuration file.
.RE
.PP
\-r, \-\-resume-only
.RS 4
Resume only. Skips snapshot creation, only resumes missing backups to
satisfy the target retention policy.
.RE
.PP
\-v, \-\-verbose
.RS 4
Verbose output (shortcut for "\-\-loglevel=info").
.RE
.PP
\-q, \-\-quiet
.RS 4
Quiet operation. If set, btrbk does not print the summary after
executing the "run" command.
.RE
.PP
\-l, \-\-loglevel <level>
.RS 4
Set the level of verbosity. Accepted levels are warn, info, debug,
and trace.
.RE
.PP
\-t, \-\-table
.RS 4
Print output in table format (shortcut for "--format=table").
.RE
.PP
\-\-format table|long|raw
.RS 4
Print output in specified format. If set to "raw", prints
space-separated key="value" pairs (machine-readable). Affects output
format for \fBrun\fR, \fBdryrun\fR, \fBlist\fR and \fBtree\fR
commands. Useful for further exporting/scripting.
.RE
.PP
\-\-progress
.RS 4
Show progress bar on send-receive operation.
.RE
.PP
\-\-print\-schedule
.RS 4
Print detailed scheduler information on "run" and "dryrun"
commands. Use the \fI\-\-format\fR command line option to switch
between different output formats.
.RE
.PP
\-\-override <config_option>=<value>
.RS 4
Override a configuration option <config_option> with <value>.
Globally, for ALL contexts. Use with care!
.RE
.SH COMMANDS
.PP
.B run
[filter...]
.RS 4
Perform snapshot and backup operations as specified in the
configuration file. If the optional [filter...] arguments are present,
snapshots and backups are only performed for the subvolumes/targets
matching a \fIFILTER STATEMENT\fR (see below).
.PP
First, btrbk reads information from the source and target btrfs
filesystems in order to perform sanity checks and identify
parent/child and received-from relationships.
.PP
If the checks succeed, btrbk creates snapshots for the source
subvolumes specified in the configuration file, according to the
\fIsnapshot_create\fR option.
.PP
Then, for each specified target, btrbk creates the backups as follows:
After comparing the backups to the source snapshots, btrbk transfers
all missing snapshots needed to satisfy the configured target
retention policy, incrementally from the latest common parent
subvolume found. If no common parent subvolume is found (or if the
\fIincremental\fR option is set to \[lq]no\[rq]), a full
(non-incremental) backup is created.
.PP
As a last step, unless the \-p (\-\-preserve) option is set, snapshots
and backup subvolumes that are not preserved by their configured
retention policy will be deleted. Note that the latest snapshot (the
one created in the first step) as well as the latest snapshot/backup
pair are always preserved, regardless of the retention policy.
.PP
See section RETENTION POLICY in
.BR btrbk.conf (5)
for information on configuring the retention policy.
.PP
Use the \fI\-\-format\fR command line option to switch between
different output formats.
.RE
.PP
.B dryrun
[filter...]
.RS 4
Don't run any btrfs commands that would alter the filesystem, just
show the snapshots and backup subvolumes that would be created/deleted
by the \fBrun\fR command. Use in conjunction with \fI\-l debug\fR to
see the btrfs commands that would be executed.
.RE
.PP
.B archive
<source> <target>
.I *experimental*
.RS 4
Recursively copy all subvolumes created by btrbk from <source> to
<target> directory, optionally rescheduled using
\fIarchive_preserve_*\fR configuration options. Also creates directory
tree on <target> (see bugs below). Useful for creating extra archive
copies (clones) from your backup disks. Note that you can continue
using btrbk after swapping your backup disk with the archive disk.
.PP
Note that this feature needs a \fBlinux kernel >=4.4\fR to work
correctly! Kernels >=4.1 and <4.4 have a bug when re-sending
subvolumes (the archived subvolumes will have incorrect received_uuid,
see <http://thread.gmane.org/gmane.comp.file-systems.btrfs/48798>), so
make sure you run a recent kernel.
.PP
Known bugs: If you want to use nested subvolumes on the target
filesystem, you need to create them by hand (e.g. by running "btrfs
subvolume create <target>/dir"). Check the output of --dry-run if
unsure.
.RE
.PP
.B stats
[filter...]
.RS 4
Print statistics of snapshot and backup subvolumes. Optionally
filtered by [filter...] arguments (see \fIFILTER STATEMENTS\fR below).
.RE
.PP
.B list
<subcommand> [filter...]
.RS 4
Print information defined by <subcommand> in a tabular form. Optionally
filtered by [filter...] arguments (see \fIFILTER STATEMENTS\fR
below).
.PP
Available subcommands:
.TP 11
.B snapshots
All snapshots (and corresponding backups).
.PD 0
.TP 11
.B backups
All backups (and corresponding snapshots).
.TP 11
.B latest
Most recent common snapshot/backup pair, or most recent snapshot if no
common found.
.TP 11
.B config
Configured source/snapshot/target relations.
.TP 11
.B source
Configured source/snapshot relations.
.TP 11
.B volume
Configured volume sections.
.TP 11
.B target
Configured targets.
.PD
.PP
Use the \fI\-\-format\fR command line option to switch between
different output formats.
.RE
.PP
.B clean
[filter...]
.RS 4
Delete incomplete (garbled) backups. Incomplete backups can be left
behind on network errors or kill signals while a send/receive
operation is ongoing, and are identified by the "received_uuid" flag
not being set on a target (backup) subvolume.
.RE
.PP
.B usage
[filter...]
.RS 4
Print filesystem usage information for all source/target
volumes. Optionally filtered by [filter...] arguments (see \fIFILTER
STATEMENTS\fR below).
.RE
.PP
.B origin
<subvolume>
.RS 4
Print the subvolume origin tree: Shows the parent-child relationships
as well as the received-from information. Use the \fI\-\-format\fR
command line option to switch between different output formats.
.RE
.PP
.B diff
<from> <to>
.RS 4
Print new files since subvolume <from> for subvolume <to>.
.RE
.PP
.B config
print|print-all
.RS 4
Prints the parsed configuration file. Use the \fI\-\-format\fR command
line option to switch between different output formats.
.RE
.SH FILTER STATEMENTS
Filter arguments are accepted in form:
.PP
[hostname:]<volume-directory>
.RS 4
Matches all subvolumes and targets of a \fIvolume\fR configuration
section.
.RE
.PP
[hostname:]<volume-directory>/<subvolume-name>
.RS 4
Matches the specified subvolume and all targets of a \fIsubvolume\fR
configuration section.
.RE
.PP
[hostname:]<target-directory>
.RS 4
Matches all targets of a \fItarget\fR configuration section.
.RE
.PP
[hostname:]<target-directory>/<snapshot-name>
.RS 4
Matches a single target of a \fItarget\fR section within a
\fIsubvolume\fR section with given <snapshot-name>.
.RE
.PP
<group-name>
.RS 4
Matches the \fIgroup\fR configuration option of a \fIvolume\fR,
\fIsubvolume\fR or \fItarget\fR section.
.RE
.PP
For convenience, [hostname:] can be specified as either "hostname:" or
"ssh://hostname/".
.SH FILES
.PP
/etc/btrbk.conf
.br
/etc/btrbk/btrbk.conf
.RS 4
Default configuration file. The file format and configuration options
are described in
.BR btrbk.conf (5).
.RE
.PD
.SH EXIT STATUS
\fBbtrbk\fR returns the following error codes:
.IP "0" 4
No problems occurred.
.IP "1" 4
Generic error code.
.IP "2" 4
Parse error: when parsing command-line options or configuration file.
.IP "10" 4
Backup abort: At least one backup task aborted.
.IP "255" 4
Script error.
.SH AVAILABILITY
Please refer to the btrbk project page \fBhttp://digint.ch/btrbk/\fR
for further details.
.SH SEE ALSO
.BR btrbk.conf (5),
.BR btrfs (1)
.PP
For more information about btrfs and incremental backups, see the web
site at https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
.SH AUTHOR
Axel Burri <axel@tty0.ch>

View File

@ -1,533 +0,0 @@
btrbk(1)
========
:date: 2023-03-25
:release-version: 0.32.6
:man manual: Btrbk Manual
:man source: Btrbk {release-version}
NAME
----
btrbk - backup tool for btrfs subvolumes
SYNOPSIS
--------
[verse]
btrbk [-h|--help] [--version]
[-c|--config <file>] [-n|--dry-run] [--exclude <filter>]
[-p|--preserve] [--preserve-snapshots] [--preserve-backups]
[-v|--verbose] [-q|--quiet] [-l|--loglevel <level>]
[-t|--table] [-L|--long] [-1|--single-column]
[--format <output-format>] [--pretty]
[-S|--print-schedule] [--progress]
[--lockfile <file>]
[--override <config_option>=<value>]
<command> [[--] <filter>...]
DESCRIPTION
-----------
*btrbk* is a backup tool for btrfs subvolumes, taking advantage of
btrfs specific capabilities to create atomic snapshots and transfer
them incrementally to a target btrfs filesystem. It is able to perform
backups from one source to multiple destinations.
For most operations, *btrbk* requires 'root privileges' to run
correctly. Alternatively, consider using "btrfs-progs-sudo" or
"btrfs-progs-btrbk" backends, both of which allows you to run btrbk as
a regular user. Refer to configuration option 'backend' in
btrbk.conf(5) for more details.
=== Snapshots and Backups
Snapshots as well as backup subvolumes are created in the form:
<snapshot-name>.<timestamp>[_N]
Where '<snapshot-name>' is identical to the source subvolume name,
unless the configuration option 'snapshot_name' is set. '<timestamp>'
is a timestamp describing the creation time (local time of the host
running btrbk) of the snapshot/backup. The format can be configured
using the 'timestamp_format' option, refer to btrbk.conf(5) for
details. If multiple snapshots/backups are created on the same
date/time, 'N' will be incremented on each snapshot, starting at 1.
If a snapshot or backup does not match the naming scheme above
(e.g. if it has been renamed manually), btrbk will leave it untouched.
Note that in btrfs terminology, a 'snapshot' is a ``subvolume with
a given initial content of the original subvolume'' (showing a
parent-uuid, see btrfs-subvolume(8)), and they can be read-write
(default) or read-only. In btrbk terminology, 'snapshot' means
``read-only btrfs snapshot'', and 'backup' means ``read-only subvolume
created with send/receive'' (showing a received-uuid).
OPTIONS
-------
-h, --help::
Prints the synopsis and a list of the commands.
--version::
Prints the btrbk version.
-c, --config <file>::
Read the configuration from <file>.
-n, --dry-run::
Don't run anything that would alter the filesystem, just show the
snapshots and backup subvolumes that would be created/deleted by
the *run*, *snapshot*, *resume*, *prune*, *archive* and *clean*
commands. Use in conjunction with '-l debug' to see the btrfs
commands that would be executed.
--exclude <filter>::
Exclude configured sections matching '<filter>' (see
<<_filter_statements,FILTER STATEMENTS>> below), or any specific
snapshot from being backuped or deleted, or any specific backup
from being deleted.
+
Note that excluding specific snapshots from being backuped has impact
on scheduling: e.g. if the "first snapshot of the day" is excluded,
the "second snapshot of the day" shifts to "first", creating a backup
as "first backup of the day".
-p, --preserve::
Preserve all snapshots and backups. Skips deletion of any
snapshots and backups, even if specified in the configuration file
(shortcut for "--preserve-snapshots --preserve-backups").
--preserve-snapshots::
Preserve all snapshots. Skips deletion of any snapshots, even if
specified in the configuration file.
--preserve-backups::
Preserve all backups. Skips deletion of any backups, even if
specified in the configuration file.
--wipe::
Ignore configured snapshot retention policy, delete all but the latest
snapshots instead. All snapshots needed for incremental backup
(latest common) are also preserved. Useful if you are getting low
on disk space (ENOSPC).
-v, --verbose::
Increase the logging level, see "--loglevel".
-q, --quiet::
Quiet operation. If set, btrbk does not print the summary after
executing the *run*, *snapshot*, *resume*, *prune*, or *archive*
commands.
-l, --loglevel <level>::
Set the level of verbosity for the stderr logging. Accepted levels
are: error, warn, info, debug, and trace. Default is info.
-t, --table::
Print output in table format (shortcut for "--format=table").
-L, --long::
Print output in long table format (shortcut for "--format=long").
-1, --single-column::
Print output as single column (not available for all commands).
--format table|long|raw|col:[h:]<columns>::
Print output in specified format. If set to "raw", prints
space-separated, quoted key=value pairs (machine readable).
+
If set to "col:", prints only the <columns> specified (comma-separated
list). Header lines are omitted if the "h:" modifier is present.
Columns prefixed with "-" are collapsed if empty. Columns postfixed
with ":RALIGN" are right-aligned.
--pretty::
Print table output with lowercase, underlined column headings
(instead of single-line uppercase headings).
-S, --print-schedule::
Print detailed scheduler information on *run*, *snapshot*,
*resume*, *prune* and *archive* commands. Use the '--format'
command line option to switch between different output formats.
--progress::
Show progress bar on send-receive operation. Requires "mbuffer"
command (version >= 20180505) installed on the host running btrbk.
--lockfile <file>::
Place an exclusive lock on <file> during program execution, using
flock(2). If the lock is held by another process, exit before
running any actions. Overrides configuration option
"lockfile". Ignored on dryrun ('-n', '--dry-run').
--override <config_option>=<value>::
Override a configuration option <config_option> with
<value>. Globally, for ALL contexts. Use with care!
COMMANDS
--------
=== Actions
The following commands are used to create snapshots and/or
backups. All actions can operate in dry-run mode ('-n', '--dry-run').
Use the '--format' command line option to switch between different
output formats.
See section RETENTION POLICY in *btrbk.conf*(5) for information on
configuring the retention policy.
*run* [filter...]::
Perform snapshot and backup operations as specified in the
configuration file. If the optional [filter...] arguments are
present, snapshots and backups are only performed for the
subvolumes/targets matching a filter statement (see
<<_filter_statements,FILTER STATEMENTS>> below).
+
*Step 0: Read Data*;;
Read information from the source and target btrfs filesystems in
order to perform sanity checks and identify parent/child and
received-from relationships.
+
*Step 1: Create Snapshots*;;
If the checks succeed, btrbk creates snapshots for the source
subvolumes specified in the configuration file, according to the
'snapshot_create' option.
+
*Step 2: Create Backups*;;
For each specified target, btrbk creates the backups as follows:
After comparing the backups to the source snapshots, btrbk
transfers all missing snapshots needed to satisfy the configured
target retention policy, incrementally from the latest common
parent subvolume found. If no common parent subvolume is found (or
if the 'incremental' option is set to ``no''), a full
(non-incremental) backup is created.
+
*Step 3: Delete Backups*;;
Unless the -p, --preserve or --preserve-backups option is set,
backup subvolumes that are not preserved by their configured
retention policy will be deleted. Note that the latest
snapshot/backup pair are always preserved, regardless of the
retention policy.
+
*Step 4: Delete Snapshots*;;
Unless the -p, --preserve or --preserve-snapshots option is set,
snapshots that are not preserved by their configured retention
policy will be deleted. Note that the latest snapshot (the one
created in step 1) as well as the latest snapshot/backup pair are
always preserved, regardless of the retention policy. If any
target is unreachable or has errors, all snapshots are preserved
in order not to break the incremental chain.
*dryrun* [filter...]::
Don't run any btrfs commands that would alter the filesystem, just
show the snapshots and backup subvolumes that would be
created/deleted by the *run* command. Use in conjunction with '-l
debug' to see the btrfs commands that would be executed.
*snapshot* [filter...]::
Snapshot only: skips backup creation and deletion (steps 2 and
3). Use in conjunction with -p, --preserve (or
--preserve-snapshots) if you also want to skip snapshot deletion
(step 4).
+
Note that snapshot deletion is skipped if the target is not
accessible, as it is still required in order to determine the latest
snapshot/backup pair (which is always preserved, regardless of the
retention policy).
*resume* [filter...]::
Resume backups: skips snapshot creation (step 1), transfers and
deletes snapshots/backups in order to satisfy their configured
retention policy. Use in conjunction with -p, --preserve,
--preserve-backups, --preserve-snapshots if you want to skip
backup and/or snapshot deletion (steps 3, 4).
*prune* [filter...]::
Prune snapshots and backups: skips snapshot and backup creation
(steps 1, 2), only deletes snapshots and backups in order to
satisfy their configured retention policy. Useful for cleaning the
disk after changing the retention policy. Use in conjunction with
--preserve-backups, --preserve-snapshots if you want to skip
backup or snapshot deletion (steps 3, 4).
+
Note that deletion is skipped if source or target is not accessible,
as it is still required in order to determine the latest
snapshot/backup pair (which is always preserved, regardless of the
retention policy).
*archive* <source> <target> [--raw]::
Recursively copy all subvolumes created by btrbk from <source> to
<target> directory, optionally rescheduled using
'archive_preserve_*' configuration options. Also creates directory
tree on <target>. Useful for creating extra archive copies
(clones) from your backup disks. Note that you can continue using
btrbk after swapping your backup disk with the archive disk.
+
If you want to use nested subvolumes on the target filesystem, you
need to create them by hand (e.g. by running "btrfs subvolume create
<target>/dir"). Check the output of --dry-run if unsure.
+
Note that this feature needs a *linux kernel >=4.4* to work correctly!
+
If '--raw' option is set, creates raw targets (experimental, see
btrbk.conf(5), TARGET TYPES).
*clean* [filter...]::
Delete incomplete (garbled) backups. Incomplete backups can be
left behind on network errors or kill signals while a send/receive
operation is ongoing, and are identified by the "received_uuid"
flag not being set on a target (backup) subvolume.
The following table gives a quick overview of the action commands and
resulting snapshot creation (S+), backup creation (B+), snapshot
deletion (S-), and backup deletion (B-):
ifdef::backend-docbook,backend-manpage[]
....
Command Option S+ B+ S- B-
--------------------------------------------
run x x x x
run --preserve x x
run --preserve-snapshots x x x
run --preserve-backups x x x
snapshot x x
snapshot --preserve x
resume x x x
resume --preserve x
resume --preserve-snapshots x x
resume --preserve-backups x x
prune x x
prune --preserve-snapshots x
prune --preserve-backups x
....
endif::backend-docbook,backend-manpage[]
ifndef::backend-docbook,backend-manpage[]
[cols="2*<m,4*^", options="header,autowidth,compact", style="monospaced"]
|=======
|Command |Option |S+ |B+ |S- |B-
|run | | x | x | x | x
|run |--preserve | x | x | |
|run |--preserve-snapshots | x | x | | x
|run |--preserve-backups | x | x | x |
|snapshot | | x | | x |
|snapshot |--preserve | x | | |
|resume | | | x | x | x
|resume |--preserve | | x | |
|resume |--preserve-snapshots | | x | | x
|resume |--preserve-backups | | x | x |
|prune | | | | x | x
|prune |--preserve-snapshots | | | | x
|prune |--preserve-backups | | | x |
|=======
endif::backend-docbook,backend-manpage[]
=== Informative Commands
The following commands are informative only, and will not alter the
file system.
*stats* [filter...]::
Print statistics of snapshot and backup subvolumes. Optionally
filtered by [filter...] arguments (see <<_filter_statements,FILTER
STATEMENTS>> below).
*list* <subcommand> [filter...]::
Print information defined by <subcommand> in a tabular
form. Optionally filtered by [filter...] arguments (see
<<_filter_statements,FILTER STATEMENTS>> below).
+
Available subcommands (default ``all''):
+
--
ifndef::backend-docbook,backend-manpage[]
[horizontal]
endif::backend-docbook,backend-manpage[]
*all*;; List all snapshots and backups created by btrbk.
*snapshots*;; List all snapshots created by btrbk.
*backups*;; List all backups (and correlated snapshots) created by
btrbk.
*latest*;; List most recent common snapshot/backup pair, or most
recent snapshot if no common found.
*config*;; List configured source/snapshot/target relations.
*source*;; List configured source/snapshot relations.
*volume*;; List configured volume sections.
*target*;; List configured targets.
--
+
Use the '--format' command line option to switch between different
output formats.
*usage* [filter...]::
Print filesystem usage information for all source/target volumes,
optionally filtered by [filter...] arguments (see
<<_filter_statements,FILTER STATEMENTS>> below). Note that the
"free" value is an estimate of the amount of data that can still
be written to the file system.
*origin* <subvolume>::
Print the subvolume origin tree: Shows the parent-child
relationships as well as the received-from information. Use the
'--format' command line option to switch between different output
formats.
*diff* <from> <to>::
List the modified files since generation (transid) of subvolume
<from> in subvolume <to>. Columns:
+
------------
SIZE file was modified for a total of SIZE bytes
COUNT file was modified in COUNT generations
FLAGS "+" file accessed at offset 0 (at least once)
"c" COMPRESS flag is set (at least once)
"i" INLINE flag is set (at least once)
------------
*extents* [diff] <subvolume>... [exclusive <subvolume>...]::
Print accurate disk space usage and diff based on extent data
(FIEMAP ioctl, slow!).
+
--
Subvolumes following the 'exclusive' keyword are added to a separate
set, and additional set-exclusive data is printed at the end of the
list. This gives a hint of how much data will be freed if deleting all
subvolumes in the set. Example:
btrbk extents diff /backup/data.* exclusive /backup/data.2010*
The EXCLUSIVE column shows the set-exclusive data of all other listed
(!) subvolumes (relative complement of block regions). Provided that
all related subvolumes (holding references to extents) are also
listed, this amount of disk space would be freed when deleting the
subvolume.
The DIFF column shows the data added to the previous subvolume
(relative complement of block regions).
If called with the '--related' option, btrbk also lists all related
subvolumes. This is not recommended for backups, as parent-uuid
relations break for received subvolumes as soon as an intermediate
subvolume is deleted.
Note that reading all extents is a disk-intensive task, expect long
execution times and high ram usage. Consider setting 'cache_dir'.
--
*ls* <path>|<url>...::
List all btrfs subvolumes below <path>. Use the '--format' command
line option to switch between different output formats. See
lsbtr(1).
*config* print|print-all::
Prints the parsed configuration file.
FILTER STATEMENTS
-----------------
Filter arguments are accepted in form:
<group-name>::
Matches the 'group' configuration option of 'volume', 'subvolume'
or 'target' sections.
<hostname>[:<port>]::
Matches the 'hostname' portion from '<url>' of 'volume' or
'target' sections.
<directory>|<url>::
Matches 'volume', 'subvolume' or 'target' sections by either
relative or absolute path (if starting with "/" or "ssh://" or
"<hostname>:/"), accepting wildcard character "*". Relative paths
are matched against the end of the pathname. Either:
+
--
<volume-directory>::
Matches 'volume' sections.
<volume-directory>/<subvolume-name>::
Matches 'subvolume' sections.
<volume-directory>/<snapshot-dir>/<snapshot-name>::
Matches 'subvolume' sections defining snapshots with the
configured 'snapshot_dir' and 'snapshot_name'.
<target-directory>::
Matches 'target' sections.
<target-directory>/<snapshot-name>::
Matches 'target' sections within 'subvolume' sections defining
snapshots with the configured 'snapshot_name'.
Accepted formats for '<url>' are:
ssh://<hostname>[:<port>]/<directory>
<hostname>:<directory>
--
Note that for *run* and *snapshot* commands, a filter matching a
'target' configuration section also enables snapshot creation of the
surrounding 'subvolume' section. If this is not desired, consider
running *snapshot* and *resume* commands separately.
Filter statements can match multiple times (e.g. on group as well as
host name). In such a case, all matches are processed.
FILES
-----
+/etc/btrbk.conf+::
+/etc/btrbk/btrbk.conf+::
Default configuration file. The file format and configuration
options are described in *btrbk.conf*(5).
EXIT STATUS
-----------
*btrbk* returns the following error codes:
ifndef::backend-docbook,backend-manpage[]
[horizontal]
endif::backend-docbook,backend-manpage[]
0:: No problems occurred.
1:: Generic error code.
2:: Parse error: when parsing command-line options or configuration
file.
3:: Lockfile error: if lockfile is present on startup.
10:: Backup abort: At least one backup task aborted.
255:: Script error.
AVAILABILITY
------------
Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
for further details.
SEE ALSO
--------
*btrbk.conf*(5),
*btrfs*(8)
For more information about btrfs and incremental backups, see the web
site at https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
AUTHOR
------
Axel Burri <axel@tty0.ch>

372
doc/btrbk.conf.5 Normal file
View File

@ -0,0 +1,372 @@
.TH "btrbk.conf" "5" "2016-04-23" "btrbk v0.23.0" ""
.\" disable hyphenation
.nh
.\" disable justification (adjust text to left margin only)
.ad l
.SH NAME
btrbk.conf \- btrbk configuration file
.SH SYNOPSIS
.B /etc/btrbk.conf
.br
.B /etc/btrbk/btrbk.conf
.SH DESCRIPTION
The btrbk configuration file specifies which btrfs subvolumes on the
filesystem are to be processed, what target subvolumes should be used
to create the backups, and where the snapshots should be
generated. The retention policy, as well as most other options can be
defined either globally or within a section.
.PP
The options specified always apply to the last section encountered,
superseding the values set in upper-level sections. This means that
global options must be set before any sections are defined.
.SH SECTIONS
.PP
\fBvolume\fR <volume-directory>|<url>
.RS 4
Directory of a btrfs volume containing the source subvolume(s) to be
backed up. \fI<volume-directory>\fR must be an absolute path and point
to a btrfs volume (or subvolume). Usually the mount point of a btrfs
filesystem mounted with the \fIsubvolid=0\fR option.
.RE
.PP
\fBsubvolume\fR <subvolume-name>
.RS 4
Subvolume to be backed up, relative to the \fI<volume-directory>\fR
specified in the \fIvolume\fR section. Multiple \fIsubvolume\fR
sections are allowed within \fIvolume\fR sections.
.RE
.PP
\fBtarget\fR <type> <target-directory>|<url>
.RS 4
Target type and directory where the backup subvolumes are to be
created. See the TARGET TYPES section for supported
\fI<type>\fR. Multiple \fItarget\fR sections are allowed within
\fIsubvolume\fR sections. A \fItarget\fR section defined in the global
context or in a \fIvolume\fR section is propagated (mulitplied) to all
underlying \fIsubvolume\fR sections, unless a target with the same
declaration already exists (hint: run "btrbk config print" to see the
resulting configuration).
.RE
.PP
For the \fIvolume\fR and \fItarget\fR sections, you can specify a
ssh-url instead of a local directory. The syntax for \fI<url>\fR is:
.PP
.RS 4
.nf
ssh://host.xz/path/to/volume
.fi
.RE
.PP
Note that btrfs is very picky on file names (mainly for security
reasons), only the characters [0-9] [a-z] [A-Z] and "._+-@" are
allowed.
.RE
.SH OPTIONS
.PP
\fBtransaction_log\fR <file>
.RS 4
If set, all transactions (snapshot create, subvolume send-receive,
subvolume delete) as well as abort messages are logged to <file>, in a
space-separated table format.
.RE
.PP
\fBtimestamp_format\fR short|long|long-iso
.RS 4
Timestamp format used as postfix for new snapshot subvolume
names. Defaults to \[lq]short\[rq].
.PP
.IP \fBshort\fR 10
YYYYMMDD[_N] (e.g. "20150825", "20150825_1")
.IP \fBlong\fR 10
YYYYMMDD<T>hhmm[_N] (e.g. "20150825T1531")
.IP \fBlong-iso\fR 10
YYYYMMDD<T>hhmmss\[t+-]hhmm[_N] (e.g. "20150825T153123+0200")
.PP
Note that a postfix "_N" is appended to the timestamp if a snapshot or
backup already exists with the timestamp of current date/time.
.PP
Use \[lq]long-iso\[rq] if you want to make sure that btrbk never
creates ambiguous time stamps (which can happen if multiple snapshots
are created during a daylight saving time clock change).
.PP
Note that using \[lq]long-iso\[rq] has implications on the scheduling,
see RETENTION POLICY (caveats) below.
.RE
.PP
\fBsnapshot_dir\fR <directory>
.RS 4
Directory in which the btrfs snapshots are created, relative to
\fI<volume-directory>\fR of the \fIvolume\fR section. Note that btrbk
does not autmatically create this directory, and the snapshot creation
will fail if it is not present.
.RE
.PP
\fBsnapshot_name\fR <basename>
.RS 4
Base name of the created snapshot (and backup). This option is only
valid in the \fIsubvolume\fR section. Defaults to
\fI<subvolume-name>\fR.
.RE
.PP
\fBsnapshot_create\fR always|onchange|ondemand|no
.RS 4
If set to \[lq]always\[rq], snapshots are always created. If set to
\[lq]onchange\[rq], snapshots are only created if the source subvolume
has changed since the last snapshot (more precisely: if the btrfs
generation has been increased since the last snapshot). If set to
\[lq]ondemand\[rq], snapshots are only created if the target subvolume
is reachable (useful if you are tight on disk space and you only need
btrbk for backups to an external disk which is not always
connected). If set to \[lq]no\[rq], the snapshots are never created
(useful if another instance of btrbk is taking care of snapshot
creation). Defaults to \[lq]always\[rq].
.RE
.PP
\fBincremental\fR yes|no|strict
.RS 4
If set, incremental backups are created. If set to \[lq]strict\[rq],
non-incremental (initial) backups are never created. Defaults to
\[lq]yes\[rq].
.RE
.PP
\fBsnapshot_preserve\fR no|<retention_policy>
.RS 4
Set retention policy for snapshots (see RETENTION POLICY below). If
set to \[lq]no\[rq], preserve snapshots according to
\fIsnapshot_preserve_min\fR only. Defaults to \[lq]no\[rq].
.RE
.PP
\fBsnapshot_preserve_min\fR all|latest|<number>{h,d,w,m,y}
.RS 4
Preserve all snapshots for a minimum amount of hours (h), days (d),
weeks (w), months (m) or years (y), regardless of how many there
are. If set to \[lq]all\[rq], preserve all snapshots forever. If set
to \[lq]latest\[rq], preserve latest snapshot. Defaults to
\[lq]all\[rq].
.RE
.PP
\fBtarget_preserve\fR no|<retention_policy>
.RS 4
Set retention policy for backups (see RETENTION POLICY below). If set
to \[lq]no\[rq], preserve backups according to
\fItarget_preserve_min\fR only. Defaults to \[lq]no\[rq].
.RE
.PP
\fBtarget_preserve_min\fR all|latest|no|<number>{h,d,w,m,y}
.RS 4
Preserve all backups for a minimum amount of hours (h), days (d),
weeks (w), months (m) or years (y), regardless of how many there
are. If set to \[lq]all\[rq], preserve all backups forever. If set to
\[lq]latest\[rq], always preserve the latest backup (useful in
conjunction with "target_preserve no", if you want to keep the latest
backup only). If set to \[lq]no\[rq], only the backups following the
\fItarget_preserve\fR policy are created. Defaults to \[lq]all\[rq].
.RE
.PP
\fBpreserve_day_of_week\fR monday|tuesday|...|sunday
.RS 4
Defines on what day a backup/snapshot is considered as a weekly
backup. Defaults to \[lq]sunday\[rq].
.RE
.PP
\fBgroup\fR <group-name>[,<group-name>]...
.RS 4
Add the current section (volume, subvolume or target) to a
user-defined group, which can be used as filter for several btrbk
commands.
.RE
.PP
\fBssh_identity\fR <file>
.RS 4
Absolute path to a ssh identity file (private key). Note that if the
private key is password protected, btrbk will prompt for user input,
which is usually not desired.
.RE
.PP
\fBssh_user\fR <username>
.RS 4
Remote username for ssh. Defaults to \[lq]root\[rq]. Note that you will
have to make sure that the remote user is able to run /sbin/btrfs
(which needs root privileges).
.RE
.PP
\fBssh_port\fR <port>
.RS 4
Port to connect to on the remote host. Defaults to \[lq]default\[rq]
(the port specified in \fIssh_config\fR, which defaults to 22).
.RE
.PP
\fBssh_compression\fR yes|no
.RS 4
Enables or disables the compression of ssh connections. Defaults to
\[lq]no\[rq].
.RE
.PP
\fBssh_cipher_spec\fR <cipher_spec>
.RS 4
Selects the cipher specification for encrypting the session
(comma-separated list of ciphers in order of preference). See the "-c
cipher_spec" option in ssh(1) for more information. Defaults to
\[lq]default\[rq] (the ciphers specified in \fIssh_config\fR).
.RE
.PP
\fBrate_limit\fR <rate>|no
.RS 4
Limit the transfer to a maximum of \fI<rate>\fR bytes per second. A
suffix of "k", "m", "g", or "t" can be added to denote kilobytes
(*1024), megabytes, and so on. Defaults to \[lq]no\[rq].
.RE
.PP
\fBbtrfs_commit_delete\fR after|each|no
.RS 4
If set, make sure the deletion of snapshot and backup subvolumes are
committed to disk when btrbk terminates. Defaults to \[lq]no\[rq].
.RE
.PP
Lines that contain a hash character (#) in the first column are
treated as comments.
.SH RETENTION POLICY
btrbk uses separate retention policies for snapshots and backups,
which are defined by the \fIsnapshot_preserve_min\fR,
\fIsnapshot_preserve\fR, \fItarget_preserve_min\fR,
\fItarget_preserve\fR, and the \fIpreserve_day_of_week\fR
configuration options.
.PP
Within this section, any statement about "backups" is always valid for
backups as well as snapshots, referring to \fItarget_preserve\fR or
\fIsnapshot_preserve\fR respectively.
.PP
The format for \fI<retention_policy>\fR is:
.PP
.RS 4
[<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]
.RE
.PP
With the following semantics:
.PP
.B hourly
.RS 4
Defines how many hours back hourly backups should be preserved. The
first backup of an hour is considered an hourly backup. Note that if
you use <hourly> scheduling, make sure to also set
\fItimestamp_format\fR to \[lq]long\[rq] or \[lq]long-iso\[rq], or the
scheduler will interpret the time as "00:00" (midnight).
.RE
.PP
.B daily
.RS 4
Defines how many days back daily backups should be preserved. The
first backup of a day is considered a daily backup.
.RE
.PP
.B weekly
.RS 4
Defines how many weeks back weekly backups should be preserved. The
first daily backup created at \fIpreserve_day_of_week\fR (or the first
backup in this week if none was made on the exact day) is considered
as a weekly backup.
.RE
.PP
.B monthly
.RS 4
Defines how many months back monthly backups should be
preserved. Every first weekly backup in a month is considered a
monthly backup.
.RE
.PP
.B yearly
.RS 4
Defines for how many years back yearly backups should be
preserved. Every first monthly backup in a year is considered a yearly
backup.
.RE
.PP
Use an asterisk for \[lq]all\[rq] (e.g. "target_preserve 60d *m"
states: "preserve daily backups for 60 days back, and all monthly
backups").
.PP
The reference time (which defines the beginning of a day, week, month
or year) for all date/time calculations is the local time of the host
running btrbk.
.PP
Caveats:
.IP \[bu] 2
If you run a setup with several btrbk instances (e.g. one
snapshot-only instance per remote client, and a separate fetch-only
instance on the backup server), it makes perfectly sense to run btrbk
with different local time on the clients, in order to make sure the
backups from all the remote hosts are preserved for "midnight", and
not at "00:00 UTC" (which would be "14:00" in Honolulu). If you want
this behaviour, do NOT use "timestamp_format long-iso".
.IP \[bu] 2
If "timestamp_format long-iso" is set, running btrbk from different
time zones leads to different interpretation of "first in day, week,
month, or year". Make sure to run btrbk with the same time zone on
every host, e.g. by setting the TZ environment variable (see
tzset(3)).
.SH TARGET TYPES
.PP
\fBsend-receive\fR
.RS 4
Backup to a btrfs filesystem, using "btrfs send/receive". This is the
recommended (standard) target type. The \fI<target-directory>\fR must
be an absolute path and point to a btrfs volume (or subvolume), or to
a directory within a subvolume. See btrfs-send(8), btrfs-receive(8).
.RE
.PP
\fBraw\fR \fI*experimental*\fR
.RS 4
Backup to a raw (filesystem independent) file from the output of
btrfs-send(8), with optional compression and encryption.
.PP
Note that the target preserve mechanism is currently disabled for raw
backups (btrbk does not delete any raw files)!
.PP
Additional options for raw targets:
.PP
.RS 4
raw_target_compress gzip|bzip2|xz|no
.PD 0
.PP
raw_target_compress_level default|<number>
.PP
raw_target_compress_threads default|<number>
.PP
raw_target_encrypt gpg|no
.PP
gpg_keyring <file>
.PP
gpg_recipient <name>
.RE
.PD
.PP
Target file name syntax:
.PP
.RS 4
<snapshot-name>--<received_uuid>[@<parent_uuid>].btrfs[.gz|.bz2|.xz][.gpg]
.RE
.PP
The <parent_uuid> is only set on \fIincremental\fR backups, and points
to the <received_uuid> of the previous backup in a incremental backup
chain.
.PP
For \fIincremental\fR backups ("incremental yes"), please note that:
.IP 1. 4
As soon as a single \fIincremental\fR backup file is lost or
corrupted, all later incremental backups become invalid, as there is
no common parent for the subsequent incremental images anymore. This
might be a good compromise for a vacation backup plan, but for the
long term make sure that a non-incremental backup is triggered from
time to time.
.IP 2. 4
There is currently no support for rotation of incremental backups: if
\fIincremental\fR is set, a full backup must be triggered manually
from time to time in order to be able to delete old backups.
.RE
.SH AVAILABILITY
Please refer to the btrbk project page \fBhttp://digint.ch/btrbk/\fR
for further details.
.SH SEE ALSO
.BR btrbk (1)
.SH AUTHOR
Axel Burri <axel@tty0.ch>

View File

@ -1,705 +0,0 @@
btrbk.conf(5)
=============
:date: 2023-03-25
:release-version: 0.32.6
:man manual: Btrbk Manual
:man source: Btrbk {release-version}
NAME
----
btrbk.conf - btrbk configuration file
SYNOPSIS
--------
[verse]
/etc/btrbk.conf
/etc/btrbk/btrbk.conf
DESCRIPTION
-----------
The btrbk configuration file specifies which btrfs subvolumes on the
filesystem are to be processed, what target subvolumes should be used
to create the backups, and where the snapshots should be
generated. The retention policy, as well as most other options can be
defined either globally or within a section.
The options specified always apply to the last section encountered,
superseding the values set in upper-level sections. This means that
global options must be set before any sections are defined.
Blank lines are ignored. A hash character (#) starts a comment
extending until end of line.
SECTIONS
--------
*volume* <volume-directory>|<url> (optional)::
Absolute path pointing to a btrfs file system containing the
source subvolume(s) to be backed up. Usually the mount point of a
btrfs filesystem mounted with the 'subvolid=5' option.
*subvolume* <subvolume-name>::
Subvolume to be backed up, relative to the '<volume-directory>' of
the 'volume' section, or absolute if the 'volume' section is
omitted. Accepts wildcard character "*".
+
--
Note that if this subvolume is btrfs root (id=5), it needs to have a
valid UUID, which is not the case for file systems created with
btrfs-progs < 4.16.
--
*target* [send-receive|raw] <target-directory>|<url>::
Target directory where the backup subvolumes are to be
created. The optional target type defaults to ``send-receive'',
see <<_target_types,TARGET TYPES>> below for details.
+
--
Multiple 'target' sections are allowed, in any context: a 'target'
defined in 'volume' or global context will be used for all underlying
'subvolume' sections (hint: run "btrbk list" or "btrbk config print"
to see the resulting configuration).
--
If a '<url>' is specified, btrbk actions (shell commands) are executed
remotely via ssh, using the <<_ssh_options,SSH Options>> described
below. Accepted formats are:
ssh://<hostname>[:<port>]/<directory>
<hostname>:<directory>
Where '<hostname>' is either a host name, an IPv4 address in
dotted-decimal form, or an IP literal encapsulated within square
brackets (e.g. "[2001:db8::7]").
If you are connecting to virtual machines, consider configuring
several 'volume' sections for a '<hostname>', with distinct '<port>'
numbers for each machine.
OPTIONS
-------
The options described here can be specified in 'global context' as
well as 'volume', 'subvolume' and 'target' sections, unless stated
otherwise.
=== Basic Options
*timestamp_format* short|long|long-iso::
Timestamp format used as postfix for new snapshot subvolume
names. Defaults to ``long''.
+
--
ifndef::backend-docbook,backend-manpage[]
[horizontal]
endif::backend-docbook,backend-manpage[]
*short*;; +YYYYMMDD[_N]+ (e.g. "20150825", "20150825_1")
*long*;; +YYYYMMDD<T>hhmm[_N]+ (e.g. "20150825T1531")
*long-iso*;; +YYYYMMDD<T>hhmmss&plusmn;hhmm[_N]+ (e.g. "20150825T153123+0200")
--
+
Note that a postfix "_N" is appended to the timestamp if a snapshot or
backup already exists with the timestamp of current date/time.
+
Use ``long-iso'' if you want to make sure that btrbk never
creates ambiguous time stamps (which can happen if multiple
snapshots are created during a daylight saving time clock
change).
+
Note that using ``long-iso'' has implications on the scheduling, see
<<_reference_time,Reference Time>> below.
*snapshot_dir* <directory>::
Directory in which the btrfs snapshots are created, relative to
'<volume-directory>' of the 'volume' section, or absolute if the
'volume' section is omitted. Note that btrbk does not
automatically create this directory, and the snapshot creation
will fail if it is not present.
*snapshot_name* <basename>::
Base name of the created snapshot (and backup). This option is
only valid in the 'subvolume' section. Defaults to
'<subvolume-name>'.
*snapshot_create* always|onchange|ondemand|no::
If set to ``always'', snapshots are always created. If set to
``onchange'', snapshots are only created if the last snapshot is
not up-to-date, i.e. the source subvolume has changed (more
precisely: the btrfs generation has been increased) since the last
snapshot was created. If set to ``ondemand'', snapshots are only
created if at least one target subvolume is reachable (useful if
you are tight on disk space and you only need btrbk for backups to
an external disk which is not always connected). If set to ``no'',
the snapshots are never created (useful if another instance of
btrbk is taking care of snapshot creation). Defaults to
``always''.
*incremental* yes|no|strict::
If set, incremental backups are created. If set to ``strict'',
non-incremental (initial) backups are never created, and
incremental backups are restricted to 'related parents' (by
parent-uuid relationship). Defaults to ``yes''.
+
--
Note that even if the parent-uuid chain is broken, snapshots and
backups can still share data (which is especially true for backups
created with 'incremental' option enabled), and are perfectly suitable
as parents for incremental send-receive operations. But as btrbk can
not be certain about this, such operations are disallowed in
"incremental strict" mode.
--
*noauto* yes|no::
If set, the context is skipped by all btrbk actions unless
explicitly enabled by a matching btrbk '<filter>' command line
argument (e.g. "btrbk run myfilter").
=== Grouping Options
*group* <group-name> [<group-name>]...::
Add the current section (volume, subvolume or target) to
user-defined groups, which can be used as filter for most btrbk
commands (see btrbk(1) FILTER STATEMENTS). This option can be set
multiple times within the same context.
=== Retention Policy Options
*preserve_day_of_week* monday|tuesday|...|sunday::
Defines on what day a snapshot/backup is considered to be a
"weekly" backup. Weekly, monthly and yearly backups are preserved
on this day of week (see <<_retention_policy,RETENTION POLICY>>
below). Defaults to ``sunday''.
*preserve_hour_of_day* [0..23]::
Defines after what time (in full hours since midnight) a
snapshot/backup is considered to be a "daily" backup. Daily,
weekly, monthly and yearly backups are preserved on this hour (see
<<_retention_policy,RETENTION POLICY>> below). Ignored on
snapshots or backups without time information ('timestamp_format
short'). Defaults to ``0''.
*snapshot_preserve* no|<retention_policy>::
Set retention policy for snapshots (see
<<_retention_policy,RETENTION POLICY>> below). If set to ``no'',
preserve snapshots according to 'snapshot_preserve_min'
only. Defaults to ``no''.
+
--
Note that 'snapshot_preserve' has no effect if 'snapshot_preserve_min'
is set to ``all'' (the default).
--
*snapshot_preserve_min* all|latest|<number>{h,d,w,m,y}::
Preserve all snapshots for a minimum amount of hours (h), days
(d), weeks (w), months (m) or years (y), regardless of how many
there are. If set to ``all'', preserve all snapshots forever. If
set to ``latest'', preserve latest snapshot. Defaults to ``all''.
*target_preserve* no|<retention_policy>::
Set retention policy for backups (see
<<_retention_policy,RETENTION POLICY>> below). If set to ``no'',
preserve backups according to 'target_preserve_min' only. Defaults
to ``no''.
+
--
Note that 'target_preserve' has no effect if 'target_preserve_min' is
set to ``all'' (the default).
--
*target_preserve_min* all|latest|no|<number>{h,d,w,m,y}::
Preserve all backups for a minimum amount of hours (h), days (d),
weeks (w), months (m) or years (y), regardless of how many there
are. If set to ``all'', preserve all backups forever. If set to
``latest'', always preserve the latest backup (useful in
conjunction with "target_preserve no", if you want to keep the
latest backup only). If set to ``no'', only the backups following
the 'target_preserve' policy are created. Defaults to ``all''.
*archive_preserve* no|<retention_policy>::
Set retention policy for archives ("btrbk archive" command), with
same semantics as 'target_preserve'.
*archive_preserve_min* all|latest|no|<number>{h,d,w,m,y}::
Set retention policy for archives ("btrbk archive" command), with
same semantics as 'target_preserve_min'.
*archive_exclude* <pattern>::
Exclude subvolumes matching <pattern> from archiving. The pattern
accepts wildcard character "*", and is matched against the end of
the pathname.
=== SSH Options
*ssh_identity* <file>|no::
Absolute path to a ssh identity file (private key). If not set,
the ssh default is used (see ssh(1), "-i identity_file"). Note
that if the identity key is password protected and no
authentication agent is used, btrbk will prompt for user input on
every connection attempt.
*ssh_user* <username>|no::
Remote username for ssh. Defaults to ``root''. Make sure the
remote user is able to run "btrfs" with root privileges (see
option 'backend' for details). If set to ``no'', the ssh default
is used.
*ssh_compression* yes|no::
Enables or disables the compression of ssh connections. Defaults
to ``no''. Note that if *stream_compress* is enabled, ssh
compression will always be disabled for send/receive operations.
*ssh_cipher_spec* default|<cipher_spec>::
Selects the cipher specification for encrypting the session
(comma-separated list of ciphers in order of preference). See the
"-c cipher_spec" option in ssh(1) for more information. Defaults
to ``default'' (the ciphers specified in ssh_config(5)).
=== Data Stream Options
*stream_compress* <compress_command>|no::
Compress the btrfs send stream before transferring it from/to
remote locations. Defaults to ``no''. If enabled, make sure that
'<compress_command>' is available on the source and target
hosts. Supported '<compress_command>': gzip, pigz, bzip2, pbzip2,
bzip3, xz, lzo, lz4, zstd.
*stream_compress_level* default|<number>::
Compression level for the specified '<compress_command>'. Refer to
the related man-page for details (usually [1..9], where 1 means
fastest compression). Defaults to ``default'' (the default
compression level of '<compress_command>').
*stream_compress_long* default|<number>::
Enable long distance matching for the specified
'<compress_command>'. Refer to the related man-page for details.
Only supported for "zstd".
*stream_compress_threads* default|<number>::
Number of threads to use for <compress_command>. Only supported
for "pigz", "pbzip2", "bzip3", "zstd" and recent versions of "xz".
*stream_compress_adapt* yes|no::
Enable adaptive compression for <compress_command>. Only supported
for "zstd" (version >= 1.3.6). Defaults to ``no''.
*stream_buffer* <size>|no::
Add a buffer to the btrfs send stream (locally, on uncompressed
data), with a maximum size of '<size>'. This can give a speed
improvement (measured up to 20%) on both local or remote
operations, but also increases system load. A suffix of "k", "m",
"g", or "%" can be added to '<size>' to denote kilobytes (*1024),
megabytes, gigabytes, or a percentage of total physical
memory. Defaults to ``no''.
+
--
If enabled, make sure that the "mbuffer" command (at least version
20180505) is available on the host running btrbk. As of btrbk-0.29.0,
mbuffer(1) is used for both 'rate_limit' and 'stream_buffer' options:
mbuffer [-m <stream_buffer>] [-r <rate_limit>]
Note that mbuffer(1) always reads defaults from "`/etc/mbuffer.rc"`
and "`~/.mbuffer.rc`".
Leave this option disabled if your main concern is a stable backup
process: while recent versions of mbuffer have proven reliable, it is
often desirable to keep things simple rather than adding an
additional, multi-threaded process to the command pipe.
--
*stream_buffer_remote* <size>|no::
Add a buffer on remote hosts (either source or target). Defaults
to ``no''.
+
--
Enable this if you prefer buffering on the remote side, or even on
both sides: reasons for this depend on available memory, disk and cpu
performance (btrfs send/receive, compression), as well as networking
constraints.
--
*rate_limit* <rate>|no::
Limit the read rate of the btrfs send stream to '<rate>' bytes per
second (locally, on uncompressed send stream). A suffix of "k",
"m", "g", or "t" can be added to denote kilobytes (*1024),
megabytes, and so on. Defaults to ``no''. Note that 'rate_limit'
implicitly adds a stream buffer (see 'stream_buffer' option
above).
*rate_limit_remote* <rate>|no::
Add rate limit on remote hosts (either source or target). Defaults
to ``no''. Note that it usually does not make much sense to enable
both 'rate_limit' and 'rate_limit_remote'.
=== System Options
*transaction_log* <file>|no::
If set, all transactions (snapshot create, subvolume send-receive,
subvolume delete) as well as abort messages are logged to <file>,
in a space-separated table format: "localtime type status
target_url source_url parent_url message".
*transaction_syslog* <facility>|no::
If set, all transactions (as described in 'transaction_log' above)
are logged to syslog. The program name used in the messages is
"btrbk". Accepted parameters for '<facility>': user, mail,
daemon, auth, lpr, news, cron, authpriv, local0..local7.
*lockfile* <file>|no::
Place an exclusive lock on <file> during program execution, using
flock(2). If the lock is held by another process, exit before
running any actions. Ignored on dryrun ('-n', '--dry-run'). See
also '--lockfile' command-line option.
*backend* <backend>::
Backend filesystem utilities to be used for btrfs specific
operations. Available backends:
+
--
*btrfs-progs*::
Default backend, btrfs commands are called as specified in
btrfs(8) (e.g. "btrfs subvolume show").
*btrfs-progs-btrbk*::
btrfs commands are separated by a dash instead of a whitespace
(e.g. "btrfs-subvolume-show" instead of "btrfs subvolume
show"). Useful for setting suid or file capabilities (setcap) on
specific btrfs commands, as implemented in
<https://github.com/digint/btrfs-progs-btrbk>.
*btrfs-progs-sudo*::
btrfs commands are prefixed with "sudo -n" (e.g. "sudo -n btrfs
subvolume show" instead of "btrfs subvolume show"). Make sure to
have appropriate (root) permissions for the "btrfs" command groups
as well as the "readlink" and "test" commands in /etc/sudoers.
*btrfs-progs-doas*::
Similar to btrfs-progs-sudo, using prefix "doas -n".
If you want to set this option for local or remote hosts only, set
*backend_local* or *backend_remote* (e.g. "backend_remote
btrfs-progs-btrbk").
If you want to set this option for regular (non-root) user only, set
*backend_local_user*.
--
*compat* <compat-option>...::
Enable compatibility options. Available 'compat-option':
+
--
*busybox*::
Use busybox compatible commands, at the expense of slight overhead
while reading filesystem information.
*ignore_receive_errors* _*experimental*_::
Tell btrfs-receive(8) to not terminate on errors by setting
"--max-errors=0" option. Print warnings instead.
+
A known use case for this are target hosts lacking xattr support
(e.g. some Synology NAS), while the send-stream contains "lsetxattr"
commands. Another case is targets failing to set otime, complaining
with "ERROR: attribute 12 requested but not present".
+
Note that there is *no guarantee that backups created with this
option enabled can be restored at all*.
If you want to set this option for local or remote hosts only, set
*compat_local* or *compat_remote* (e.g. "compat_remote busybox").
--
*cache_dir* <directory>::
If set, cache extent maps for the "btrbk extents" command.
=== Btrfs Specific Options
*incremental_prefs* <list-spec>[:<amount>]...::
Specify the preferences to determine the best common (correlated)
parent and clone sources for incremental backups, by choosing from
predefined candidate lists.
+
--
The 'list-spec' defines from what candidate list the next
parent/clone-src should be appended to a result list; 'amount' defines
how many (e.g. "sro:1 sro:1" is identical to "sro:2"), or all if
omitted. Any candidate which is already in the results is dropped.
The resulting list of subvolumes is then used as parameters for the
btrfs-send(8) command: the first for "-p <parent>", all others for
"-c <clone-src>".
Available 'list-spec' (candidate lists = filtered subsets of
correlated subvolumes):
*sro*,*srn*:: All from 'snapshot_dir' matching 'snapshot_name', with
parent_uuid relationship, sorted by btrbk timestamp (o=older
n=newer).
*sao*,*san*:: All from 'snapshot_dir' matching 'snapshot_name', sorted
by btrbk timestamp (o=older n=newer).
*aro*,*arn*:: All from 'incremental_resolve', with parent_uuid
relationship, sorted by cgen (o=older n=newer).
Defaults to "sro:1 srn:1 sao:1 san:1 aro:1 arn:1". Note that for
most operations the default resolves a single parent, as there usually
are no newer snapshots, and all "sro:1 sao:1 aro:1" resolve to the
same snapshot.
Example: "defaults,sao,san,aro,arn" takes the defaults, and adds clone
sources for all (!) known candidates on the filesystem.
--
*incremental_clones* yes|no::
If enabled, btrbk adds "-c <clone-src>" to the btrfs-send(8)
command for all correlated subvolumes resolved by
'incremental_prefs'. If disabled, only "-p <parent>" is
used. Defaults to ``yes''.
*incremental_resolve* mountpoint|directory::
Specifies where to search for the best common parent for
incremental backups. If set to ``mountpoint'', use parents in the
filesystem tree below the mount point of the snapshot and target
directory. If set to ``directory'', use parents strictly below
snapshot/target directories. Set this to ``directory'' if you get
access problems (when not running btrbk as root). Defaults to
``mountpoint''.
*btrfs_commit_delete* yes|no::
If set, wait for the transaction commit at the end of each
snapshot or backup deletion (sets '--commit-each' option for
"btrfs subvolume delete"). Defaults to ``no''.
*send_protocol* <number>|no _*experimental*_::
Use btrfs send protocol version N. If enabled on 'target', btrbk
adds "--proto <number>" to the btrfs-send(8) command. Defaults to
``no'' (btrfs default).
*send_compressed_data* yes|no _*experimental*_::
Send data that is compressed on the filesystem directly without
decompressing it. This requires protocol version 2 or higher
(btrfs-progs >= 5.19), and implies "send_protocol 2". If enabled
on 'target', btrbk adds "--compressed-data" to the btrfs-send(8)
command. Defaults to ``no'' (btrfs default).
*snapshot_qgroup_destroy* yes|no _*experimental*_:: {blank}
*target_qgroup_destroy* yes|no _*experimental*_:: {blank}
*archive_qgroup_destroy* yes|no _*experimental*_::
Whenever a subvolume is deleted, also destroy corresponding
default qgroup "+0/<subvol-id>+". Only useful if you have enabled
btrfs quota support. See also:
<https://bugzilla.kernel.org/show_bug.cgi?id=91751>
=== Informative Options
*warn_unknown_targets* yes|no::
If set, prints a warning if btrbk encounters a target subvolume at
a unknown location (i.e. not following btrbk naming scheme, or
outside the target directory). Defaults to ``no''.
RETENTION POLICY
----------------
Retention policies are defined individually for snapshots, backups and
archives (summarized as "backups" in the following text), using a
combination of:
**_preserve_min* all|latest|no|<number>{h,d,w,m,y}::
Amount of time (duration) in which all backups are preserved.
**_preserve* no|<retention_policy>::
Schedule (single points in time) for which individual backups are
preserved.
Note that if "preserve_min" is set to ``all'' (the default), any
setting of "preserve" obviously has no effect.
The format for '<retention_policy>' is:
[<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]
*hourly*::
Defines how many hours back hourly backups should be
preserved. The first backup of an hour is considered an hourly
backup.
*daily*::
Defines how many days back daily backups should be preserved. The
first backup of a day (starting at 'preserve_hour_of_day') is
considered a daily backup.
*weekly*::
Defines how many weeks back weekly backups should be
preserved. The first daily backup created at
'preserve_day_of_week' (or the first backup in this week if none
was made on the exact day) is considered as a weekly backup.
*monthly*::
Defines how many months back monthly backups should be
preserved. Every first weekly backup in a month is considered a
monthly backup.
*yearly*::
Defines for how many years back yearly backups should be
preserved. Every first monthly backup in a year is considered a
yearly backup.
Use an asterisk for ``all'' (e.g. "target_preserve 60d *m"
states: "preserve daily backups for 60 days back, and all monthly
backups").
Hint: Run btrbk with the '-S', '--print-schedule' option to get a
comprehensive output of the scheduler results.
=== Reference Time
The local time on the host running btrbk defines the reference time
for all date/time calculations, especially for "beginning of a day",
and as a consequence for the first daily, weekly, monthly or yearly
backups. The local time on remote hosts (ssh source/target) is never
used.
Unless "timestamp_format long-iso" is set, daily backups are preserved
at "preserve_hour_of_day" (defaults to midnight) of the respective
time zone (and not for "00:00 UTC", which would be "14:00" in
Honolulu). This becomes relevant for setups with multiple btrbk
instances, e.g. many snapshot-only instances (spread around the
world), and a fetch-only instance on the backup server.
Caveat:
* If "timestamp_format long-iso" is set, each btrbk instance on has a
different interpretation of "first in day". Make sure to run btrbk
with the same time zone on every host, e.g. by setting the TZ
environment variable (see tzset(3)).
TARGET TYPES
------------
*send-receive*::
Backup to a btrfs filesystem, using "btrfs send/receive". This is
the recommended (standard) target type. The '<target-directory>'
must be an absolute path and point to a subvolume or directory
within a btrfs file system. See btrfs-send(8), btrfs-receive(8).
*raw* _*experimental*_::
Backup to a raw (filesystem independent) file from the output of
btrfs-send(8), with optional compression and encryption.
+
--
Raw backups consist of two files: the main data file containing the
btrfs send stream, and a sidecar file ".info" containing metadata:
<snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg]
<snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg].info
For 'incremental' backups ("incremental yes"), please note that:
* As soon as a single 'incremental' backup file is lost or corrupted,
all later incremental backups become invalid, as there is no common
parent for the subsequent incremental images anymore. This might be
a good compromise for a vacation backup plan, but for the long term
make sure that a non-incremental backup is triggered from time to
time.
* The scheduler will never delete dependent parents of backups
preserved by the retention policy (run btrbk with the '-S',
'--print-schedule' option to get a comprehensive output of the
scheduler results).
* There is currently no support for rotation of incremental backups:
if 'incremental' is set, a full backup must be triggered manually
from time to time in order to be able to delete old backups.
Additional options for raw targets:
*raw_target_compress* <compress_command>|no::
Compression algorithm to use for raw backup target. Supported
'<compress_command>': gzip, pigz, bzip2, pbzip2, bzip3, xz, lzo,
lz4, zstd.
*raw_target_compress_level* default|<number>::
Compression level for the specified <compress_command>.
*raw_target_compress_long* default|<number>::
Enable long distance matching for the specified
'<compress_command>'.
*raw_target_compress_threads* default|<number>::
Number of threads to use for <compress_command>.
*raw_target_split* <size>|no::
Split the raw backup file into pieces of size '<size>'.
*raw_target_block_size* <number>::
Block size to use for writing the raw backup file. Defaults to
``128K''.
*raw_target_encrypt* gpg|openssl_enc|no::
If enabled, encrypt the target raw file using gpg or openssl_enc.
Additional options for "raw_target_encrypt gpg":
*gpg_keyring* <file>::
Keyring to use for gpg, e.g. "`/etc/btrbk/gpg/pubring.kbx`".
*gpg_recipient* <name>...::
Encrypt for user id '<name>' (email address).
Additional options for "raw_target_encrypt openssl_enc" ('very
experimental'):
*openssl_ciphername*{nbsp}<name>::
Defaults to ``aes-256-cbc''.
*openssl_iv_size* <size-in-bytes>|no::
Depends on selected cipher.
*openssl_keyfile* <file>|no::
Point to a key file in hex (absolute path). Example key file
creation (256bit key):
+
------------
# dd if=/dev/urandom bs=1 count=32 \
| od -x -A n \
| tr -d "[:space:]" > /path/to/keyfile
------------
*kdf_backend* <file>|no::
KDF backend to be executed,
e.g. "`/usr/share/btrbk/scripts/kdf_pbkdf2.py`".
*kdf_keysize* <size-in-bytes>::
Defaults to ``32''.
*kdf_keygen* once|each::
Defaults to ``once''.
--
AVAILABILITY
------------
Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
for further details.
SEE ALSO
--------
*btrbk*(1)
AUTHOR
------
Axel Burri <axel@tty0.ch>

View File

@ -1,68 +0,0 @@
Installation
============
Btrbk is a single perl script, and does not require any special
installation procedures or libraries. In order to install the btrbk
executable along with the documentation and an example configuration
file, choose one of the following methods:
### Generic Linux System
Install [asciidoctor] or [asciidoc] if you want to build the
documentation.
Download and unpack the latest [btrbk source tarball] and type:
sudo make install
#### Try latest master from Github:
wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
chmod +x btrbk
sudo ./btrbk ls /
### Debian Based Distros
btrbk is in debian stable (utils): https://packages.debian.org/stable/utils/btrbk
Packages are also available via NeuroDebian: http://neuro.debian.net/pkgs/btrbk.html
### Fedora Linux
btrbk is in the official Fedora repos: https://src.fedoraproject.org/rpms/btrbk
sudo dnf install btrbk
### Arch Linux
btrbk is in AUR: https://aur.archlinux.org/packages/btrbk/
### Alpine Linux
btrbk is in the community repository
apk add btrbk
### Gentoo Linux
btrbk is in portage:
emerge app-backup/btrbk
### Void Linux
btrbk is in Void's `current` repository
xbps-install -S btrbk
[btrbk source tarball]: https://digint.ch/download/btrbk/releases/
[asciidoctor]: https://asciidoctor.org
[asciidoc]: https://asciidoc.org

View File

@ -1,126 +0,0 @@
lsbtr(1)
========
:date: 2023-03-25
:release-version: 0.32.6
:man manual: Btrbk Manual
:man source: Btrbk {release-version}
NAME
----
lsbtr - list btrfs subvolumes
SYNOPSIS
--------
[verse]
lsbtr [-h|--help] [--version]
[-l|--long] [-u|--uuid] [-1|--single-column] [--raw]
[--format <output-format>]
[-v|--verbose]
[-c|--config <file>]
[--override <config_option>=<value>]
[[--] <path>|<url>...]
DESCRIPTION
-----------
List btrfs subvolumes and their mount points visible by the file
system below '<path>'.
*lsbtr* is part of *btrbk* (basically a shortcut for "btrbk ls"), and
takes some global configuration options from btrbk.conf(5) if present.
*lsbtr* requires 'root privileges' to run correctly. Alternatively,
consider using "btrfs-progs-sudo" or "btrfs-progs-btrbk" backends,
both of which allows you to run lsbtr as a regular user. Refer to
configuration option 'backend' in btrbk.conf(5) for more details.
OPTIONS
-------
-h, --help::
Prints the synopsis and a list of the commands.
--version::
Prints the btrbk version.
-l, --long::
Print output in long table format (additionally print subvolume
path).
-u, --uuid::
Print UUID table (parent/received relations).
-1, --single-column::
Print path column only (delimited by newline).
--raw::
Print space-separated key="value" pairs (machine readable).
--format table|long|raw|col:[h:]<columns>::
Print output in specified format. If set to "raw", prints
space-separated key="value" pairs (machine readable).
+
If set to "col:", prints only the <columns> specified (comma-separated
list). Header lines are omitted if the "h:" modifier is present.
Columns prefixed with "-" are collapsed if empty. Columns postfixed
with ":RALIGN" are right-aligned.
-v, --verbose::
Increase the level of verbosity.
-c, --config <file>::
Read the configuration from <file>.
--override <config_option>=<value>::
Override a configuration option <config_option> with <value>.
FILES
-----
+/etc/btrbk.conf+::
+/etc/btrbk/btrbk.conf+::
Default configuration file. The file format and configuration
options are described in *btrbk.conf*(5).
EXIT STATUS
-----------
*lsbtr* returns the following error codes:
ifndef::backend-docbook,backend-manpage[]
[horizontal]
endif::backend-docbook,backend-manpage[]
0:: No problems occurred.
1:: Generic error code.
2:: Parse error: when parsing command-line options or configuration
file.
255:: Script error.
AVAILABILITY
------------
Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
for further details.
SEE ALSO
--------
*btrbk*(1),
*btrbk.conf*(5),
*btrfs*(8)
AUTHOR
------
Axel Burri <axel@tty0.ch>

96
doc/ssh_filter_btrbk.1 Normal file
View File

@ -0,0 +1,96 @@
.TH "ssh_filter_btrbk" "1" "2016-04-23" "btrbk v0.23.0" ""
.\" disable hyphenation
.nh
.\" disable justification (adjust text to left margin only)
.ad l
.SH NAME
ssh_filter_btrbk.sh \- ssh command filter script for btrbk
.SH SYNOPSIS
.nf
\fBssh_filter_btrbk.sh\fR [\-s|\-\-source] [\-t|\-\-target] [\-d|\-\-delete] [\-i|\-\-info]
[\-p|\-\-restrict\-path <path>] [\-l|\-\-log] [\-\-sudo]
.fi
.SH DESCRIPTION
\fBssh_filter_btrbk.sh\fR restricts SSH commands to \fIbtrfs\fR
commands used by \fIbtrbk\fR. It examines the SSH_ORIGINAL_COMMAND
environment variable (set by sshd) and executes it only if it matches
commands used by \fIbtrbk\fR. The accepted commands are specified by
the "\-\-source", "\-\-target", "\-\-delete" and "\-\-info" options.
.PP
Note that the following btrfs commands are always allowed: "btrfs
subvolume show", "btrfs subvolume list".
.PP
Example line in /root/.ssh/authorized_keys on a backup target host:
.PP
.RS 4
.nf
command="ssh_filter_btrbk.sh \-\-target \-\-delete \-\-restrict\-path /mnt/btr_backup" ssh\-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@mydomain.com
.fi
.RE
.SH OPTIONS
.PP
\-s, \-\-source
.RS 4
Allow commands for backup source: "btrfs subvolume snapshot", "btrfs
send". Equivalent to "\-\-snapshot \-\-send".
.RE
.PP
\-t, \-\-target
.RS 4
Allow commands for backup target: "btrfs receive", "realpath" and "cat
/proc/self/mounts".
.RE
.PP
\-d, \-\-delete
.RS 4
Allow commands for subvolume deletion: "btrfs subvolume delete". This
is used for backup source if \fIsnapshot_preserve_daily\fR is not set
to \[lq]all\[rq], and for backup targets if
\fItarget_preserve_daily\fR is not set to \[lq]all\[rq].
.RE
.PP
\-i, \-\-info
.RS 4
Allow informative commands: "btrfs subvolume find-new", "btrfs
filesystem usage". This is used by btrbk \fIinfo\fR and \fIdiff\fR
commands.
.RE
.PP
\-\-snapshot
.RS 4
Allow btrfs snapshot command: "btrfs subvolume snapshot".
.RE
.PP
\-\-send
.RS 4
Allow btrfs send command: "btrfs send".
.RE
.PP
\-\-receive
.RS 4
Allow btrfs receive command: "btrfs receive".
.RE
.PP
\-p, \-\-restrict\-path <path>
.RS 4
Restrict btrfs commands to <path>.
.RE
.PP
\-l, \-\-log
.RS 4
Log ACCEPT and REJECT messages to the system log.
.RE
.PP
\-\-sudo
.RS 4
Call SSH_ORIGINAL_COMMAND using sudo.
.RE
.SH AVAILABILITY
Please refer to the btrbk project page \fBhttp://digint.ch/btrbk/\fR
for further details.
.SH SEE ALSO
.BR btrbk (1),
.BR btrbk.conf (5),
.BR btrfs (1)
.SH AUTHOR
Axel Burri <axel@tty0.ch>

View File

@ -1,117 +0,0 @@
ssh_filter_btrbk(1)
===================
:date: 2023-03-25
:release-version: 0.32.6
:man manual: Btrbk Manual
:man source: Btrbk {release-version}
NAME
----
ssh_filter_btrbk - ssh command filter script for btrbk
SYNOPSIS
--------
[verse]
ssh_filter_btrbk.sh [-s|--source] [-t|--target] [-d|--delete]
[-i|--info] [--snapshot] [--send] [--receive]
[-p|--restrict-path <path>] [-l|--log] [--sudo]
DESCRIPTION
-----------
*ssh_filter_btrbk.sh* restricts SSH commands to commands used by
'btrbk'. It examines the SSH_ORIGINAL_COMMAND environment variable
(set by sshd) and executes it only if it contains commands used by
'btrbk'.
The accepted commands are specified by the "--source", "--target",
"--delete" and "--info" options.
The following commands are always allowed:
- "btrfs subvolume show" (not affected by "--restrict-path")
- "btrfs subvolume list" (not affected by "--restrict-path")
- "readlink"
- "test -d" (only if "compat busybox" configuration option is set)
- "cat /proc/self/mountinfo"
- pipes through "gzip", "pigz", "bzip2", "pbzip2", "bzip3", "xz",
"lzop", "lz4", "zstd" (stream_compress)
- pipes through "mbuffer" (stream_buffer, rate_limit)
Example line in /root/.ssh/authorized_keys on a backup target host:
command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup",restrict ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@example.org
OPTIONS
-------
-s, --source::
Allow commands for backup source: "btrfs subvolume snapshot",
"btrfs send". Equivalent to "--snapshot --send".
-t, --target::
Allow commands for backup and archive target: "btrfs receive",
"mkdir".
-d, --delete::
Allow commands for subvolume deletion: "btrfs subvolume
delete". This is used for backup source if
'snapshot_preserve_daily' is not set to ``all'', and for backup
targets if 'target_preserve_daily' is not set to ``all''.
-i, --info::
Allow informative commands: "btrfs subvolume find-new", "btrfs
filesystem usage". This is used by btrbk 'info' and 'diff'
commands.
--snapshot::
Allow btrfs snapshot command: "btrfs subvolume snapshot".
--send::
Allow btrfs send command: "btrfs send".
--receive::
Allow btrfs receive command: "btrfs receive".
-p, --restrict-path <path>::
Restrict commands to <path>. Note that "btrfs subvolume show",
"btrfs subvolume list" are NOT affected by this option.
+
It is not possible to restrict commands to exact subvolume names, as
btrfs-receive(8) takes a <path> as argument (directory, not including
the subvolume file name to be created, this is encoded in the
send-stream).
-l, --log::
Log ACCEPT and REJECT messages to the system log.
--sudo::
Allow btrfs commands to be called via sudo. Enable this if you
have "backend btrfs-progs-sudo" in your btrbk configuration file.
AVAILABILITY
------------
Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
for further details.
SEE ALSO
--------
*btrbk*(1),
*btrbk.conf*(5),
*btrfs*(8)
AUTHOR
------
Axel Burri <axel@tty0.ch>

92
doc/upgrade_to_v0.23.0.md Normal file
View File

@ -0,0 +1,92 @@
Upgrading to btrbk-v0.23.0
==========================
In order to keep btrbk simple and intuitive while adding new features,
it became inevitable to change the semantics of the "retention policy"
related configuration options.
What has changed?
-----------------
### Preserve *first* instead of *last* snapshot/backup
btrbk used to *always* transfer the latest snapshot to the target
location, while considering the *last* snapshot/backup of a day as a
daily backup (and also the last weekly as a monthly). This made it
very cumbersome when running btrbk in a cron job as well as manually,
because the last manually created snapshot was immediately transferred
on every run, and used as the daily backup (instead of the one created
periodically by the cron job).
The new semantics are to consider the *first* (instead of *last*)
snapshot of a hour/day/week/month as the one to be preserved, while
only transferring the snapshots needed to satisfy the target retention
policy.
### Preserve snapshots for a minimum amount of time
In order to specify a minimum amount of time in which *all* snapshots
should be preserved, the new "snapshot_preserve_min" and
"target_preserve_min" configuration options were introduced. This was
previously covered by "snapshot_preserve_daily", which caused a lot of
confusion among users.
Upgrading the configuration file: /etc/btrbk/btrbk.conf
-------------------------------------------------------
Please read the description of the "run" command in [btrbk(1)], as
well as the "RETENTION POLICY" section in [btrbk.conf(5)] for a
detailed description. Make sure to understand the new concept, and run
`btrbk --print-schedule dryrun` after updating the configuration.
### Upgrade retention policy
If you want the same behaviour as before:
# replace this:
snapshot_preserve_daily <daily>
snapshot_preserve_weekly <weekly>
snapshot_preserve_monthly <monthly>
# with:
snapshot_preserve_min <daily>d
snapshot_preserve <weekly>w <monthly>m
# ... do the same with "target_preserve_*" options
But what you probably want is something like:
snapshot_preserve_min 5d
snapshot_preserve <daily>d <weekly>w <monthly>m
target_preserve_min no
target_preserve <daily>d <weekly>w <monthly>m *y
This states:
* Keep all snapshots for five days (no matter how many there are)
* Transfer only the first snapshot of a day to the target
* Keep all "first snapshots of a day" for `<daily>` days, etc.
### Upgrade "resume_missing"
If you have a line: "resume_missing yes" somwhere in your config,
simply remove it. btrbk always resumes missing backups.
If you have "resume_missing no", you can imitate this behaviour by
setting:
target_preserve_min latest
target_preserve no
This states: "always transfer the latest snapshot to the target".
[btrbk(1)]: http://digint.ch/btrbk/doc/btrbk.html
[btrbk.conf(5)]: http://digint.ch/btrbk/doc/btrbk.conf.html

1
lsbtr
View File

@ -1 +0,0 @@
btrbk

View File

@ -1,142 +1,92 @@
#!/bin/sh
#!/bin/bash
# initialise and sanitise the shell execution environment
unset -v IFS
export LC_ALL=C
export PATH='/sbin:/bin:/usr/sbin:/usr/bin'
set -e
set -u
set -e -u
export PATH=/sbin:/bin:/usr/sbin:/usr/bin
enable_log=
use_sudo=
restrict_path_list=
allow_list=
allow_exact_list=
allow_rate_limit=1
allow_stream_buffer=1
allow_compress=1
compress_list='gzip|pigz|bzip2|pbzip2|bzip3|xz|lzop|lz4|zstd'
# note that the backslash is NOT a metacharacter in a POSIX bracket expression!
option_match='-[a-zA-Z0-9=-]+' # matches short as well as long options
file_match_sane='/[0-9a-zA-Z_@+./-]*' # matches file path (equal to ${file_match} in btrbk < 0.32.0)
file_match="/[^']*" # btrbk >= 0.32.0 quotes file arguments: match all but single quote
file_arg_match="('${file_match}'|${file_match_sane})" # support btrbk < 0.32.0
log_cmd()
{
local priority="$1"
local authorisation_decision="$2"
local reason="${3-}"
if [ -n "${enable_log}" ]; then
logger -p "${priority}" -t ssh_filter_btrbk.sh "${authorisation_decision} (Name: ${LOGNAME:-<unknown>}; Connection: ${SSH_CONNECTION:-<unknown>})${reason:+: ${reason}}: ${SSH_ORIGINAL_COMMAND}"
if [[ -n "$enable_log" ]]; then
logger -p $1 -t ssh_filter_btrbk.sh "$2 (Name: ${LOGNAME:-<unknown>}; Remote: ${SSH_CLIENT:-<unknown>})${3:+: $3}: $SSH_ORIGINAL_COMMAND"
fi
}
allow_cmd()
{
local cmd="$1"
allow_list="${allow_list}|${cmd}"
allow_list="${allow_list}|$1"
}
allow_exact_cmd()
{
local cmd="$1"
allow_exact_list="${allow_exact_list}|${cmd}"
allow_exact_list="${allow_exact_list}|$1"
}
reject_and_die()
{
local reason="$1"
log_cmd 'auth.err' 'btrbk REJECT' "${reason}"
printf 'ERROR: ssh_filter_btrbk.sh: ssh command rejected: %s: %s\n' "${reason}" "${SSH_ORIGINAL_COMMAND}" >&2
exit 255
local reason=$1
log_cmd "auth.err" "btrbk REJECT" "$reason"
echo "ERROR: ssh_filter_btrbk.sh: ssh command rejected: $reason: $SSH_ORIGINAL_COMMAND" 1>&2
exit 1
}
run_cmd()
{
log_cmd 'auth.info' 'btrbk ACCEPT'
eval " ${SSH_ORIGINAL_COMMAND}"
log_cmd "auth.info" "btrbk ACCEPT"
$use_sudo $SSH_ORIGINAL_COMMAND
}
reject_filtered_cmd()
{
if [ -n "${restrict_path_list}" ]; then
# match any of restrict_path_list,
# note that the backslash is NOT a metacharacter in a POSIX bracket expression!
option_match='-[a-zA-Z-]+' # matches short as well as long options
file_match='[0-9a-zA-Z_@+./-]+' # matches file path (equal to $file_match in btrbk)
if [[ -n "$restrict_path_list" ]]; then
# match any of restrict_path_list with or without trailing slash,
# or any file/directory (matching file_match) below restrict_path
path_match="'(${restrict_path_list})(${file_match})?'"
path_match_legacy="(${restrict_path_list})(${file_match_sane})?"
path_match="(${restrict_path_list})(/|/${file_match})?"
else
# match any absolute file/directory (matching file_match)
path_match="'${file_match}'"
path_match_legacy="${file_match_sane}"
fi
# btrbk >= 0.32.0 quotes files, allow both (legacy)
path_match="(${path_match}|${path_match_legacy})"
if [ -n "${allow_compress}" ]; then
decompress_match="(${compress_list}) -d -c( -[pT][0-9]+)?"
compress_match="(${compress_list}) -c( -[0-9])?( -[pT][0-9]+)?"
else
decompress_match=
compress_match=
fi
# rate_limit_remote and stream_buffer_remote use combined
# "mbuffer" as of btrbk-0.29.0
if [ -n "${allow_stream_buffer}" ] || [ -n "${allow_rate_limit}" ]; then
mbuffer_match='mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?'
else
mbuffer_match=
path_match="/${file_match}"
fi
# allow multiple paths (e.g. "btrfs subvolume snapshot <src> <dst>")
allow_cmd_match="(${allow_list})( ${option_match})*( ${path_match})+"
stream_in_match="(${decompress_match} \| )?(${mbuffer_match} \| )?"
stream_out_match="( \| ${mbuffer_match})?( \| ${compress_match}$)?"
btrfs_cmd_match="^(${allow_list})( ${option_match})*( $path_match)+$"
# `grep`s `-q`-option is not used as it may cause an exit status of `0` even
# when an error occurred.
allow_stream_match="^${stream_in_match}${allow_cmd_match}${stream_out_match}"
if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${allow_stream_match}" >/dev/null 2>/dev/null; then
if [[ $SSH_ORIGINAL_COMMAND =~ $btrfs_cmd_match ]] ; then
return 0
fi
exact_cmd_match="^(${allow_exact_list})$";
if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${exact_cmd_match}" >/dev/null 2>/dev/null; then
exact_cmd_match="^${allow_exact_list}$";
if [[ $SSH_ORIGINAL_COMMAND =~ $exact_cmd_match ]] ; then
return 0
fi
local formatted_restrict_path_list="$(printf '%s' "${restrict_path_list}" | sed 's/|/", "/g')"
reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${formatted_restrict_path_list}\")}"
reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${restrict_path_list//|/\", \"}\")}"
}
# check for "--sudo" option before processing other options
sudo_prefix=
for key in "$@"; do
if [ "${key}" = '--sudo' ]; then
sudo_prefix='sudo -n '
fi
if [ "${key}" = '--doas' ]; then
sudo_prefix='doas -n '
fi
done
while [ "$#" -ge 1 ]; do
allow_cmd "btrfs subvolume show"; # subvolume queries are always allowed
allow_cmd "btrfs subvolume list"; # subvolume queries are always allowed
while [[ "$#" -ge 1 ]]; do
key="$1"
case "${key}" in
case $key in
-l|--log)
enable_log=1
;;
--sudo|--doas)
# already processed above
--sudo)
use_sudo="sudo"
;;
-p|--restrict-path)
@ -145,75 +95,65 @@ while [ "$#" -ge 1 ]; do
;;
-s|--source)
allow_cmd "${sudo_prefix}btrfs subvolume snapshot"
allow_cmd "${sudo_prefix}btrfs send"
allow_cmd "btrfs subvolume snapshot"
allow_cmd "btrfs send"
;;
-t|--target)
allow_cmd "${sudo_prefix}btrfs receive"
allow_cmd "${sudo_prefix}mkdir"
;;
-c|--compress)
# deprecated option, compression is always allowed
allow_cmd "btrfs receive"
# the following are needed if targets point to a directory
allow_cmd "realpath"
allow_exact_cmd "cat /proc/self/mounts"
;;
-d|--delete)
allow_cmd "${sudo_prefix}btrfs subvolume delete"
allow_cmd "btrfs subvolume delete"
;;
-i|--info)
allow_cmd "${sudo_prefix}btrfs subvolume find-new"
allow_cmd "${sudo_prefix}btrfs filesystem usage"
allow_cmd "btrfs subvolume find-new"
allow_cmd "btrfs filesystem usage"
;;
--snapshot)
allow_cmd "${sudo_prefix}btrfs subvolume snapshot"
allow_cmd "btrfs subvolume snapshot"
;;
--send)
allow_cmd "${sudo_prefix}btrfs send"
allow_cmd "btrfs send"
;;
--receive)
allow_cmd "${sudo_prefix}btrfs receive"
allow_cmd "btrfs receive"
;;
*)
printf 'ERROR: ssh_filter_btrbk.sh: failed to parse command line option: %s\n' "${key}" >&2
exit 255
echo "ERROR: ssh_filter_btrbk.sh: failed to parse command line option: $key" 1>&2
exit 1
;;
esac
shift
done
# NOTE: subvolume queries are NOT affected by "--restrict-path":
# btrbk also calls show/list on the mount point of the subvolume
allow_exact_cmd "${sudo_prefix}btrfs subvolume (show|list)( ${option_match})* ${file_arg_match}";
allow_cmd "${sudo_prefix}readlink" # resolve symlink
allow_exact_cmd "${sudo_prefix}test -d ${file_arg_match}" # check directory (only for compat=busybox)
allow_exact_cmd 'cat /proc/self/mountinfo' # resolve mountpoints
allow_exact_cmd 'cat /proc/self/mounts' # legacy, for btrbk < 0.27.0
# remove leading "|" on alternation lists
allow_list="${allow_list#\|}"
allow_exact_list="${allow_exact_list#\|}"
restrict_path_list="${restrict_path_list#\|}"
allow_list=${allow_list#\|}
allow_exact_list=${allow_exact_list#\|}
restrict_path_list=${restrict_path_list#\|}
case "${SSH_ORIGINAL_COMMAND}" in
*\.\./*) reject_and_die 'directory traversal' ;;
*'
'*) reject_and_die 'unsafe character LF' ;;
*\$*) reject_and_die 'unsafe character "$"' ;;
*\&*) reject_and_die 'unsafe character "&"' ;;
*\(*) reject_and_die 'unsafe character "("' ;;
*\{*) reject_and_die 'unsafe character "{"' ;;
*\;*) reject_and_die 'unsafe character ";"' ;;
*\<*) reject_and_die 'unsafe character "<"' ;;
*\>*) reject_and_die 'unsafe character ">"' ;;
*\`*) reject_and_die 'unsafe character "`"' ;;
*\|*) [ -n "${allow_compress}" ] || [ -n "${allow_rate_limit}" ] || [ -n "${allow_stream_buffer}" ] || reject_and_die 'unsafe character "|"' ;;
case "$SSH_ORIGINAL_COMMAND" in
*\$*) reject_and_die "unsafe character" ;;
*\&*) reject_and_die "unsafe character" ;;
*\(*) reject_and_die "unsafe character" ;;
*\{*) reject_and_die "unsafe character" ;;
*\;*) reject_and_die "unsafe character" ;;
*\<*) reject_and_die "unsafe character" ;;
*\>*) reject_and_die "unsafe character" ;;
*\`*) reject_and_die "unsafe character" ;;
*\|*) reject_and_die "unsafe character" ;;
*\.\./*) reject_and_die "directory traversal" ;;
*)
reject_filtered_cmd
run_cmd
;;
esac
reject_filtered_cmd
run_cmd