Commit Graph

487 Commits (f5dc4e0a36a7f12ee18a3250d908e76042a8f42a)

Author SHA1 Message Date
Axel Burri f5dc4e0a36 btrbk: add known mountpoints to btr_tree nodes as anchor for reverse lookup 2018-05-10 11:48:34 +02:00
Axel Burri e9374b3b1d btrbk: replace url_cache by spec_cache 2018-05-10 11:48:34 +02:00
Axel Burri 0ea0430c43 btrbk: cleanup (cosmetics, documentation) 2018-05-10 11:48:34 +02:00
Axel Burri b37ef84e36 btrbk: always read mountpoints; include all snapshots from mountpoint as candidates for best common parent
Dropped readin of subvolid and realpath by btrfs_subvolume_show(), we
now always read /proc/self/mounts (and call readlink).

When picking the best common parent in get_best_parent(), we want to
list as many snapshots as possible. For now, we list all from the
mountpoint of snaproot ($sroot/<snapshot_dir>), due to a bug in
btrfs-progs [1]. Also added code (commented out) to list snapshots
from all known mountpoints.

  [1] https://github.com/kdave/btrfs-progs/issues/96
2018-05-10 11:48:05 +02:00
Axel Burri b549e11b43 btrbk: raw targets: move tree readin to separate function; add caching
Adds new function vinfo_init_raw_root(), similar to vinfo_init_root().
Note that we don't (yet) resolve realpath for raw targets.
2018-05-10 11:36:21 +02:00
Axel Burri 7a1bc257c1 btrbk: raw targets: create fake btr_tree instead of maintaining separate list 2018-05-10 11:36:21 +02:00
Axel Burri 6c502cbdcc btrbk: search complete target tree for correlated subvolumes
- move matching for correlated subvolumes from get_receive_targets
  into new function _receive_target_nodes
- add lookup tables in btr_tree (RECEIVED_UUID_HASH, UUID_HASH),
  allowing for faster matching in _receive_target_nodes
- add vinfo_resolved() for mapping nodes to vinfo
- rename get_latest_common to get_best_parent (while moving some
  functionality to new function get_related)
- cleanup
2018-05-10 11:36:21 +02:00
Axel Burri 490f680f41 btrbk: bugfix: add backend mappings for "btrfs filesystem usage" 2018-04-30 14:15:30 +02:00
Axel Burri 719fb5fb74 btrbk: bugfix: correct scheduling of "first weekly backup in month/year"
In the scheduler, a month (or year) does not start at the first day,
but at the first `preserve_day_of_week`. Make sure that all days
before `preserve_day_of_week` in a month get delta_months+1.

Example (corner case):
  - `preserve_day_of_week  sunday`
  - `target_preserve  *m`
  - no backups in 2018-02
  - backup with timestamp 2018-03-01 (which is a thursday)
  - backup with timestamp 2018-03-04 (which is a sunday)

Without this patch, because there are no sunday backups in 2018-02,
the first backup is considered a weekly (+4d after sunday), and as
such "first weekly of month 2018-03", and the second one is discarded.

With this patch, the first item is considered "first weekly of month
2018-02", and the second gets "first weekly of month 2018-03".

NOTE: This change may result in (previously preserved) backups to be
deleted!
2018-04-05 18:00:29 +02:00
Axel Burri e4a0938712 btrbk: ignore "preserve_hour_of_day" on snapshots/backups having "timestamp_format=short"
Snapshots and backups having no exact time information (created with
"timestamp_format=short") are set to 00:00, which would be regarded as
"previous day" if preserve_hour_of_day is greater than 0. Fix this by
ignoring preserve_hour_of_day in this case.
2018-04-05 18:00:29 +02:00
Axel Burri c89ffd4213 btrbk: schedule: format preserve text only if needed
No functional changes, improves performance of schedule().
2018-04-05 18:00:24 +02:00
Thiodwitnir 5791d72171 btrbk: add "preserve_hour_of_day" configuration option
Introduces the new config option "preserve_hour_of_day" to specify
after what time backups should be considered as dailies.

Based on pull request #204, with changes:
 - calculation of weekly backups
 - change format of preserve_matrix
2018-04-05 16:37:31 +02:00
Axel Burri 951ae256fa btrbk: bump version to 0.27.0-dev 2018-04-05 16:26:57 +02:00
Axel Burri 1fd3a547df change version to 0.26.1; bump copyright year 2018-03-05 11:55:56 +01:00
Axel Burri 6f38536052 btrbk: set c_default for config keys with computed default
Suppress "Option redefined" warning for snapshot_name config option,
which has hardcoded (computed) default already set when checking.

fix regression: 0ebe2ea2e1
2018-02-26 16:06:12 +01:00
Axel Burri e75cf0c72c btrbk: fix exit status and transaction log when archive_exclude is set
Similar to ABORTED=USER_SKIP (active commandline filter), archives
having ABORTED=ARCHIVE_EXCLUDE_SKIP (active archive_exclude
configuration) do not cause exit status 10 and are hidden from
transaction log.
2018-02-26 14:35:26 +01:00
Axel Burri 5651c96e4e btrbk: archive: always print "[-]" if no action was performed for target 2018-02-14 00:18:44 +01:00
Axel Burri 8610e75459 btrbk: add archive_exclude configuration option
Support wildcard characters, matches against both "$sroot->{PATH}" and
"$sroot->{PATH}/$snapshot_name".
2018-02-13 22:50:23 +01:00
Axel Burri 0ebe2ea2e1 btrbk: add "allow_multiple" flag for config_options declaration; use for option "group" 2018-02-13 19:30:24 +01:00
Axel Burri 3aaafa3d88 btrbk: add "meta" context for config defaults
Preparatory patch to allow multiple identical config keys in same
section.
2018-02-13 19:28:38 +01:00
Axel Burri b2b43cf199 btrbk: add configurable error_statement for check functions
Corrects error messages from --exclude option.
Applies to in check_file(), check_url() and append_config_option().
2018-02-13 17:40:14 +01:00
Axel Burri db1fe2d11a btrbk: fix output of "btrbk list backups" and "btrbk stats" (match all uuid/received_uuid combinations) 2018-02-07 16:24:49 +01:00
Axel Burri a3641cff74 btrbk: fix filter statement matching for volume="/"
While $vol->{URL} can contain "//" if volume="/" (intentionally, this
is an assembled path), the filter statements are sanitized using
check_url(). This means we need to match the filter statement against
check_url($vol->{URL}). Same applies to subvol.
2018-02-03 12:55:21 +01:00
Axel Burri 90fed6525e btrbk: bugfix: restrict snapshot_dir and snapshot_create to root/volume/subvolume context 2018-01-31 19:01:00 +01:00
Axel Burri 753d68b2ed btrbk: add documentation on caching in btr_tree() 2018-01-30 14:02:36 +01:00
Axel Burri 84bac59ab4 btrbk: fix parsing of "openssl_iv_size" configuration option 2018-01-21 18:53:29 +01:00
Axel Burri 13f37491ed btrbk: cosmetics: change order of commands in help message 2017-10-12 17:08:46 +02:00
Axel Burri 79cbf7c0c3 change version to 0.26.0 2017-10-12 00:39:06 +02:00
Axel Burri 909c68f164 btrbk: transaction_log and transaction_syslog are only allowed in root context 2017-10-11 20:54:41 +02:00
Axel Burri d941ecfce9 documentation: change digint links to https:// 2017-10-11 20:54:41 +02:00
Axel Burri e3ee674085 btrbk: limit search depth to 256 when resolving ancestors
Note that the current implementation is not very optimized, we should
introduce a parent/child hash table for all queries in
get_latest_common().
2017-10-10 16:46:05 +02:00
Axel Burri 3be65b9f67 btrbk: resolve ancestors (recursive on parent_uuid chain) when searching for latest common subvolume 2017-10-10 13:10:33 +02:00
Axel Burri 0799820556 btrbk: support btrfs-progs 4.13.2: adapt parsing of "btrfs subvolume list"
btrfs-progs v4.13.2 changed the output of "btrfs subvolume list", in
order to make the tabular output format look nicer (?!?):

e4c6772f69
2017-10-09 23:04:07 +02:00
Axel Burri e9a517f161 btrbk: add options "{snapshot,target,archive}_qgroup_destroy": destroy qgroups whenever a subvolume is deleted
Btrfs does not destroy qgroups when subvolumes are deleted (see
https://bugzilla.kernel.org/show_bug.cgi?id=91751). As a workaround
for this, btrbk can be configured to always destroy the corresponding
default qgroup "0/<subvol-id>" whenever a subvolume (snapshot, backup
or archive) is deleted.

Added configuration options:
 - snapshot_qgroup_destroy
 - target_qgroup_destroy
 - archive_qgroup_destroy
2017-10-02 16:23:23 +02:00
Axel Burri 6cb98700d8 btrbk: allow archive_preserve_* options in root context only 2017-10-02 13:54:37 +02:00
Axel Burri 2e974c1f4f btrbk: add "--wipe" command line option 2017-09-28 14:17:00 +02:00
Axel Burri 5bdc3e527b btrbk: add "prune" command 2017-09-28 14:17:00 +02:00
Axel Burri 2809dc54c5 btrbk: remove sorting of deleted subvolumes in summary (not needed anymore)
Deleted subvolumes are already sorted by schedule().
Sorting by PATH is bad anyways, as it does not sort _NN postfix
correctly.
2017-09-28 14:17:00 +02:00
Axel Burri 512aca5332 btrbk: parse output of "btrfs subvolume delete"
When doing a batch delete (multiple deletes with one call to "btrfs
subvolume delete"), we want to know which subvolumes have failed. For
this, we need parse the error output.

On any parsing failure, we assume that nothing has been deleted, and
warn accordingly (forward compatibility).
2017-09-28 14:17:00 +02:00
Axel Burri 9d9527ca9a btrbk: simplify transaction function calls
Prefix transaction status with "dryrun_" in start_transaction() /
end_transaction if $dryrun is set.
2017-09-28 14:17:00 +02:00
Axel Burri 422d52c063 btrbk: add key derivation for encrypted raw targets using external backend 2017-09-28 14:17:00 +02:00
Axel Burri de7628ac7c btrbk: add openssl_enc encryption for raw targets; add system_urandom()
Example:

Manually create a key:

    # KEYFILE=/some/secure/place/btrbk.key
    # dd if=/dev/urandom bs=1 count=32 | od -x -A n | tr -d "[:space:]" > $KEYFILE

btrbk.conf:

    volume /mnt/btr_pool
      incremental no
      raw_target_encrypt  openssl_enc
      openssl_ciphername  aes-256-cbc
      openssl_iv_size     16  # NOTE: set to "no" if no IV is needed by the selected cipher
      openssl_keyfile     /some/secure/place/btrbk.key

      subvolume home
        target raw ssh://cloud.example.com/backup
2017-09-28 14:17:00 +02:00
Axel Burri 251c2fb2a1 btrbk: re-enable parsing of deprecated raw file format (uuid suffix) 2017-09-28 14:17:00 +02:00
Axel Burri e804930b5e btrbk: allow deletion of non-incremental raw targets
implemented directly in btrfs_subvolume_delete()
2017-09-28 14:17:00 +02:00
Axel Burri cd8d7e3a0a btrbk: use sidecar file "*.info" instead of encoding uuids into filename for raw targets
pros:

 - better forward compatibility, e.g. symmetrical encryption
 - better readability of files

cons:

 - two files per backup
2017-09-28 14:16:53 +02:00
Axel Burri 571dae4428 btrbk: do not run in perl taint mode by default: remove "perl -T" in hashbang; hardcode $PATH only if taint mode is enabled
While taint mode [1] is a nice feature of perl, e.g. it disallows
using variables (such as filenames from the config file) which were
not validated in system() commands, it also treats $PATH as insecure
(which inherently is, as perl cannot know who messed around with it).

  [1] perlsec(1): http://perldoc.perl.org/perlsec.html
  [2] perlrun(1): http://perldoc.perl.org/perlrun.html

Note that btrbk still does all taint checks, and can be run in taint
mode:

  - by executing `perl -T /usr/sbin/btrbk`,
  - or by changing the hashbang to: `!#/usr/bin/perl -T`.
2017-09-25 17:53:35 +02:00
Axel Burri 5f867c2347 btrbk: run_cmd(): catch all possible errors when executing system command 2017-09-25 15:00:40 +02:00
Axel Burri e402435dc8 btrbk: treat deprecated warnings as non-fatal
Despite FATAL warnings are discouraged in perl and may break forward
compatibility [1], we still use it as btrbk is usually run as root and
we really want perl to die on programmatic errors.

  [1] "perldoc warnings"
2017-09-25 15:00:40 +02:00
Axel Burri 6c2d56d621 btrbk: action "clean": dont list failed deletes in summary 2017-09-21 12:39:48 +02:00
Axel Burri f113436626 btrbk: bugfix: check path when expanding wildcards (warn and ignore if check fails)
Before this patch, btrbk errored (unsafe command), and then died.
Fixes #181.
2017-09-11 18:49:14 +02:00