From 11e0450346b22ef1e02f2fcbac2888ce5c8f25bd Mon Sep 17 00:00:00 2001 From: jdrch Date: Wed, 11 Sep 2019 00:07:44 -0500 Subject: [PATCH 1/2] Update restic.md Fixed instructions for creating password file environment variable. Previous version actually didn't work. Updated crontab example to match example line given in merge commit message (Thanks!) --- restic.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/restic.md b/restic.md index c7dc958..15181b9 100644 --- a/restic.md +++ b/restic.md @@ -117,8 +117,18 @@ It is important that only root can read the file so that the password isn't comp Automated backups require the corresponding respository password without the user manually entering it. `restic` does this by [reading an environment variable whose value is the password file's path](https://restic.readthedocs.io/en/latest/faq.html?highlight=password#how-can-i-specify-encryption-passwords-automatically). Create the environment variable by: -1. In QTerminal, run `export RESTIC_PASSWORD_FILE=/root/PasswordFilename` -2. Test that the above command worked by running `$RESTIC_PASSWORD_FILE`. It should return `/root/PasswordFilename` +1. In QTerminal, run `sudo lte ~/.zshrc` +2. On a new line in the window that pops up, enter: + +``` +# Export restic password file environment variable +export RESTIC_PASSWORD_FILE=/root/PasswordFilename +``` + +in a new line + +3. In QTerminal, run `source ~/.zshrc` to update the shell with the new exported environment variable +4. Test that the above command worked by running `$RESTIC_PASSWORD_FILE`. It should return `/root/PasswordFilename` **STEP 13: Construct your backup command** @@ -138,7 +148,7 @@ A couple details about 2.1: * /*n*, where *n* is a number, does not work in the **Minutes** field. Without going into details, just avoid it * The `@`*n*`s` syntax, where *n* is a whole number, is much easier to understand than the individual fields. It ensures that each successive invocation happens *n* seconds *after the previous one has completed*. This means tasks in the same crontab entry never collide. The main drawback is it's more difficult to set jobs based on absolute calendar date and time -An example of a crontab schedule is `0 14 * * *` (tab separated), which translates to: +Crontab schedules consist of five numeric or asterisk `*` values separated by tabs. For example, `0 14 * * *` translates to: * Every time the system clock time value is 0 minutes, 14 hours (represented by `0 14`, evaluates to 14:00/2:00 PM) ... * Regardless of the day of the month, the month, or the day of the week (what `* * *` stand for, respectively) From 9ae2ab70a1ab40fde9a0a252acef5f1735f9002e Mon Sep 17 00:00:00 2001 From: jdrch Date: Fri, 13 Sep 2019 22:33:23 -0500 Subject: [PATCH 2/2] Update restic.md Added proper headings and changed `-p` option argument to be an absolute path to password file. For some reason the environment variable option isn't working reliably in Trident. --- restic.md | 79 ++++++++++++++++++++++--------------------------------- 1 file changed, 31 insertions(+), 48 deletions(-) diff --git a/restic.md b/restic.md index 15181b9..c159ea8 100644 --- a/restic.md +++ b/restic.md @@ -2,7 +2,7 @@ Author: [jdrch](https://github.com/jdrch) Last updated: 2019-09-09 -**How to set up regular recurring, incremental, online filesystem backups using `restic`** +# How to set up regular recurring, incremental, online filesystem backups using `restic` *What this guide will get you:* @@ -18,7 +18,7 @@ Last updated: 2019-09-09 * Instantaneous snapshots. See [`zfsnap`](https://github.com/project-trident/trident-docs/blob/master/zfssnap.md) for that * Device-level backup. For that, try [Bareos](https://www.bareos.org/en/), [Bacula](https://www.bacula.org/), or [Amanda](http://www.amanda.org/). No claim about those tools is made or implied by this statement -**STEP 0: Get the right mindset** +## STEP 0: Get the right mindset Be prepared to NOT understand things at first, but also be patient. Eventually you'll get the hang of what you're doing. @@ -26,11 +26,11 @@ Snapshots are just backups and `restic` operates on those exclusively, so there This guide is written to enable people unfamiliar basic programming concepts and who do better with GUIs than CLIs to understand how the commands involved work. Too many Linux/Unix guides assume a lot of prior knowledge for which no easily accessible corresponding guide or documentation exists. This leaving users struggling to understand what they need to do to get their desired effects. -**STEP 1: Read the `restic` documentation** +## STEP 1: Read the `restic` documentation Read the [`restic`](https://restic.readthedocs.io) documentation. -**STEP 2: Read FreeBSD's crontab and ZFS documentation** +## STEP 2: Read FreeBSD's crontab and ZFS documentation While `restic` creates and automatically prunes backups, this guide uses `cron` to invoke it at preset intervals. As such, you'll have to be familiar with crontab syntax to create the `restic` run schedule you want. crontab syntax varies based on OS implementation, and Project Trident (ultimately) [uses FreeBSD's implementation thereof](https://t.me/ProjectTrident/33871). @@ -41,7 +41,7 @@ Read: 3. The [`zpool` man page](https://www.freebsd.org/cgi/man.cgi?query=zpool&sektion=8&apropos=0&manpath=FreeBSD+12.0-RELEASE+and+Ports) 4. The [`zfs` man page](https://www.freebsd.org/cgi/man.cgi?query=zpool&sektion=8&apropos=0&manpath=FreeBSD+12.0-RELEASE+and+Ports) -**STEP 3: Read the ZFS Administration Guide** +## STEP 3: Read the ZFS Administration Guide ZFS is the basis for the physical storage and data corruption redundancies pointed out at the outset, so you'll need to be familiar with it. @@ -49,19 +49,19 @@ The [ZFS Administration Guide](https://pthree.org/2012/04/17/install-zfs-on-debi It is entirely OK, acceptable, and even preferable to take a long time going through the administration guide. -**STEP 4: Ensure at least 2 extra HDDs or SSDs besides the source one are installed on the computer** +## STEP 4: Ensure at least 2 extra HDDs or SSDs besides the source one are installed on the computer Ensure none of the extra drives contain data you need, as said data will be destroyed during zpool creation. Internal drives are preferable. Given the wide range of possibilities for fulfulling this step, details are not provided. -**STEP 5: Create a zpool from the drives from Step 4** +## STEP 5: Create a zpool from the drives from Step 4 This is done using the `zpool create` command. See these [examples](https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs/). It is *strongly* recommended by this author that the `autoexpand` option is enabled for the pool during or after creation. To enable it after creation in QTerminal for a zpool named *tank* - following the example in the Administration Guide - run `zpool set autoexpand=on tank`. -**STEP 6: Create a ZFS filesystem on the zpool created in Step 5** +## STEP 6: Create a ZFS filesystem on the zpool created in Step 5 This is done using the `zfs create` command. See these [examples](https://pthree.org/2012/12/17/zfs-administration-part-x-creating-filesystems/). @@ -69,7 +69,7 @@ It is *strongly* recommended by this author that the `compression` option is ena Enable `dedup` for the filesystem if you have [sufficient RAM](https://pthree.org/2013/12/18/zfs-administration-appendix-d-the-true-cost-of-deduplication/). To do this in QTerminal after `zfs create` has completed, run `zfs set dedup=on tank/ZFSFileSystemName`. -**STEP 7: Download and install `restic`** +## STEP 7: Download and install `restic` 1. On the Project Trident desktop, open the Lumina start menu 2. Search for *AppCafe* @@ -79,7 +79,7 @@ Enable `dedup` for the filesystem if you have [sufficient RAM](https://pthree.or 6. Click the *Pending* tab 7. Wait for the status of the installation job to go to *Finished* -**STEP 8: Determine the directory structure of your backups** +## STEP 8: Determine the directory structure of your backups Each `restic` `backup` command specifies a source and a repository. The main things to know about repositories are: @@ -93,18 +93,18 @@ Each `restic` `backup` command specifies a source and a repository. The main thi This guide will assume the user is backing up a single source directory to a single repository. -**STEP 9: Initialize the repository** +## STEP 9: Initialize the repository Folllow these [instructions](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local). -**STEP 10: Create a file containing the repository password** +## STEP 10: Create a file containing the repository password 1. In QTerminal, launch Lumina Text Editor as root using `sudo lte` 2. Enter the respository password. Ensure there is absolutely nothing else in the file 3. Click the Save icon 4. Name the file - this guide will call it *PasswordFilename* - and save it in the `/root` directory -**STEP 11: Ensure only root can read the repository password file** +## STEP 11: Ensure only root can read the repository password file It is important that only root can read the file so that the password isn't compromised. @@ -113,30 +113,13 @@ It is important that only root can read the file so that the password isn't comp 3. Change the owner of the file to root by running `sudo chown root PasswordFilename` 4. Test that you can't read the file by trying to open it from Lumina File Manager. Even if the app to read the file launches, it should just display blank content -**STEP 12: Create an environment variable for the password file** - -Automated backups require the corresponding respository password without the user manually entering it. `restic` does this by [reading an environment variable whose value is the password file's path](https://restic.readthedocs.io/en/latest/faq.html?highlight=password#how-can-i-specify-encryption-passwords-automatically). Create the environment variable by: - -1. In QTerminal, run `sudo lte ~/.zshrc` -2. On a new line in the window that pops up, enter: - -``` -# Export restic password file environment variable -export RESTIC_PASSWORD_FILE=/root/PasswordFilename -``` - -in a new line - -3. In QTerminal, run `source ~/.zshrc` to update the shell with the new exported environment variable -4. Test that the above command worked by running `$RESTIC_PASSWORD_FILE`. It should return `/root/PasswordFilename` - -**STEP 13: Construct your backup command** +## STEP 12: Construct your backup command 1. In QTerminal, run `which restic`. This will give you an absolute path to `restic`, which this guide will call `AbsolutePathToRestic`. Typically the absolute path is `/usr/local/bin/restic` -2. Run `AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository --verbose backup AbsolutePathToSource` -3. If 2) above is successful, then the backup command is the same but without the `--verbose`: `AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository backup AbsolutePathToSource` +2. Run `AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository --verbose backup AbsolutePathToSource` +3. If 2) above is successful, then the backup command is the same but without the `--verbose`: `AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository backup AbsolutePathToSource` -**STEP 14: Determine the crontab schedule for your backups** +## STEP 13: Determine the crontab schedule for your backups 1. Think about how often you want to create backups 2. The above bulletpoint is determined entirely by crontab, so *write down* a crontab schedule that matches the above, based on the crontab syntax in 2.1 above @@ -155,11 +138,11 @@ Crontab schedules consist of five numeric or asterisk `*` values separated by ta Or, put into one sentence: Every day at 14:00/2 PM. -**STEP 15: Match desired backups with their corresponding crontab schedules to create single, complete crontab entries for each backup, with the user the `restic` command should run as** +## STEP 14: Match desired backups with their corresponding crontab schedules to create single, complete crontab entries for each backup, with the user the `restic` command should run as** -For example, putting the examples in Steps 13 and 14 together - in that sequence - into a sample crontab entry gives: +For example, putting the examples in Steps 12 and 14 together - in that sequence - into a sample crontab entry gives: -`0 14 * * * root AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository backup AbsolutePathToSource` +`0 14 * * * root AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository backup AbsolutePathToSource` Which translates to: @@ -167,34 +150,34 @@ Which translates to: * Regardless of the day of the month, the month, or the day of the week (what `* * *` stand for, respectively) ... * As the user root (represented by `root`) ... * Invoke `restic` (represented by the absolute path to the `restic` command) to ... -* Read the repository password from the password file (represented by `-p $RESTIC_PASSWORD_FILE`) and ... +* Read the repository password from the password file (represented by `-p /root/PasswordFilename`) and ... * Open the repository located at `AbsolutePathToRepository` (represented by `-r AbsolutePathToRepository`) and ... * Backup the source directory (represented by `backup AbsolutePathToSource`) Or, put into one sentence: As root, open the repository using the password and backup the source to the repository every day at 14:00/2 PM. -**STEP 16: Create a backup retention policy** +## STEP 15: Create a backup retention policy This is described [here](https://restic.readthedocs.io/en/latest/060_forget.html#removing-snapshots-according-to-a-policy). -The command you come up with should look like: `AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository forget --ForgetParameters -prune`. Note the lack of reference to the source directories; this command covers *all* the contents of a repository regardless of source. +The command you come up with should look like: `AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository forget --ForgetParameters -prune`. Note the lack of reference to the source directories; this command covers *all* the contents of a repository regardless of source. -**STEP 17: Determine the crontab schedule for your backup pruning** +## STEP 16: Determine the crontab schedule for your backup pruning** This is similar to Step 14, while ensuring that backups and prunes don't occur simultaneously. -**STEP 18: Match desired prunes with their corresponding crontab schedules to create single, complete crontab entries for each prune, with the user the `restic` command should run as** +## STEP 17: Match desired prunes with their corresponding crontab schedules to create single, complete crontab entries for each prune, with the user the `restic` command should run as** An example of a combined command for the above is: -`@100000 root AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository forget --keep-last 7 -prune` +`@100000 root AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository forget --keep-last 7 -prune` Which translates to: * 100000s after the completion of the previous invocation of the following command by `cron` (represented by `@100000s`) ... * As the user root (represented by `root`) ... * Invoke `restic` (represented by the absolute path to the `restic` command) to ... -* Read the repository password from the password file (represented by `-p $RESTIC_PASSWORD_FILE`) and ... +* Read the repository password from the password file (represented by `-p /root/PasswordFilename`) and ... * Open the repository located at `AbsolutePathToRepository` (represented by `-r AbsolutePathToRepository`) and ... * Forget backups (represented by `forget`) that meet the criteria of ... * Keeping the most recent 7 backups (represented by `--keep-last 7`) and ... @@ -202,9 +185,9 @@ Which translates to: Or, put into a single sentence: As root, open the repository using the password and delete all backups made more than 100000 seconds after the completion of the previous invocation and their associated data that except the 7 most recent backups. -**STEP 19: Put each combined `cron` + `restic` command into `/etc/crontab`** +## STEP 19: Put each combined `cron` + `restic` command into `/etc/crontab` -1. Open crontab in Lumina Text Editor via `sudo lte /etc/crontab`. Unlike on (some) Linux distros, crontab can be edited directly in a GUI application on Project Trident. +1. Open crontab in Lumina Text Editor via `sudo lte /etc/crontab`. Unlike on (some) Linux distros, crontab can be edited directly in a GUI application on Project Trident 2. Put each combined command into the crontab file, with each line preceded by a comment describing what the line does. This is helpful for troubleshooting 3. Save the crontab file and exit Lumina Text Editor @@ -213,12 +196,12 @@ The additional crontab entries should look like this: ``` # Create a backup of source directory daily at 2 PM -0 14 * * * root AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository backup AbsolutePathToSource +0 14 * * * root AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository backup AbsolutePathToSource ``` ``` # Delete all backups and associated data from repository except for the most recent 7 -@100000 root AbsolutePathToRestic -p $RESTIC_PASSWORD_FILE -r AbsolutePathToRepository forget --keep-last 7 -prune +@100000 root AbsolutePathToRestic -p /root/PasswordFilename -r AbsolutePathToRepository forget --keep-last 7 -prune ``` And that's it!