diff --git a/docs/salt.md b/docs/salt.md index e4362775..776b9f63 100644 --- a/docs/salt.md +++ b/docs/salt.md @@ -1,6 +1,244 @@ # Notes on Salt -## Looking up previous runs +## Salt overview + +We use [SaltStack](https://saltstack.com/) (Salt for short) +to configure our infrastructure machines. + +We're currently on the Salt 2016.3 release branch, so make sure to look at the +[right version of the docs](https://docs.saltstack.com/en/2016.3/contents.html). + +Salt configurations are meant to be idempotent and can be applied as many times +as you like; a single deploy is termed a `highstate`, which +runs code to make minion(s) match the requested configuration. + +Salt has a number of different modes of operation, including masterless and SSH; +we are using the default mode, which is a single master and many minions. +Note that the `servo-master1` machine runs both a `salt-master` process +and a `salt-minion` process, so we can Salt that machine itself. +Deploys should be run from the master machine, `servo-master1`. + +Also, note that we may occasionally run multiple masters in a Salt multi-master +setup, e.g. this occurred when we migrated from a master on Linode to a master +on EC2. See [this issue](https://github.com/servo/saltfs/issues/281) for more. + +## Salt workflow + +This (the servo/saltfs) repository holds all of our Salt code, which describes +how we want to configure our machines. +We are using a feature of Salt called gitfs, which means that the Salt master +will automatically use the latest version of this repo on the master branch +at all times, which makes it hard to get out of sync and makes deploys easier. + +For this reason, the recommended workflow for making changes is to make PRs +to this repository; however, note that there is a +[manual override]([Discouraged: Testing in production]) +that can be used if necessary. + +### Getting started + +The best way to make changes is to try them out locally, +shepherd them through the review and pull request process, +then apply them in production. + +Clone the [servo/saltfs](https://github.com/servo/saltfs/) repo locally, +and create a new feature branch: + +```console +$ git clone git@github.com:servo/saltfs.git servo-saltfs +$ cd servo-saltfs +$ git checkout -b new-feature +``` + +Make changes as needed on your branch. + +### Testing with Vagrant + +You can test your changes using [Vagrant](https://vagrantup.com); +a Vagrantfile is included in the repo. +Vagrant's Salt provisioner tends to make backwards-incompatible changes often, +so only a few versions of Vagrant will work at any time; +the Vagrantfile has information about which ones. +Currently, version 1.8.1 is recommended; +note that many OSs like Ubuntu may still have older versions like 1.7.x, +so you may need to +[download a different version](https://releases.hashicorp.com/vagrant/). +Note that we are using the [VirtualBox](https://www.virtualbox.org/) backend +for the actual VMs; the Vagrant installer should automatically install it for +you on Windows and macOS. + +Each node type has a corresponding Vagrant target, +and Vagrant will run the Salt states when provisioning. +For example, you might run these commands from within +your checkout of the saltfs repo (same directory as the Vagrantfile): + +* `$ vagrant up`: Start + provision all VMs configured in Vagrantfile, + alternately `$ vagrant up servo-master1` to start just one +* `$ vagrant ssh servo-master1`: Connect to a running VM + +Note that your prompt will change once you're inside a VM. + +* `vagrant@vagrant-ubuntu-trusty-64$ sudo apt-get -y install cowsay; cowsay 'servo-saltfs ❤ Vagrant'`: + Run commands in the VM +* `vagrant@vagrant-ubuntu-trusty-64$ exit`: Leave the VM + +Back outside the VM: + +* `$ vagrant provision servo-master1`: (Re-)apply the Salt states + on the servo-master1 VM +* `$ vagrant halt`: Halt all running VMs + +Note that Salt states are (should be) idempotent, +so you can run the provisioning as many times as you want. +Remember to re-run the `vagrant provision` command to apply your changes +to your VMs after you edit their config files. +If you're making changes, please run `vagrant provision` at least twice +to ensure this invariant of idempotency. + +:warning: The Vagrantfile gives each VM 1 GB of memory, +which is enough to apply the Salt states but not enough to build Servo. +If you need to build Servo, edit the Vagrantfile locally to allot 4 GB +(remember not to commit this change!): + +```ruby +vbox.memory = 4096 +``` + +:dash: [vagrant-cachier](http://fgrehm.viewdocs.io/vagrant-cachier/) is +recommended to make running `vagrant provision` faster +by caching downloaded packages. +The Vagrantfile will automatically take advantage of it: +run ```vagrant plugin install vagrant-cachier``` to install it. + +There's also a test script that currently has a few lints; +it requires Python 3.5 (currently must be installed manually) +and can be run as follows (must be inside the VM): + +```console +vagrant@vagrant-ubuntu-trusty-64$ ./test.py +``` + +### PR + Review process + +Once you're happy, `git commit` your changes, +`git push` them up to your fork of the saltfs repo on Github, +and [open a pull request](https://github.com/servo/saltfs/compare). + +A Servo reviewer should review your changes, +roughly along [these lines](https://github.com/servo/servo/wiki/Code-review). + +:memo: Take a look at the STYLE_GUIDE.md in the repository root +for some helpful hints. + +### Deploying changes + +Once your PR has been approved and merged by Travis, +it will be available on the master branch and Salt will automatically +use that code for deploys. + +Examples of highstates, which should be run as root from the Salt master: + +* `$ salt '*' state.highstate`: Apply all configs to all hosts +* `$ salt 'servo-linux*' state.highstate test=True`: See what would happen + if we applied all configs to all servo-linux hosts + +The '*' is a glob which means to apply the Salt states on all of the minions; +you can specify just one minion ID instead to only run the Salt states there. +[More complicated targeting is also available.] +(https://docs.saltstack.com/en/2016.3/ref/cli/index.html#using-the-salt-command) + +:warning: Make sure to run with `test=True` mode first to see what effect your +changes will make before actually deploying! Read through the results carefully +and investigate anything that looks strange. + +Steps to take: + +```console +$ ssh root@servo-master1.servo.org +root@servo-master1$ sudo salt '*' state.highstate test=True 2>&1 --force-color | less -R +root@servo-master1$ sudo salt '*' state.highstate 2>&1 --force-color | tee salt_highstate_pr_number.log | less -R +``` + +Please make sure to log all Salt actions which perform changes to files, +e.g. by using `tee` as in the last command line! +Occasionally things go wrong, and having logs really helps debug, +particularly when we have to hand off between administrators. + +Some deploys will require additionally manual steps, such as restarting +Buildbot or another service; these should be recorded in a checklist on the PR, +and checked off when completed to keep track of things and if a PR is deployed. + +See [the wiki](https://github.com/servo/servo/wiki/Buildbot-administration) +for more information on how to restart Buildbot, Homu, etc. cleanly. + +### Reverting changes + +Since the Salt master will pull from the latest git master, +the easiest way to revert a change is to `git revert` the change +and shepard the revert through a PR + review into master. + +### Discouraged: Testing in production + +To avoid configuration drift, it's highly recommended to always use gitfs. +However, sometimes it's necessary to test or run deploys directly on the master, +without going through the saltfs master. + +You can clone the saltfs repo to `/tmp/salt-testing-root/saltfs` on the master, +which acts as an override directory. +See the `salt/files/master/ADMIN_README` in this repo for more information. + +:warning: Make sure to clean out the directly when you are done using it! + + +You can also run Salt locally on a minion. +It's easier to do this with Vagrant (use `vagrant provision` from outside +the VM) on a local VM, +but it's also possible to do on our actual infrastructure if needed. + +:memo: Make sure you're logged in as root or use sudo when doing this! +`salt-call` needs to run as root. + +As always, use `test=True` mode first to verify any changes. +This is especially helpful for debugging, +as you can see what changes Salt will apply without actually performing them. +Remove the `test=True` to actually run the highstate. + +```console +$ salt-call state.highstate test=True 2>&1 --force-color | less -R +$ salt-call state.highstate 2>&1 --force-color | tee salt_call_highstate_pr_number.log | less -R +``` + +This has the same effect as running Salt from the master, +but Salt will additionally print out debug information as it runs each state, +which can be helpful when debugging. + +The above will reach out and pull down the state tree and relevant pillar info +from the Salt master. +You can also use a local state and pillar tree, i.e. for testing changes. +First, create local copies of the state tree and pillar tree. +The location doesn't matter, +but let's say you put them in `/tmp/salt/states` and `/tmp/salt/pillars`. +You can make any changes you'd like to test. +Then, to run Salt completely locally, run: + +```console +$ salt-call --local --file-root=/tmp/salt/states --pillar-root=/tmp/salt/pillars state.highstate test=True 2>&1 --force-color | less -R +``` +Change the folder paths as necessary. + +:warning: This is great for one-off testing because you have local versions of +the Salt states and pillars that you can tweak without affecting other machines, +but it can be easy to get out of sync with other machines. +If you make any kind of permanent change, +make sure it finds it way into the saltfs repo to make it repeatable! + + +## Salt administration + +See [the wiki](https://github.com/servo/servo/wiki/SaltStack-Administration) +for more information about setting up new minions and/or masters. + +### Looking up previous runs Salt keeps track of the results of each `job` it performs, where a single `job` may be running a single `state.highstate` on one minion, @@ -38,3 +276,16 @@ root@servo-master1$ salt-run jobs.lookup_jid --out=highstate ``` See [the Salt docs](https://docs.saltstack.com/en/2016.3/ref/runners/all/salt.runners.jobs.html#module-salt.runners.jobs) for futher documentation. + +### Upgrading Salt + +We're using a manual update process for now since there are so few machines: +first on the salt master, then on the minions. Things to be aware of: + +* It's necessary to restart the salt-master and salt-minion services to apply + config changes, but restarting a master or minion service will interrupt an + ongoing highstate. + Make sure to run the highstate more than once to fully converge on changes. +* Masters need to be updated before minions, but `salt '*' state.highstate` + cannot enforce ordering - make sure to update just the master first with + `salt 'servo-master1' state.highstate`. diff --git a/salt/files/master/ADMIN_README b/salt/files/master/ADMIN_README index db21c8a7..6bf00939 100644 --- a/salt/files/master/ADMIN_README +++ b/salt/files/master/ADMIN_README @@ -5,8 +5,7 @@ This folder takes precedence over gitfs (which reads file from the master branch of the saltfs repository), so it is important to follow a few guidelines: - Avoid using this folder if possible: prefer to test changes locally using the -steps under "Making changes" at -https://github.com/servo/servo/wiki/Buildbot-administration +steps under "Salt workflow" in the `docs/salt.md` file. - If you do need to use this folder, use it only as long as you need it, and make sure to remove the cloned saltfs folder afterwards. Any files that are