diff --git a/.github/workflows/jekyll.yaml b/.github/workflows/jekyll.yaml index 2e38e741a..e900a9550 100644 --- a/.github/workflows/jekyll.yaml +++ b/.github/workflows/jekyll.yaml @@ -40,3 +40,5 @@ jobs: run: | ruby --version bundle exec jekyll build + - name: Check links + run: bundle exec rake links diff --git a/Gemfile b/Gemfile index 4aa0e7f4d..2ff33e2e1 100644 --- a/Gemfile +++ b/Gemfile @@ -2,3 +2,4 @@ ruby '>=3.2' source 'https://rubygems.org' gem 'jekyll', '>= 4.2.0' gem 'jekyll-redirect-from', '>= 0.16.0' +gem 'html-proofer', '>= 5.2' diff --git a/Gemfile.lock b/Gemfile.lock index 919793d08..92293b0c2 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,25 +1,57 @@ GEM remote: https://rubygems.org/ specs: + Ascii85 (2.0.1) addressable (2.8.8) public_suffix (>= 2.0.2, < 8.0) + afm (1.0.0) + async (2.37.0) + console (~> 1.29) + fiber-annotation + io-event (~> 1.11) + metrics (~> 0.12) + traces (~> 0.18) base64 (0.3.0) + benchmark (0.5.0) bigdecimal (4.0.1) colorator (1.1.0) concurrent-ruby (1.3.6) + console (1.34.3) + fiber-annotation + fiber-local (~> 1.1) + json csv (3.3.5) em-websocket (0.5.3) eventmachine (>= 0.12.9) http_parser.rb (~> 0) + ethon (0.18.0) + ffi (>= 1.15.0) + logger eventmachine (1.2.7) ffi (1.17.3-x86_64-linux-gnu) + fiber-annotation (0.2.0) + fiber-local (1.1.0) + fiber-storage + fiber-storage (1.0.1) forwardable-extended (2.6.0) google-protobuf (4.33.5-x86_64-linux-gnu) bigdecimal rake (>= 13) + hashery (2.1.2) + html-proofer (5.2.2) + addressable (~> 2.3) + async (~> 2.1) + benchmark (~> 0.5) + nokogiri (~> 1.13) + pdf-reader (~> 2.11) + rainbow (~> 3.0) + typhoeus (~> 1.3) + yell (~> 2.0) + zeitwerk (~> 2.5) http_parser.rb (0.8.1) i18n (1.14.8) concurrent-ruby (~> 1.0) + io-event (1.11.2) jekyll (4.4.1) addressable (~> 2.4) base64 (~> 0.2) @@ -57,9 +89,19 @@ GEM rb-inotify (~> 0.9, >= 0.9.10) logger (1.7.0) mercenary (0.4.0) + metrics (0.15.0) + nokogiri (1.19.4-x86_64-linux-gnu) + racc (~> 1.4) pathutil (0.16.2) forwardable-extended (~> 2.6) + pdf-reader (2.16.0) + Ascii85 (>= 1.0, < 3.0, != 2.0.0) + afm (>= 0.2.1, < 2) + hashery (~> 2.0) + ttfunk public_suffix (7.0.2) + racc (1.8.1) + rainbow (3.1.1) rake (13.3.1) rb-fsevent (0.11.2) rb-inotify (0.11.1) @@ -71,13 +113,20 @@ GEM google-protobuf (~> 4.31) terminal-table (3.0.2) unicode-display_width (>= 1.1.1, < 3) + traces (0.18.2) + ttfunk (1.7.0) + typhoeus (1.6.0) + ethon (>= 0.18.0) unicode-display_width (2.6.0) webrick (1.9.2) + yell (2.2.2) + zeitwerk (2.8.3) PLATFORMS x86_64-linux DEPENDENCIES + html-proofer (>= 5.2) jekyll (>= 4.2.0) jekyll-redirect-from (>= 0.16.0) @@ -85,4 +134,4 @@ RUBY VERSION ruby 3.2.2 BUNDLED WITH - 4.0.6 + 4.0.21 diff --git a/README.md b/README.md index 0ed389a2c..20f663c53 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,12 @@ HTML styled "just right". Jekyll will print a local URL where the site can be viewed (usually, [http://0.0.0.0:4000/](http://0.0.0.0:4000/)). +To check internal links (also run in CI; scope is set in `Rakefile`): + +```bash +bundle exec jekyll build && bundle exec rake links +``` + ### Testing using a Container environment Note: The example commands below use `podman`, but you can replace it with a diff --git a/Rakefile b/Rakefile new file mode 100644 index 000000000..b4eaed3dc --- /dev/null +++ b/Rakefile @@ -0,0 +1,32 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +require 'html-proofer' + +# Internal link check; run `jekyll build` first +task :links do + HTMLProofer.check_directory('./_site', { + disable_external: true, + check_internal_hash: true, + enforce_https: false, + allow_missing_href: true, + # only docs/4.x is checked for now + ignore_files: [%r{^\./_site/(?!docs/4\.x/)}, %r{/apidocs/}], + }).run +end diff --git a/_docs-4/administration/caching.md b/_docs-4/administration/caching.md index cea0fd68c..03596f573 100644 --- a/_docs-4/administration/caching.md +++ b/_docs-4/administration/caching.md @@ -18,7 +18,7 @@ for tables where read performance is critical. ## Configuration -The {% plink tserver.cache.manager.class %} property controls which block cache implementation is used within the tablet server. Users +The {% plink general.block.cache.manager.class %} property controls which block cache implementation is used within the tablet server. Users can supply their own implementation and set custom configuration properties to control its behavior (see org.apache.accumulo.core.spi.cache.BlockCacheManager$Configuration.java). The index and data block caches are configured for tables by the following properties: diff --git a/_docs-4/administration/replication.md b/_docs-4/administration/replication.md deleted file mode 100644 index 91cb4b353..000000000 --- a/_docs-4/administration/replication.md +++ /dev/null @@ -1,419 +0,0 @@ ---- -title: Replication -category: administration -order: 5 ---- - -## Overview - -Replication is a feature of Accumulo which provides a mechanism to automatically -copy data to other systems, typically for the purpose of disaster recovery, -high availability, or geographic locality. It is best to consider this feature -as a framework for automatic replication instead of the ability to copy data -from to another Accumulo instance as copying to another Accumulo cluster is -only an implementation detail. The local Accumulo cluster is hereby referred -to as the `primary` while systems being replicated to are known as -`peers`. - -This replication framework makes two Accumulo instances, where one instance -replicates to another, eventually consistent between one another, as opposed -to the strong consistency that each single Accumulo instance still holds. That -is to say, attempts to read data from a table on a peer which has pending replication -from the primary will not wait for that data to be replicated before running the scan. -This is desirable for a number of reasons, the most important is that the replication -framework is not limited by network outages or offline peers, but only by the HDFS -space available on the primary system. - -Replication configurations can be considered as a directed graph which allows cycles. -The systems in which data was replicated from is maintained in each Mutation which -allow each system to determine if a peer already has the data in which -the system wants to send. - -Data is replicated by using the Write-Ahead logs (WAL) that each TabletServer is -already maintaining. TabletServers record which WALs have data that need to be -replicated to the `accumulo.metadata` table. The Manager uses these records, -combined with the local Accumulo table that the WAL was used with, to create records -in the `replication` table which track which peers the given WAL should be -replicated to. The Manager latter uses these work entries to assign the actual -replication task to a local TabletServer using ZooKeeper. A TabletServer will get -a lock in ZooKeeper for the replication of this file to a peer, and proceed to -replicate to the peer, recording progress in the `replication` table as -data is successfully replicated on the peer. Later, the Manager and Garbage Collector -will remove records from the `accumulo.metadata` and `replication` tables -and files from HDFS, respectively, after replication to all peers is complete. - -## Configuration - -Configuration of Accumulo to replicate data to another system can be categorized -into the following sections. - -### Site Configuration - -Each system involved in replication (even the primary) needs a name that uniquely -identifies it across all peers in the replication graph. This should be considered -fixed for an instance, and set using {% plink replication.name %} in `accumulo.properties`. - -``` -# Unique name for this system used by replication -replication.name=primary -``` - -### Instance Configuration - -For each peer of this system, Accumulo needs to know the name of that peer, -the class used to replicate data to that system and some configuration information -to connect to this remote peer. In the case of Accumulo, this additional data -is the Accumulo instance name and ZooKeeper quorum; however, this varies on the -replication implementation for the peer. - -These can be set in `accumulo.properties` to ease deployments; however, as they may -change, it can be useful to set this information using the Accumulo shell. - -To configure a peer with the name `peer1` which is an Accumulo system with an instance name of `accumulo_peer` -and a ZooKeeper quorum of `10.0.0.1,10.0.2.1,10.0.3.1`, invoke the following -command in the shell. - -```console -root@accumulo_primary> config -s -replication.peer.peer1=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,accumulo_peer,10.0.0.1,10.0.2.1,10.0.3.1 -``` - -Since this is an Accumulo system, we also want to set a username and password -to use when authenticating with this peer. On our peer, we make a special user -which has permission to write to the tables we want to replicate data into, "replication" -with a password of "password". We then need to record this in the primary's configuration. - -```console -root@accumulo_primary> config -s replication.peer.user.peer1=replication -root@accumulo_primary> config -s replication.peer.password.peer1=password -``` - -Alternatively, when configuring replication on Accumulo running Kerberos, a keytab -file per peer can be configured instead of a password. The provided keytabs must be readable -by the unix user running Accumulo. They keytab for a peer can be unique from the -keytab used by Accumulo or any keytabs for other peers. - -```console -accumulo@EXAMPLE.COM@accumulo_primary> config -s replication.peer.user.peer1=replication@EXAMPLE.COM -accumulo@EXAMPLE.COM@accumulo_primary> config -s replication.peer.keytab.peer1=/path/to/replication.keytab -``` - -### Table Configuration - -Now, we presently have a peer defined, so we just need to configure which tables will -replicate to that peer. We also need to configure an identifier to determine where -this data will be replicated on the peer. Since we're replicating to another Accumulo -cluster, this is a table ID. In this example, we want to enable replication on -`my_table` and configure our peer `accumulo_peer` as a target, sending -the data to the table with an ID of `2` in `accumulo_peer`. - -```console -root@accumulo_primary> config -t my_table -s table.replication=true -root@accumulo_primary> config -t my_table -s table.replication.target.accumulo_peer=2 -``` - -To replicate a single table on the primary to multiple peers, the second command -in the above shell snippet can be issued, for each peer and remote identifier pair. - -## Monitoring - -Basic information about replication status from a primary can be found on the Accumulo -Monitor server, using the `Replication` link the sidebar. - -On this page, information is broken down into the following sections: - -1. Files pending replication by peer and target -2. Files queued for replication, with progress made - -## Work Assignment - -Depending on the schema of a table, different implementations of the [WorkAssigner] -used could be configured. The implementation is controlled via the property {% plink replication.work.assigner %} -and the full class name for the implementation. This can be configured via the shell or `accumulo.properties`. - -Two implementations of [WorkAssigner] are provided: - -1. The {% jlink org.apache.accumulo.manager.replication.UnorderedWorkAssigner %} can be used to overcome the limitation -of only a single WAL being replicated to a target and peer at any time. Depending on the table schema, -it's possible that multiple versions of the same Key with different values are infrequent or nonexistent. -In this case, parallel replication to a peer and target is possible without any downsides. In the case -where this implementation is used were column updates are frequent, it is possible that there will be -an inconsistency between the primary and the peer. - -2. The {% jlink org.apache.accumulo.manager.replication.SequentialWorkAssigner %} is configured for an -instance. The SequentialWorkAssigner ensures that, per peer and each remote identifier, each WAL is -replicated in the order in which they were created. This is sufficient to ensure that updates to a table -will be replayed in the correct order on the peer. This implementation has the downside of only replicating -a single WAL at a time. - - -## ReplicaSystems - -[ReplicaSystem] is the interface which allows abstraction of replication of data -to peers of various types. Presently, only an [AccumuloReplicaSystem] is provided -which will replicate data to another Accumulo instance. A [ReplicaSystem] implementation -is run inside of the TabletServer process, and can be configured as mentioned in [Instance Configuration][InstanceConfig] -section of this document. Theoretically, an implementation of this interface could send data to other filesystems, databases, etc. - -### AccumuloReplicaSystem - -The [AccumuloReplicaSystem] uses Thrift to communicate with a peer Accumulo instance -and replicate the necessary data. The TabletServer running on the primary will communicate -with the Manager on the peer to request the address of a TabletServer on the peer which -this TabletServer will use to replicate the data. - -The TabletServer on the primary will then replicate data in batches of a configurable -size ({% plink replication.max.unit.size %}). The TabletServer on the peer will report how many -records were applied back to the primary, which will be used to record how many records -were successfully replicated. The TabletServer on the primary will continue to replicate -data in these batches until no more data can be read from the file. - -## Other Configuration - -There are a number of configuration values that can be used to control how -the implementation of various components operate. - -* {% plink replication.max.work.queue %} - Maximum number of files queued for replication at one time -* {% plink replication.work.assignment.sleep %} - Time between invocations of the [WorkAssigner] -* {% plink replication.worker.threads %} - Size of threadpool used to replicate data to peers -* {% plink replication.receipt.service.port %} - Thrift service port to listen for replication requests, can use '0' for a random port -* {% plink replication.work.attempts %} - Number of attempts to replicate to a peer before aborting the attempt -* {% plink replication.receiver.min.threads %} - Minimum number of idle threads for handling incoming replication -* {% plink replication.receiver.threadcheck.time %} - Time between attempting adjustments of thread pool for incoming replications -* {% plink replication.max.unit.size %} - Maximum amount of data to be replicated in one RPC -* {% plink replication.work.assigner %} - [WorkAssigner] implementation -* {% plink tserver.replication.batchwriter.replayer.memory %} - Size of BatchWriter cache to use in applying replication requests - -## Example Practical Configuration - -A real-life example is now provided to give concrete application of replication configuration. This -example is a two instance Accumulo system, one primary system and one peer system. They are called -**primary** and **peer**, respectively. Each system also have a table of the same name, `my_table`. The instance -name for each is also the same (`primary` and `peer`), and both have ZooKeeper hosts on a node with a hostname -with that name as well (primary:2181 and peer:2181). - -We want to configure these systems so that `my_table` on **primary** replicates to `my_table` on **peer**. - -### accumulo.properties - -We can assign the "unique" name that identifies this Accumulo instance among all others that might participate -in replication together. In this example, we will use the names provided in the description. - -#### Primary - -``` -replication.name=primary -``` - -#### Peer - -``` -replication.name=peer -``` - -### managers and tservers files - -Be *sure* to use non-local IP addresses. Other nodes need to connect to it and using localhost will likely result in -a local node talking to another local node. - -### Start both instances - -The rest of the configuration is dynamic and is best configured on the fly (in ZooKeeper) than in accumulo.properties. - -### Peer - -The next series of command are to be run on the peer system. Create a user account for the primary instance called -"peer". The password for this account will need to be saved in the configuration on the primary - -```console -root@peer> createtable my_table -root@peer> createuser peer -root@peer> grant -t my_table -u peer Table.WRITE -root@peer> grant -t my_table -u peer Table.READ -root@peer> tables -l -``` - -Remember what the table ID for 'my_table' is. You'll need that to configure the primary instance. - -### Primary - -Next, configure the primary instance. - -#### Set up the table - -```console -root@primary> createtable my_table -``` - -#### Define the Peer as a replication peer to the Primary - -We're defining the instance with {% plink replication.name %} of `peer` as a peer. We provide the implementation of [ReplicaSystem] -that we want to use, and the configuration for the [AccumuloReplicaSystem]. In this case, the configuration is the Accumulo -Instance name for `peer` and the ZooKeeper quorum string. The configuration key is of the form -`replication.peer.$peer_name`. - -```console -root@primary> config -s replication.peer.peer=org.apache.accumulo.tserver.replication.AccumuloReplicaSystem,peer,$peer_zk_quorum -``` - -#### Set the authentication credentials - -We want to use that special username and password that we created on the peer, so we have a means to write data to -the table that we want to replicate to. The configuration key is of the form "replication.peer.user.$peer_name". - -```console -root@primary> config -s replication.peer.user.peer=peer -root@primary> config -s replication.peer.password.peer=peer -``` - -#### Enable replication on the table - -Now that we have defined the peer on the primary and provided the authentication credentials, we need to configure -our table with the implementation of [ReplicaSystem] we want to use to replicate to the peer. In this case, our peer -is an Accumulo instance, so we want to use the [AccumuloReplicaSystem]. - -The configuration for the AccumuloReplicaSystem is the table ID for the table on the peer instance that we -want to replicate into. Be sure to use the correct value for $peer_table_id. The configuration key is of -the form "table.replication.target.$peer_name". - -```console -root@primary> config -t my_table -s table.replication.target.peer=$peer_table_id -``` - -Finally, we can enable replication on this table. - -```console -root@primary> config -t my_table -s table.replication=true -``` - -## Extra considerations for use - -While this feature is intended for general-purpose use, its implementation does carry some baggage. Like any software, -replication is a feature that operates well within some set of use cases but is not meant to support all use cases. -For the benefit of the users, we can enumerate these cases. - -### Latency - -As previously mentioned, the replication feature uses the Write-Ahead Log files for a number of reasons, one of which -is to prevent the need for data to be written to RFiles before it is available to be replicated. While this can help -reduce the latency for a batch of Mutations that have been written to Accumulo, the latency is at least seconds to tens -of seconds for replication once ingest is active. For a table which replication has just been enabled on, this is likely -to take a few minutes before replication will begin. - -Once ingest is active and flowing into the system at a regular rate, replication should be occurring at a similar rate, -given sufficient computing resources. Replication attempts to copy data at a rate that is to be considered low latency -but is not a replacement for custom indexing code which can ensure near real-time referential integrity on secondary indexes. - -### Table-Configured Iterators - -Accumulo Iterators tend to be a heavy hammer which can be used to solve a variety of problems. In general, it is highly -recommended that Iterators which are applied at major compaction time are both idempotent and associative due to the -non-determinism in which some set of files for a Tablet might be compacted. In practice, this translates to common patterns, -such as aggregation, which are implemented in a manner resilient to duplication (such as using a Set instead of a List). - -Due to the asynchronous nature of replication and the expectation that hardware failures and network partitions will exist, -it is generally not recommended to not configure replication on a table which has Iterators set which are not idempotent. -While the replication implementation can make some simple assertions to try to avoid re-replication of data, it is not -presently guaranteed that all data will only be sent to a peer once. Data will be replicated at least once. Typically, -this is not a problem as the VersioningIterator will automatically deduplicate this over-replication because they will -have the same timestamp; however, certain Combiners may result in inaccurate aggregations. - -As a concrete example, consider a table which has the SummingCombiner configured to sum all values for -multiple versions of the same Key. For some key, consider a set of numeric values that are written to a table on the -primary: [1, 2, 3]. On the primary, all of these are successfully written and thus the current value for the given key -would be 6, (1 + 2 + 3). Consider, however, that each of these updates to the peer were done independently (because -other data was also included in the write-ahead log that needed to be replicated). The update with a value of 1 was -successfully replicated, and then we attempted to replicate the update with a value of 2 but the remote server never -responded. The primary does not know whether the update with a value of 2 was actually applied or not, so the -only recourse is to re-send the update. After we receive confirmation that the update with a value of 2 was replicated, -we will then replicate the update with 3. If the peer did never apply the first update of '2', the summation is accurate. -If the update was applied but the acknowledgement was lost for some reason (system failure, network partition), the -update will be resent to the peer. Because addition is non-idempotent, we have created an inconsistency between the -primary and peer. As such, the SummingCombiner wouldn't be recommended on a table being replicated. - -While there are changes that could be made to the replication implementation which could attempt to mitigate this risk, -presently, it is not recommended to configure Iterators or Combiners which are not idempotent to support cases where -inaccuracy of aggregations is not acceptable. - -### Duplicate Keys - -In Accumulo, when more than one key exists that are exactly the same, keys that are equal down to the timestamp, -the retained value is non-deterministic. Replication introduces another level of non-determinism in this case. -For a table that is being replicated and has multiple equal keys with different values inserted into it, the final -value in that table on the primary instance is not guaranteed to be the final value on all replicas. - -For example, say the values that were inserted on the primary instance were `value1` and `value2` and the final -value was `value1`, it is not guaranteed that all replicas will have `value1` like the primary. The final value is -non-deterministic for each instance. - -As is the recommendation without replication enabled, if multiple values for the same key (sans timestamp) are written to -Accumulo, it is strongly recommended that the value in the timestamp properly reflects the intended version by -the client. That is to say, newer values inserted into the table should have larger timestamps. If the time between -writing updates to the same key is significant (order minutes), this concern can likely be ignored. - -### Bulk Imports - -Currently, files that are bulk imported into a table configured for replication are not replicated. There is no -technical reason why it was not implemented, it was simply omitted from the initial implementation. This is considered a -fair limitation because bulk importing generated files multiple locations is much simpler than bifurcating "live" ingest -data into two instances. Given some existing bulk import process which creates files and them imports them into an -Accumulo instance, it is trivial to copy those files to a new HDFS instance and import them into another Accumulo -instance using the same process. Hadoop's `distcp` command provides an easy way to copy large amounts of data to another -HDFS instance which makes the problem of duplicating bulk imports very easy to solve. - -## Table Schema - -The following describes the kinds of keys, their format, and their general function for the purposes of individuals -understanding what the replication table describes. Because the replication table is essentially a state machine, -this data is often the source of truth for why Accumulo is doing what it is with respect to replication. There are -three "sections" in this table: "repl", "work", and "order". - -### Repl section - -This section is for the tracking of a WAL file that needs to be replicated to one or more Accumulo remote tables. -This entry is tracking that replication needs to happen on the given WAL file, but also that the local Accumulo table, -as specified by the column qualifier "local table ID", has information in this WAL file. - -The structure of the key-value is as follows: - -``` - repl: [] -> -``` - -This entry is created based on a replication entry from the Accumulo metadata table, and is deleted from the replication table -when the WAL has been fully replicated to all remote Accumulo tables. - -### Work section - -This section is for the tracking of a WAL file that needs to be replicated to a single Accumulo table in a remote -Accumulo cluster. If a WAL must be replicated to multiple tables, there will be multiple entries. The Value for this -Key is a serialized ProtocolBuffer message which encapsulates the portion of the WAL which was already sent for -this file. The "replication target" is the unique location of where the file needs to be replicated: the identifier -for the remote Accumulo cluster and the table ID in that remote Accumulo cluster. The protocol buffer in the value -tracks the progress of replication to the remote cluster. - -``` - work: [] -> -``` - -The "work" entry is created when a WAL has an "order" entry, and deleted after the WAL is replicated to all -necessary remote clusters. - -### Order section - -This section is used to order and schedule (create) replication work. In some cases, data with the same timestamp -may be provided multiple times. In this case, it is important that WALs are replicated in the same order they were -created/used. In this case (and in cases where this is not important), the order entry ensures that oldest WALs -are processed most quickly and pushed through the replication framework. - -``` -\x00 order: [] -> -``` - -The "order" entry is created when the WAL is closed (no longer being written to) and is removed when -the WAL is fully replicated to all remote locations. - -[WorkAssigner]: {% jurl org.apache.accumulo.server.replication.WorkAssigner %} -[ReplicaSystem]: {% jurl org.apache.accumulo.server.replication.ReplicaSystem %} -[AccumuloReplicaSystem]: {% jurl org.apache.accumulo.server.replication.AccumuloReplicaSystem %} -[InstanceConfig]: {% durl administration/replication#instance-configuration %} diff --git a/_docs-4/administration/upgrading.md b/_docs-4/administration/upgrading.md index fcf971cc1..d979c0e7f 100644 --- a/_docs-4/administration/upgrading.md +++ b/_docs-4/administration/upgrading.md @@ -191,10 +191,11 @@ Below are some changes in 2.0 that you should be aware of: ``` accumulo convert-config -x old/accumulo-site.xml -p new/accumulo.properties ``` -* The following [server properties]({% durl configuration/server-properties %}) were deprecated for 2.0: - * {% plink general.classpaths %} - * {% plink tserver.metadata.readahead.concurrent.max %} - * {% plink tserver.readahead.concurrent.max %} +* The following [server properties]({% durl configuration/server-properties %}) were deprecated in 2.0 + and no longer exist in 4.x: + * `general.classpaths` + * `tserver.metadata.readahead.concurrent.max` + * `tserver.readahead.concurrent.max` * `accumulo-client.properties` has replaced `client.conf`. The [client properties]({% durl configuration/client-properties %}) in the new file are different so take care when customizing. * `accumulo-cluster` script has replaced the `start-all.sh` & `stop-all.sh` scripts. diff --git a/_docs-4/configuration/client-properties.md b/_docs-4/configuration/client-properties.md index d7a8f05f7..a2837195b 100644 --- a/_docs-4/configuration/client-properties.md +++ b/_docs-4/configuration/client-properties.md @@ -7,7 +7,7 @@ order: 3 -Below are properties set in `accumulo-client.properties` that configure [Accumulo clients]({{ page.docs_baseurl }}/getting-started/clients#connecting). All properties have been part of the API since 2.0.0 (unless otherwise specified): +Below are properties set in `accumulo-client.properties` that configure [Accumulo clients]({{ page.docs_baseurl }}/getting-started/clients#creating-an-accumulo-client). All properties have been part of the API since 2.0.0 (unless otherwise specified): | Property | Default value | Since | Description | |----------|---------------|-------|-------------| diff --git a/_docs-4/configuration/files.md b/_docs-4/configuration/files.md index a6402ba55..72782304d 100644 --- a/_docs-4/configuration/files.md +++ b/_docs-4/configuration/files.md @@ -78,12 +78,7 @@ multiple tablet servers are run for improved fault tolerance and performance. Contains a list of hosts where [ScanServer] processes should run. While only one host is needed, it is recommended that multiple ScanServers are run for improved performance. -### compaction coordinator - -Contains a list of hosts where [CompactionCoordinator] processes should run. While only one host is needed, -others can be specified to run standby CompactionCoordinators that can take over if the lead CompactionCoordinator fails. - -### compaction compactor +### compactor Contains a list of hosts where [Compactor] processes should run. While only one host is needed, it is recommended that multiple Compactors are run for improved external compaction performance. @@ -92,7 +87,6 @@ multiple Compactors are run for improved external compaction performance. [Manager]: {% durl getting-started/design#manager %} [Tablet Server]: {% durl getting-started/design#tablet-server %} [Monitor]: {% durl getting-started/design#monitor %} -[CompactionCoordinator]: {% durl getting-started/design#compaction-coordinator-experimental %} -[Compactor]: {% durl getting-started/design#compactor-experimental %} -[ScanServer]: {% durl getting-started/design#scan-server-experimental %} +[Compactor]: {% durl getting-started/design#compactor %} +[ScanServer]: {% durl getting-started/design#scan-server %} [quick start]: {% durl getting-started/quickstart#configuring-accumulo %} diff --git a/_docs-4/configuration/server-properties.md b/_docs-4/configuration/server-properties.md index ede3f3208..c65a387e8 100644 --- a/_docs-4/configuration/server-properties.md +++ b/_docs-4/configuration/server-properties.md @@ -177,7 +177,7 @@ Below are properties set in `accumulo.properties` or the Accumulo shell that con | sserver.group | **Available since:** 3.0.0
Resource group name for this ScanServer. Resource groups support at least two use cases: dedicating resources to scans and/or using different hardware for scans. Clients can configure the ConfigurableScanServerSelector to specify the resource group to use for eventual consistency scans.

**type:** STRING, **zk mutable:** yes but requires restart of the sserver, **default value:** `default` | | sserver.port.client | **Available since:** 2.1.0
The port used for handling client connections on the tablet servers.
**type:** PORT, **zk mutable:** yes but requires restart of the sserver, **default value:** `9700-9799` | | sserver.scan.allowed.tables | **Available since:** 4.0.0
A regular expression that determines which tables are allowed to be scanned for servers in the specified group. The property name should end with the scan server group and the property value should take into account the table namespace and name. The default value disallows scans on tables in the accumulo namespace.
**type:** STRING, **zk mutable:** yes, **default value:** `^(?!accumulo\.).*$` | -| ~~ **sserver.scan.allowed.tables.group.***~~ | **Available since:** 2.1.5
*Deprecated since:* 4.0.0
*Replaced by:* sserver.scan.allowed.tables
~~A regular expression that determines which tables are allowed to be scanned for servers in the specified group. The property name should end with the scan server group and the property value should take into account the table namespace and name. The default value disallows scans on tables in the accumulo namespace.~~ | +| ~~ **sserver.scan.allowed.tables.group.***~~ | **Available since:** 2.1.5
*Deprecated since:* 4.0.0
*Replaced by:* sserver.scan.allowed.tables
~~A regular expression that determines which tables are allowed to be scanned for servers in the specified group. The property name should end with the scan server group and the property value should take into account the table namespace and name. The default value disallows scans on tables in the accumulo namespace.~~ | | **sserver.scan.executors.*** | **Available since:** 2.1.0
Prefix for defining executors to service scans. See [scan executors]({% durl administration/scan-executors %}) for an overview of why and how to use this property. For each executor the number of threads, thread priority, and an optional prioritizer can be configured. To configure a new executor, set `sserver.scan.executors..threads=`. Optionally, can also set `sserver.scan.executors..priority=`, `sserver.scan.executors..prioritizer=`, and `sserver.scan.executors..prioritizer.opts.=`.
| | sserver.scan.executors.default.prioritizer | **Available since:** 2.1.0
Prioritizer for the default scan executor. Defaults to none which results in FIFO priority. Set to a class that implements org.apache.accumulo.core.spi.scan.ScanPrioritizer + " to configure one.

**type:** STRING, **zk mutable:** yes, **default value:** empty | | sserver.scan.executors.default.threads | **Available since:** 2.1.0
The number of threads for the scan executor that tables use by default.
**type:** COUNT, **zk mutable:** yes, **default value:** `16` | diff --git a/_docs-4/getting-started/quickstart.md b/_docs-4/getting-started/quickstart.md index 2b9485e56..d9f60e394 100644 --- a/_docs-4/getting-started/quickstart.md +++ b/_docs-4/getting-started/quickstart.md @@ -281,10 +281,10 @@ When finished, use the following commands to stop Accumulo: [accumulo-client.properties]: {% durl configuration/files#accumulo-clientproperties %} [gc]: {% durl configuration/files#gc %} [monitor]: {% durl configuration/files#monitor %} -[manager]: {% durl configuration/files#managers %} -[tserver]: {% durl configuration/files#tservers %} +[manager]: {% durl configuration/files#manager %} +[tserver]: {% durl configuration/files#tserver %} [tracer]: {% durl configuration/files#tracers %} -[compactor]: {% durl configuration/files#compaction%20compactor %} +[compactor]: {% durl configuration/files#compactor %} [sserver]: {% durl configuration/files#sserver %} [Uno]: https://github.com/apache/accumulo-uno [Erasure Coding]: https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html diff --git a/_docs-4/security/kerberos.md b/_docs-4/security/kerberos.md index 8cbf6da47..334cd9bf9 100644 --- a/_docs-4/security/kerberos.md +++ b/_docs-4/security/kerberos.md @@ -148,9 +148,6 @@ in `accumulo.properties`. | {% plink instance.security.authenticator %} | {% jlink -f org.apache.accumulo.server.security.handler.KerberosAuthenticator %} | Configures Accumulo to use the Kerberos principal as the Accumulo username/principal | {% plink instance.security.authorizor %} | {% jlink -f org.apache.accumulo.server.security.handler.KerberosAuthorizor %} | Configures Accumulo to use the Kerberos principal for authorization purposes | {% plink instance.security.permissionHandler %} | {% jlink -f org.apache.accumulo.server.security.handler.KerberosPermissionHandler %} | Configures Accumulo to use the Kerberos principal for permission purposes -| {% plink trace.token.type %} | {% jlink -f org.apache.accumulo.core.client.security.tokens.KerberosToken %} | Configures the Accumulo Tracer to use the [KerberosToken] for authentication when serializing traces to the trace table. -| {% plink trace.user %} | accumulo/_HOST@REALM | The tracer process needs valid credentials to serialize traces to Accumulo. While the other server processes are creating a SystemToken from the provided keytab and principal, we can still use a normal [KerberosToken] and the same keytab/principal to serialize traces. Like non-Kerberized instances, the table must be created and permissions granted to the trace.user. The same `_HOST` replacement is performed on this value, substituted the FQDN for `_HOST`. -| trace.token.property.keytab | | You can optionally specify the path to a keytab file for the principal given in the `trace.user` property. If you don't set this path, it will default to the value given in `general.kerberos.principal`. | {% plink general.delegation.token.lifetime %} | 7d | The length of time that the server-side secret used to create delegation tokens is valid. After a server-side secret expires, a delegation token created with that secret is no longer valid. | {% plink general.delegation.token.update.interval %} | 1d | The frequency in which new server-side secrets should be generated to create delegation tokens for clients. Generating new secrets reduces the likelihood of cryptographic attacks. diff --git a/_docs-4/security/on-disk-encryption.md b/_docs-4/security/on-disk-encryption.md index bf9a2c031..069d38ca5 100644 --- a/_docs-4/security/on-disk-encryption.md +++ b/_docs-4/security/on-disk-encryption.md @@ -25,7 +25,7 @@ instance.crypto.opts.factory=org.apache.accumulo.core.spi.crypto.GenericCryptoSe ``` The `GenericCryptoServiceFactory` requires configuring a crypto service to load and this can be done by setting the -{% plink general.custom.crypto.service %} property. The value of this property is the +`general.custom.crypto.service` property. The value of this property is the class name of the service which will perform crypto on RFiles and WALs. ``` general.custom.crypto.service=org.apache.accumulo.core.spi.crypto.AESCryptoService @@ -40,7 +40,7 @@ instance.crypto.opts.factory=org.apache.accumulo.core.spi.crypto.PerTableCryptoS ``` The `PerTableCryptoServiceFactory` requires configuring a crypto service to load for the table RFiles and this can be done by adding the -{% plink table.crypto.opts.service %} property to a table. Example in the accumulo shell: +`table.crypto.opts.service` property to a table. Example in the accumulo shell: ``` createtable table1 -prop table.crypto.opts.service=org.apache.accumulo.core.spi.crypto.AESCryptoService ``` @@ -53,7 +53,7 @@ general.custom.crypto.wal.service=org.apache.accumulo.core.spi.crypto.AESCryptoS Out of the box, Accumulo provides the `AESCryptoService` for basic encryption needs. This class provides AES encryption with Galois/Counter Mode (GCM) for RFiles and Cipher Block Chaining (CBC) mode for WALs. The additional property -below is required by this crypto service to be set using the {% plink general.custom.crypto.\* %} prefix. +below is required by this crypto service to be set using the {% plink general.custom.\* %} prefix. ``` general.custom.crypto.key.uri=file:///secure/path/to/crypto-key-file ``` diff --git a/_docs-4/troubleshooting/basic.md b/_docs-4/troubleshooting/basic.md index 1b756ba2b..7d5af03b6 100644 --- a/_docs-4/troubleshooting/basic.md +++ b/_docs-4/troubleshooting/basic.md @@ -241,6 +241,6 @@ Node count: 22524 Check zookeeper status, verify that it has a quorum, and has not exceeded maxClientCnxns. -[rfile-info]: {% durl troubleshooting/tools#RFileInfo %} +[rfile-info]: {% durl troubleshooting/tools#rfileinfo %} [native-maps]: {% durl administration/in-depth-install#native-map %} [zk-4lw]: https://zookeeper.apache.org/doc/r3.5.7/zookeeperAdmin.html#sc_4lw