Introduction to ETCD (Hands on)
ETCD is an open-source key value store that is used for configuration, transnational locks, elections, service discovery, distributed queue and more. It has been referred to as "the heart of cloud native" as well. It is normally deployed in a distributed manner, multiple nodes in a cluster. The other open-source tools that do similar things are Consul and Apache Zookeeper.
This blog will not go into architecture and deployment of a production system, for that please refer to Kelsey Hightower's video here. This blog will focus on interacting with / leveraging ETCD, once it is up and running.
At a high level ETCD can be used for any micro-service and is often used for container clusters. ETCD V2 is now the older release (but still highly used) and V3 is the new release. The command line tool for ETCD is called etcdctl. By default etcdctl calls version 2. Version 3 adds some new features and changes syntax in quite a few of the commands. To use V3 set the environment variable as shown in figure 1.
$ export ETCDCTL_API=3
Figure 1.
The figures 2 - 7 highlights some of the differences in commands between V2 and V3.
Version 2 write to keystore:
$ etcdctlset/fname John John
Figure 2.
$ ETCDCTL_API=3 etcdctlputfname John OK
Figure 3.
$ etcdctlget/fname John
Figure 4.
$ ETCDCTL_API=3 etcdctlgetfname fname John
Figure 5.
$ etcdctlrm/fname PrevNode.Value: John
Figure 6.
$ ETCDCTL_API=3 etcdctldelfname 1
Figure 7.
$ etcdctl ls / --recursive
/name
$ ETCDCTL_API=3 etcdctl get --prefix ""
id
342323
Figure 8.
New Features in V3
NOTE: The following is not an exhaustive list as each release in V3 adds more capability and/or scale.Many new scale enhancements have been put into version 3 allowing for faster concurrent processing. For instance the ETCDv2 API was based on REST/Json and could only handle so many clients and keys. In ETCDv3 the REST/Json API is still there (more on that later), but GRPC was implemented to handle faster processing of records. If both API's are used in version 3, then records won't be shared between the two as the API's are isolated from each other (related to statement above leveraging etcdctl tool).
Another cool feature addition are transactions (aka txn). Transactions allow you to do atomic updates to sets of keys. A transaction is composed of an if(cond1, cond2), then(op1, op2), else(op1, op2) type structure (much like many programming languages).
As an example, suppose you wanted to check to see if a key exists before you insert, or you may want to create a lock variable and check if it is locked before trying to alter the record This can be done with transactions, shown in firgures 9 and 10.
The first example is checking if a lock is present. If not, then it sets lock to true and writes data then commits in one transaction, as shown in figure 9.
$ ETCDCTL_API=3 etcdctl txn -i
compares:
mod("PL32343/lock") = "0"
success requests (get, put, del):
put PL32343/lock 1
put PL32343/type "test type"
failure requests (get, put, del):
get PL32343/type
SUCCESS
OK
OK
$ ETCDCTL_API=3 etcdctl get --prefix ""
PL32343/lock
1
PL32343/type
test type
Figure 9. Interactive Mode
ETCDCTL_API=3 etcdctl txn <<<'mod("PL32343/lock") = "0"
put PL32343/lock 1
put PL32343/type "test type"
get PL32343/type
'
FAILURE
PL32343/type
test type
Figure 10. Non-Interactive Mode
NOTE: When using etcdctl tool in these modes it is important to know that spacing is strict in rows as well as columns for separation.
ETCDv3 API
The ETCDv3 REST API is very different vs the V2 REST API. This is due to the fact that the V3 REST API is generated from the protocol buffers file via grpc-gateway. In ETCDv2 you use standard REST methods like GET, PUT, DELETE, ... to interact. Another difference is that the V2 key is treated as a hierarchy vs V3's string key. For example the key "PL32343/type" will return the following: $ etcdctl ls / --recursive
/PL99337
/PL99337/type
Figure 11.
If you check the key "PL99337" it returns you a pointer to another key "/PL99337/type" that has a value. Shown in Figure 12.
$ curl -X GET http://127.0.0.1:2379/v2/keys/PL99337 | jq
{
"action": "get",
"node": {
"key": "/PL99337",
"dir": true,
"nodes": [
{
"key": "/PL99337/type",
"value": "test 3",
"modifiedIndex": 19,
"createdIndex": 19
}
],
"modifiedIndex": 19,
"createdIndex": 19
}
}
Figure 12.
REST
The REST API was derived from grpc-gateway and follows the protocol buffers file very closely. To get sample curl commands you need to read the protocol buffers file to derive your endpoints. The following is an excerpts from the protobuf file located as follows: github.com/coreos/etcd/blob/master/etcdserver/etcdserverpb/rpc.proto
1: service KV {
2: // Range gets the keys in the range from the key-value store.
3: rpc Range(RangeRequest) returns (RangeResponse) {
4: option (google.api.http) = {
5: post: "/v3alpha/kv/range"
6: body: "*"
7: };
8: }
9: // Put puts the given key into the key-value store.
10: // A put request increments the revision of the key-value store
11: // and generates one event in the event history.
12: rpc Put(PutRequest) returns (PutResponse) {
13: option (google.api.http) = {
14: post: "/v3alpha/kv/put"
15: body: "*"
16: };
17: }
18: }
19: message PutRequest {
20: // key is the key, in bytes, to put into the key-value store.
21: bytes key = 1;
22: // value is the value, in bytes, to associate with the key in the key-value store.
23: bytes value = 2;
24: // lease is the lease ID to associate with the key in the key-value store. A lease
25: // value of 0 indicates no lease.
26: int64 lease = 3;
27: // If prev_kv is set, etcd gets the previous key-value pair before changing it.
28: // The previous key-value pair will be returned in the put response.
29: bool prev_kv = 4;
30: // If ignore_value is set, etcd updates the key using its current value.
31: // Returns an error if the key does not exist.
32: bool ignore_value = 5;
33: // If ignore_lease is set, etcd updates the key using its current lease.
34: // Returns an error if the key does not exist.
35: bool ignore_lease = 6;
36: }
Figure 13.
$ curl -X POST -d '{"key": "UEwzMjM0My90eXBl", "value": "dGVzdCB0eXBl"}' http://127.0.0.1:2379/v3alpha/kv/put | jq
{
"header": {
"cluster_id": "14841639068965178418",
"member_id": "10276657743932975437",
"revision": "121",
"raft_term": "8"
}
}
Figure 14.
$ curl -X POST -d '{"key": "UEwzMjM0My90eXBl"}' http://localhost:2379/v3alpha/kv/range | jq
{
"header": {
"cluster_id": "14841639068965178418",
"member_id": "10276657743932975437",
"revision": "120",
"raft_term": "8"
},
"kvs": [
{
"key": "UEwzMjM0My90eXBl",
"create_revision": "115",
"mod_revision": "115",
"version": "1",
"value": "dGVzdCB0eXBl"
}
],
"count": "1"
}
Figure 15.
GRPC
Other Operations
Watch
Another interesting feature of etcd is the ability to watch for activity on a given key. In the following example we are watching for changes in the PL32343/lock key on system 1 and changing the key on system 2.System1:
$ ETCDCTL_API=3 etcdctl watch PL32343/lock
PUT
PL32343/lock
0
Figure 16.
$ ETCDCTL_API=3 etcdctl put PL32343/lock 0
OK
Figure 17.
We see that system 2 set the key to zero and system 1 sees the key and the new value and then enters watching the key again.
Lease
Another difference between V2 and V3 is that TTL was changed to Lease. The idea is to put a timer on a value and have it expire after some period of time. V3 changes how you have to accomplish this by creating the lease first and then assigning a lease to a new key/value pair. Figure 18 shows creating a lease and assigning the lease to a variable. $ ETCDCTL_API=3 etcdctl lease grant 60
lease 694d5e63073a7d9e granted with TTL(60s)
$ ETCDCTL_API=3 etcdctl put lname doe --lease=694d5e63073a7d9e
OK
$ ETCDCTL_API=3 etcdctl get lname
lname
doe
Figure 18.
Figure 19 shows that after 60 seconds the value has expired.
$ ETCDCTL_API=3 etcdctl get lname
Figure 19.
I had the thought of doing the lease creation and the assignment in one transaction using the etcdctl tool, the problem is that transactions are atomic and you wouldn't be able to derive the lease number in order to assign it in your transaction. But lets face it, your not going to implement a micro-service that leverages ETCDv3 using the etcdctl tool. Using a programming language and the proper libraries you can create the lease and do a transaction fast enough.
I hope this provides you a better way to understand the differences between ETCD v2 and V3, as well as, provide a way of cracking the code that is V3. Hopefully you can see the power that the various capabilities combined could provide your micro-service. In a future post I'll show some of those combinations and how you can use them together.
