by Example
Regional and Cross Region Streams (Supercluster) in Use Cases
This example demonstrates the configuration required to setup a three regional clusters and a cross-region cluster that form a supercluster.
The use case is for streams that are resilient to regional failures by having replicas spread out across three different regions. This of course comes at a cost of latency, but this trade-off may be appropriate for streams that are critical for availability, but can tolerate slightly higher latencies.
$ nbe run use-cases/cross-region-streams-supercluster/cliView the source code or learn how to run this example yourself
Code
#!/bin/bash
set -euo pipefail
NATS_URL="nats://localhost:4222"
Create a shared configuration which enables JetStream and defines the
unique_tag option which enforces all replicas for a given stream or
consumer to be placed on nodes with different availability zones (AZ).
cat <<- EOF > regional-shared.conf
jetstream: {
unique_tag: "az:"
}
accounts: {
\$SYS: {
users: [{user: sys, password: sys}]
}
APP: {
jetstream: true
users: [{user: user, password: user}]
}
}
EOF
The cross-region cluster defines a unique_tag on the cluster. Any AZ can be chosen as long as the region is unique.
cat <<- EOF > cross-region-shared.conf
jetstream: {
unique_tag: "rg:"
}
accounts: {
\$SYS: {
users: [{user: sys, password: sys}]
}
APP: {
jetstream: true
users: [{user: user, password: user}]
}
}
EOF
cat <<- EOF > gateway-routes.conf
gateways: [
{name: rg1, urls: [
nats://localhost:7222,
nats://localhost:7223,
nats://localhost:7224,
]},
{name: rg2, urls: [
nats://localhost:7225,
nats://localhost:7226,
nats://localhost:7227,
]},
{name: rg3, urls: [
nats://localhost:7228,
nats://localhost:7229,
nats://localhost:7230,
]},
{name: xr, urls: [
nats://localhost:7231,
nats://localhost:7232,
nats://localhost:7233,
]},
]
EOF
Define the server configs for region 1 cluster.
cat <<- EOF > "rg1-az1.conf"
server_name: rg1-az1
server_tags: [az:1]
port: 4222
http_port: 8222
include regional-shared.conf
cluster: {
name: rg1
port: 6222
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
}
gateway {
name: rg1
port: 7222
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az1
}
EOF
cat <<- EOF > "rg1-az2.conf"
server_name: rg1-az2
server_tags: [az:2]
port: 4223
include regional-shared.conf
cluster: {
name: rg1
port: 6223
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
}
gateway {
name: rg1
port: 7223
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az2
}
EOF
cat <<- EOF > "rg1-az3.conf"
server_name: rg1-az3
server_tags: [az:3]
port: 4224
include regional-shared.conf
cluster: {
name: rg1
port: 6224
routes: [
nats-route://127.0.0.1:6222,
nats-route://127.0.0.1:6223,
nats-route://127.0.0.1:6224,
]
}
gateway {
name: rg1
port: 7224
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az3
}
EOF
Server configs for region 2 cluster.
cat <<- EOF > "rg2-az1.conf"
server_name: rg2-az1
server_tags: [az:1]
port: 4225
http_port: 8223
include regional-shared.conf
cluster: {
name: rg2
port: 6225
routes: [
nats-route://127.0.0.1:6225,
nats-route://127.0.0.1:6226,
nats-route://127.0.0.1:6227,
]
}
gateway {
name: rg2
port: 7225
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az1
}
EOF
cat <<- EOF > "rg2-az2.conf"
server_name: rg2-az2
server_tags: [az:2]
port: 4226
include regional-shared.conf
cluster: {
name: rg2
port: 6226
routes: [
nats-route://127.0.0.1:6225,
nats-route://127.0.0.1:6226,
nats-route://127.0.0.1:6227,
]
}
gateway {
name: rg2
port: 7226
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az2
}
EOF
cat <<- EOF > "rg2-az3.conf"
server_name: rg2-az3
server_tags: [az:3]
port: 4227
include regional-shared.conf
cluster: {
name: rg2
port: 6227
routes: [
nats-route://127.0.0.1:6225,
nats-route://127.0.0.1:6226,
nats-route://127.0.0.1:6227,
]
}
gateway {
name: rg2
port: 7227
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az3
}
EOF
Server configs for region 3 cluster.
cat <<- EOF > "rg3-az1.conf"
server_name: rg3-az1
server_tags: [az:1]
port: 4228
http_port: 8224
include regional-shared.conf
cluster: {
name: rg3
port: 6228
routes: [
nats-route://127.0.0.1:6228,
nats-route://127.0.0.1:6229,
nats-route://127.0.0.1:6230,
]
}
gateway {
name: rg3
port: 7228
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az1
}
EOF
cat <<- EOF > "rg3-az2.conf"
server_name: rg3-az2
server_tags: [az:2]
port: 4229
include regional-shared.conf
cluster: {
name: rg3
port: 6229
routes: [
nats-route://127.0.0.1:6228,
nats-route://127.0.0.1:6229,
nats-route://127.0.0.1:6230,
]
}
gateway {
name: rg3
port: 7229
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az2
}
EOF
cat <<- EOF > "rg3-az3.conf"
server_name: rg3-az3
server_tags: [az:3]
port: 4230
include regional-shared.conf
cluster: {
name: rg3
port: 6230
routes: [
nats-route://127.0.0.1:6228,
nats-route://127.0.0.1:6229,
nats-route://127.0.0.1:6230,
]
}
gateway {
name: rg3
port: 7230
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az3
}
EOF
Server configs for cross-region cluster. In this case, an arbitrary AZ is chosen per region. If may be desirable to have a six or nine node cross-region cluster if more AZs per region are desired.
cat <<- EOF > "rg1-az1-x.conf"
server_name: rg1-az1-x
server_tags: [rg:1, az:1]
port: 4231
http_port: 8225
include cross-region-shared.conf
cluster: {
name: xr
port: 6231
routes: [
nats-route://127.0.0.1:6231,
nats-route://127.0.0.1:6232,
nats-route://127.0.0.1:6233,
]
}
gateway {
name: xr
port: 7231
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg1-az1-x
}
EOF
cat <<- EOF > "rg2-az2-x.conf"
server_name: rg2-az2-x
server_tags: [rg:2, az:2]
port: 4232
include cross-region-shared.conf
cluster: {
name: xr
port: 6232
routes: [
nats-route://127.0.0.1:6231,
nats-route://127.0.0.1:6232,
nats-route://127.0.0.1:6233,
]
}
gateway {
name: xr
port: 7232
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg2-az2-x
}
EOF
cat <<- EOF > "rg3-az3-x.conf"
server_name: rg3-az3-x
server_tags: [rg:3, az:3]
port: 4233
include cross-region-shared.conf
cluster: {
name: xr
port: 6233
routes: [
nats-route://127.0.0.1:6231,
nats-route://127.0.0.1:6232,
nats-route://127.0.0.1:6233,
]
}
gateway {
name: xr
port: 7233
include gateway-routes.conf
}
jetstream: {
store_dir: /tmp/nats/storage/rg3-az3-x
}
EOF
Start a server for each configuration and sleep a second in between so the seeds can startup and get healthy.
for c in $(ls rg*.conf); do
echo "Starting server ${c%.*}"
nats-server -c $c -l "${c%.*}.log" > /dev/null 2>&1 &
sleep 1
done
Wait until the servers up and healthy.
echo 'Cluster 1 healthy?'
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8222/healthz; echo
echo 'Cluster 2 healthy?'
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8223/healthz; echo
echo 'Cluster 3 healthy?'
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8224/healthz; echo
echo 'Cluster 4 healthy?'
curl --fail --silent \
--retry 5 \
--retry-delay 1 \
http://localhost:8225/healthz; echo
Show the server lit and JetStream report.
nats --user sys --password sys server info rg2-az1
nats --user sys --password sys server list
nats --user sys --password sys server report jetstream
Create a cross-region stream specifying the xr cluster.
nats --user user --password user stream add \
--cluster=xr \
--retention=limits \
--storage=file \
--replicas=3 \
--discard=old \
--dupe-window=2m \
--max-age=-1 \
--max-msgs=-1 \
--max-bytes=-1 \
--max-msg-size=-1 \
--max-msgs-per-subject=-1 \
--max-consumers=-1 \
--allow-rollup \
--no-deny-delete \
--no-deny-purge \
--subjects="events.*" \
EVENTS
Create a regional stream specifying one of the regional clusters, e.g.
rg2.
nats --user user --password user stream add \
--cluster=rg2 \
--retention=limits \
--storage=file \
--replicas=3 \
--discard=old \
--dupe-window=2m \
--max-age=-1 \
--max-msgs=-1 \
--max-bytes=-1 \
--max-msg-size=-1 \
--max-msgs-per-subject=-1 \
--max-consumers=-1 \
--allow-rollup \
--no-deny-delete \
--no-deny-purge \
--subjects="orders.*" \
ORDERS
Report out the streams.
nats --user user --password user stream reportOutput
Starting server rg1-az1-x
Starting server rg1-az1
Starting server rg1-az2
Starting server rg1-az3
Starting server rg2-az1
Starting server rg2-az2-x
Starting server rg2-az2
Starting server rg2-az3
Starting server rg3-az1
Starting server rg3-az2
Starting server rg3-az3-x
Starting server rg3-az3
Cluster 1 healthy?
{"status":"ok"}
Cluster 2 healthy?
{"status":"ok"}
Cluster 3 healthy?
{"status":"ok"}
Cluster 4 healthy?
{"status":"ok"}
Server information for rg2-az1 (NCYRXWEEZTZVL76ACBYAX6GGSFWCKGO7RX6U5EI6CLAJ2YVHDMPSPEPV)
Process Details:
Version: 2.10.1
Git Commit:
Go Version: go1.21.3
Start Time: 2023-10-23 19:31:26
Config Load Time: 2023-10-23 19:31:26
Uptime: 11s
Connection Details:
Auth Required: true
TLS Required: false
Host: 0.0.0.0:4225
Client URLs: 172.29.0.2:4225
172.29.0.2:4226
172.29.0.2:4227
JetStream:
Domain:
Storage Directory: /tmp/nats/storage/rg2-az1/jetstream
Max Memory: 5.8 GiB
Max File: 20 GiB
Active Accounts: 1
Memory In Use: 0 B
File In Use: 0 B
API Requests: 0
API Errors: 0
Always sync writes to disk: false
Write sync Frequency: 2m0s
Limits:
Max Conn: 65,536
Max Subs: 0
Max Payload: 1.0 MiB
TLS Timeout: 2.00s
Write Deadline: 10.00s
Statistics:
CPU Cores: 8 0.50%
Memory: 17 MiB
Connections: 0
Subscriptions: 174
Messages: 145 in 247 out
Bytes: 212 KiB in 324 KiB out
Slow Consumers: 0
Cluster:
Name: rg2
Tags: az:1
Host: 0.0.0.0:6225
URLs: 127.0.0.1:6225
127.0.0.1:6226
127.0.0.1:6227
Super Cluster:
Name: rg2
Host: 0.0.0.0:7225
Clusters: rg1
rg2
rg3
xr
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Server Overview │
├───────────┬─────────┬──────┬─────────┬─────┬───────┬───────┬────────┬─────┬─────────┬───────┬───────┬──────┬────────┬─────┤
│ Name │ Cluster │ Host │ Version │ JS │ Conns │ Subs │ Routes │ GWs │ Mem │ CPU % │ Cores │ Slow │ Uptime │ RTT │
├───────────┼─────────┼──────┼─────────┼─────┼───────┼───────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ rg1-az1 │ rg1 │ 0 │ 2.10.1 │ yes │ 1 │ 184 │ 8 │ 3 │ 17 MiB │ 2 │ 8 │ 0 │ 14.14s │ 1ms │
│ rg1-az1-x │ xr │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 17 MiB │ 0 │ 8 │ 0 │ 15.14s │ 1ms │
│ rg3-az1 │ rg3 │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 17 MiB │ 2 │ 8 │ 0 │ 7.10s │ 1ms │
│ rg1-az3 │ rg1 │ 0 │ 2.10.1 │ yes │ 0 │ 184 │ 8 │ 3 │ 17 MiB │ 1 │ 8 │ 0 │ 12.12s │ 1ms │
│ rg1-az2 │ rg1 │ 0 │ 2.10.1 │ yes │ 0 │ 184 │ 8 │ 3 │ 17 MiB │ 2 │ 8 │ 0 │ 13.12s │ 1ms │
│ rg2-az2-x │ xr │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 16 MiB │ 0 │ 8 │ 0 │ 10.11s │ 1ms │
│ rg3-az3-x │ xr │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 15 MiB │ 0 │ 8 │ 0 │ 5.09s │ 1ms │
│ rg2-az1 │ rg2 │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 17 MiB │ 0 │ 8 │ 0 │ 11.11s │ 1ms │
│ rg2-az3 │ rg2 │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 16 MiB │ 1 │ 8 │ 0 │ 8.10s │ 1ms │
│ rg2-az2 │ rg2 │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 16 MiB │ 2 │ 8 │ 0 │ 9.11s │ 2ms │
│ rg3-az2 │ rg3 │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 16 MiB │ 1 │ 8 │ 0 │ 6.09s │ 2ms │
│ rg3-az3 │ rg3 │ 0 │ 2.10.1 │ yes │ 0 │ 174 │ 8 │ 3 │ 16 MiB │ 0 │ 8 │ 0 │ 4.08s │ 2ms │
├───────────┼─────────┼──────┼─────────┼─────┼───────┼───────┼────────┼─────┼─────────┼───────┼───────┼──────┼────────┼─────┤
│ │ 4 │ 12 │ │ 12 │ 1 │ 2,118 │ │ │ 198 MIB │ │ │ 0 │ │ │
╰───────────┴─────────┴──────┴─────────┴─────┴───────┴───────┴────────┴─────┴─────────┴───────┴───────┴──────┴────────┴─────╯
╭────────────────────────────────────────────────────────────────────────────╮
│ Cluster Overview │
├─────────┬────────────┬───────────────────┬───────────────────┬─────────────┤
│ Cluster │ Node Count │ Outgoing Gateways │ Incoming Gateways │ Connections │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ xr │ 3 │ 9 │ 9 │ 0 │
│ rg3 │ 3 │ 9 │ 9 │ 0 │
│ rg2 │ 3 │ 9 │ 9 │ 0 │
│ rg1 │ 3 │ 9 │ 9 │ 1 │
├─────────┼────────────┼───────────────────┼───────────────────┼─────────────┤
│ │ 12 │ 36 │ 36 │ 1 │
╰─────────┴────────────┴───────────────────┴───────────────────┴─────────────╯
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
│ JetStream Summary │
├───────────┬─────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server │ Cluster │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ rg1-az1* │ rg1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg1-az2 │ rg1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg1-az3 │ rg1 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az1 │ rg2 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az2 │ rg2 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az3 │ rg2 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az1 │ rg3 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az2 │ rg3 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az3 │ rg3 │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg1-az1-x │ xr │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg2-az2-x │ xr │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
│ rg3-az3-x │ xr │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
├───────────┼─────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ │ │ 0 │ 0 │ 0 │ 0 B │ 0 B │ 0 B │ 0 │ 0 │
╰───────────┴─────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯
╭─────────────────────────────────────────────────────────────────╮
│ RAFT Meta Group Information │
├───────────┬──────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name │ ID │ Leader │ Current │ Online │ Active │ Lag │
├───────────┼──────────┼────────┼─────────┼────────┼────────┼─────┤
│ rg1-az1 │ 6s4h5MOv │ yes │ true │ true │ 0s │ 0 │
│ rg1-az1-x │ PMcvOJvm │ │ true │ true │ 656ms │ 0 │
│ rg1-az2 │ X8wPeL6S │ │ true │ true │ 656ms │ 0 │
│ rg1-az3 │ FozYtzby │ │ true │ true │ 656ms │ 0 │
│ rg2-az1 │ 8ShHFKtZ │ │ true │ true │ 656ms │ 0 │
│ rg2-az2 │ Kx5WF0Q6 │ │ true │ true │ 656ms │ 0 │
│ rg2-az2-x │ UEoET1zo │ │ true │ true │ 656ms │ 0 │
│ rg2-az3 │ NSUWzTzT │ │ true │ true │ 656ms │ 0 │
│ rg3-az1 │ OEQTh4pP │ │ true │ true │ 656ms │ 0 │
│ rg3-az2 │ LJVb57VO │ │ true │ true │ 656ms │ 0 │
│ rg3-az3 │ uEtIi639 │ │ true │ true │ 655ms │ 0 │
│ rg3-az3-x │ cwYBGolT │ │ true │ true │ 656ms │ 0 │
╰───────────┴──────────┴────────┴─────────┴────────┴────────┴─────╯
Stream EVENTS was created
Information for Stream EVENTS created 2023-10-23 19:31:38
Subjects: events.*
Replicas: 3
Storage: File
Placement Cluster: xr
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: true
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: xr
Leader: rg3-az3-x
Replica: rg1-az1-x, current, seen 753µs ago
Replica: rg2-az2-x, current, seen 1ms ago
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Stream ORDERS was created
Information for Stream ORDERS created 2023-10-23 19:31:38
Subjects: orders.*
Replicas: 3
Storage: File
Placement Cluster: rg2
Options:
Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Direct Get: true
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: true
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited
Cluster Information:
Name: rg2
Leader: rg2-az1
Replica: rg2-az2, current, seen 448µs ago
Replica: rg2-az3, current, seen 46µs ago
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 0
Obtaining Stream stats
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Stream Report │
├────────┬─────────┬───────────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────────────────────────────┤
│ Stream │ Storage │ Placement │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├────────┼─────────┼───────────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────────────────────────────┤
│ EVENTS │ File │ cluster: xr │ 0 │ 0 │ 0 B │ 0 │ 0 │ rg1-az1-x, rg2-az2-x, rg3-az3-x* │
│ ORDERS │ File │ cluster: rg2 │ 0 │ 0 │ 0 B │ 0 │ 0 │ rg2-az1*, rg2-az2, rg2-az3 │
╰────────┴─────────┴───────────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────────────────────────────╯
Recording
Note, playback is half speed to make it a bit easier to follow.