Blog

Get started with HashiCorp Nomad & Consul

From 0 to 100 with HashiCorps Nomad & Consul including initial server setup, load balancing, service connection any much more.

Get started with HashiCorp Nomad & Consul
08 Apr 2023
|
3 min read

About this Guide

This post will guide you through the initial setup of Nomad, Consul and Vault.

Additionally, I will cover some common additional steps for

  • AWS CLI (for ECR)

  • Docker (+ Authentication)

1. Prequisites

CNI Bridge

Nomad uses CNI plugins to configure the network namespace used to secure the Consul service mesh sidecar proxy. All Nomad client nodes using network namespaces must have CNI plugins installed. See the Consul CNI Docs for more information.

See Nomad Install Docs for more information

1curl -L -o cni-plugins.tgz "https://github.com/containernetworking/plugins/releases/download/v1.0.0/cni-plugins-linux-$( [ $(uname -m) = aarch64 ] && echo arm64 || echo amd64)"-v1.0.0.tgz && \
2  sudo mkdir -p /opt/cni/bin && \
3  sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz

Configure environment values

This script configures multiple env values NOMAD_ADDR, VAULT_ADDR, CONSUL_HTTP_ADDR so we can run cli commands without appending the address and port every time.

1PRIVATE_IP=$(/sbin/ip -o -4 addr list ens18 | awk '{print $4}' | cut -d/ -f1)
2
3echo -e "\nexport NOMAD_ADDR=http://$PRIVATE_IP:4646" >> /root/.bashrc
4echo -e "export VAULT_ADDR=https://$PRIVATE_IP:8200" >> /root/.bashrc
5echo -e "export CONSUL_HTTP_ADDR=http://$PRIVATE_IP:8500" >> /root/.bashrc
6source /root/.bashrc

2. Installation

Get private interface IP

You need to obtain the private IP of your chosen network interface. Check if the command below fits your needs or set the IP address manually.

1PRIVATE_IP=$(/sbin/ip -o -4 addr list ens18 | awk '{print $4}' | cut -d/ -f1)
2
3echo $PRIVATE_IP

Install require packages

1apt-get update && apt-get upgrade -y
2apt-get install curl wget gpg gnupg coreutils ca-certificates lsb-release
3
4
5# AWS CLI
6apt-get install -y awscli amazon-ecr-credential-helper

Install Docker

See the official Docker install guide for more information.

1curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
2
3echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
4
5apt-get update
6apt-get install -y docker-ce docker-ce-cli containerd.io

Add HashiCorps PPAs

See the official install guide if your prefer to use the prebuilt binary.

1wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
2
3echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

Install Nomad, Consul & Vault

1apt-get update
2apt-get install -y nomad consul vault
3
4systemctl enable nomad
5systemctl enable consul
6systemctl enable vault

3. Configuration

3.1 Docker AWS ECR Authentication

We will create a new /etc/docker/config.json file to provide Nomad with our Docker login credentials. Replace <aws_id> and <aws_region> with your own values.

 1mkdir -p /etc/docker
 2
 3cat <<EOT >> /etc/docker/config.json
 4{
 5    "credHelpers": {
 6        "public.ecr.aws": "ecr-login",
 7        "<aws_id>.dkr.ecr.<aws_region>.amazonaws.com": "ecr-login"
 8    }
 9}
10EOT

3.2 Nomad

Before getting started with Nomad instances, we need to configure some environment values in /etc/nomad.d/nomad.env

 1mkdir -p /etc/nomad.d
 2
 3cat <<EOT >> /etc/nomad.d/nomad.env
 4AWS_ACCESS_KEY_ID=******
 5AWS_SECRET_ACCESS_KEY=******
 6AWS_DEFAULT_REGION=<aws_region>
 7
 8VAULT_ADDR=http://127.0.0.1:8200
 9VAULT_TOKEN=
10
11CONSUL_HTTP_ADDR=$PRIVATE_IP:8500
12CONSUL_CACERT=/etc/consul.d/certs/consul-agent-ca.pem
13CONSUL_CLIENT_CERT=/etc/consul.d/certs/dc1-server-consul.pem
14CONSUL_CLIENT_KEY=/etc/consul.d/certs/dc1-server-consul-key.pem
15CONSUL_HTTP_SSL=false
16EOT

We will now configure your Nomad instances as client and/or server. Place this file in /etc/nomad.d/nomad.hcl

1rm -f /etc/nomad.d/nomad.hcl && nano /etc/nomad.d/nomad.hcl

Nomad Client

 1datacenter = "dc1"
 2data_dir   = "/opt/nomad"
 3bind_addr  = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
 4
 5server {
 6  enabled = false
 7}
 8
 9client {
10  enabled = true
11  network_interface = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"name\" }}"
12
13  template {
14    disable_file_sandbox = true
15  }
16}
17
18consul {}
19
20plugin "docker" {
21  config {
22    volumes {
23      enabled = true
24    }
25    auth {
26      config = "/etc/docker/config.json"
27    }
28  }
29}

Nomad Client & Server

 1datacenter = "dc1"
 2data_dir   = "/opt/nomad"
 3bind_addr  = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
 4
 5server {
 6  enabled          = true
 7  bootstrap_expect = 3
 8}
 9
10client {
11  enabled = true
12  network_interface = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"name\" }}"
13
14  template {
15    disable_file_sandbox = true
16  }
17}
18
19consul {}
20
21plugin "docker" {
22  config {
23    volumes {
24      enabled = true
25    }
26    auth {
27      config = "/etc/docker/config.json"
28    }
29  }
30}

Some values explained

  • The server server.bootstrap_expect values defined, how many nomad server instances need to be running in order to select a leader.

  • The client.template.disable_file_sandbox allows you to mount host files into you job allcos.

3.3 Consul

Setup TLS & encryption (optional)

To enable internal TLS encryption we need to generate a certificate using the following commands. See the Consul TLS docs for more information.

1mkdir -p /etc/consul.d/certs && cd /etc/consul.d/certs
2
3consul keygen
4# UyaZRVMUdoNinDtEDxMZFiqpQmjbsIQXUeGYDWgi=
5
6consul tls ca create -domain consul
7# ==> Saved consul-agent-ca.pem
8# ==> Saved consul-agent-ca-key.pem

You now need to distribute the generated consul-agent-ca.pem certificate to all consul agents and place it in /etc/consul.d/certs/consul-agent-ca.pem

Generate agent certificates

On your host Consul server, generate an agent certificate for each Consul agent you want to deploy.

1consul tls cert create -server -dc dc1 -domain consul
2# ==> WARNING: Server Certificates grants authority to become a
3#     server and access all state in the cluster including root keys
4#     and all ACL tokens. Do not distribute them to production hosts
5#     that are not server nodes. Store them as securely as CA keys.
6# ==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
7# ==> Saved dc1-server-consul-0.pem
8# ==> Saved dc1-server-consul-0-key.pem

Distribute your agent certificate to the respective server

1scp 10.1.10.1:/etc/consul.d/certs/dc1-server-consul-1-key.pem .
2scp 10.1.10.1:/etc/consul.d/certs/dc1-server-consul-1.pem .

Configure each agent

Again, choose which servers will be used as client or server for your Consul instances.

 1mkdir -p /etc/consul.d/certs
 2chown -R consul:consul /etc/consul.d
 3chown -R consul:consul /opt/consul
 4
 5rm -f /etc/consul.d/consul.hcl && nano /etc/consul.d/consul.hcl
 6chmod 640 /etc/consul.d/consul.hcl
 7
 8# Paste consul keys
 9/etc/consul.d/certs/dc1-server-consul.pem
10/etc/consul.d/certs/dc1-server-consul-key.pem
11
12# Bootstrap ACL
13consul acl bootstrap

Consul Client

 1datacenter = "dc1"
 2data_dir   = "/opt/consul"
 3
 4bind_addr   = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
 5client_addr = "{{ GetPrivateInterfaces | exclude \"name\" \"docker.*\" | join \"address\" \" \" }} {{ GetAllInterfaces | include \"flags\" \"loopback\" | join \"address\" \" \" }}"
 6
 7retry_join = ["<add all o your consul clients & server ip addresses>"] # also include this server ip
 8
 9ca_file   = "/etc/consul.d/certs/consul-agent-ca.pem"
10cert_file = "/etc/consul.d/certs/dc1-server-consul.pem"
11key_file  = "/etc/consul.d/certs/dc1-server-consul-key.pem"
12
13tls {
14  grpc {
15    use_auto_cert = false
16  }
17}
18
19ports {
20  grpc     = 8502
21  grpc_tls = -1
22}
23
24connect {
25  enabled = true
26}
27
28dns_config {
29  allow_stale   = true
30  node_ttl      = "5s"
31  use_cache     = true
32  cache_max_age = "5s"
33}
34
35log_level  = "info"

Consul Server

 1datacenter = "dc1"
 2data_dir   = "/opt/consul"
 3
 4server           = true
 5bootstrap_expect = 3    # your server count
 6
 7bind_addr   = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
 8client_addr = "{{ GetPrivateInterfaces | exclude \"name\" \"docker.*\" | join \"address\" \" \" }} {{ GetAllInterfaces | include \"flags\" \"loopback\" | join \"address\" \" \" }}"
 9
10retry_join = ["<add all o your consul clients & server ip addresses>"] # also include this server ip
11
12ca_file   = "/etc/consul.d/certs/consul-agent-ca.pem"
13cert_file = "/etc/consul.d/certs/dc1-server-consul-0.pem"
14key_file  = "/etc/consul.d/certs/dc1-server-consul-0-key.pem"
15
16ui_config {
17  enabled = true
18}
19
20acl {
21  enabled                  = true
22  default_policy           = "allow" # change this
23  enable_token_persistence = true
24}
25
26tls {
27  grpc {
28    use_auto_cert = false
29  }
30}
31
32ports {
33  grpc     = 8502
34  grpc_tls = -1
35}
36
37connect {
38  enabled = true
39}
40
41dns_config {
42  allow_stale   = true
43  node_ttl      = "5s"
44  use_cache     = true
45  cache_max_age = "5s"
46}
47
48log_level  = "info"

4. Cheat-Sheet

Here are some handy commands I commonly use for debugging.

 1# nomad cleanup allocation history & summary
 2nomad system gc
 3nomad system reconcile summaries
 4
 5# show service logs
 6nomad monitor
 7
 8# attach shell to job
 9nomad alloc exec -task=<task> <alloc> /bin/bash
10
11# show open ports
12lsof -i -P -n | grep LISTEN
13ss -tulpn
14
15# test tcp connection
16nc -z -v -w 2 <host> <port>
17
18# test consul dns
19dig @127.0.0.1 -p 8600 _<service-name>._tcp.service.consul
20
21# query container from inside
22curl -H "Host: domain.tld" 10.1.10.1:21021
23
24# query some endpoint
25curl -H "Host: domain.tld" -X POST <host>/api
26
27# list service instances "address:port"
28curl -s http://127.0.0.1:8500/v1/catalog/service/<service-name>|jq -j '.[] | .ServiceAddress,":",.ServicePort,"\n"'
29
30# consul filtering
31curl --get http://127.0.0.1:8500/v1/agent/services --data-urlencode 'filter=Service == "<service-name>"'|jq -j '.[] | .Address,":",.Port,"\n"'

Read more...

Fix Laravel Job Queue not processing with MaxAttemptsExceededException

Fix Laravel Job Queue not processing with MaxAttemptsExceededException

Fix issues with the Laravel Job Queue not processing jobs due to max attempts exceeding.

12 Jan 2025
|
2 min read
2025 Edition: Best macOS Apps

2025 Edition: Best macOS Apps

It's January, new me? New apps! Again a collection of macOS & iOS apps I learned to love in the last year.

11 Jan 2025
|
2 min read