Engineering a Zero-Trust mTLS Proxy for HashiCorp Nomad

In modern distributed systems, relying solely on perimeter security is an anti-pattern. Once a threat actor breaches the edge, internal service-to-service communication is often entirely unencrypted and vulnerable.

To solve this within our infrastructure, I engineered Phalanx, a lightweight, high-performance sidecar proxy written in Go that enforces Mutual TLS (mTLS) across all services running on HashiCorp Nomad.

The Problem with Perimeter Security

Traditional architectures operate on a "castle and moat" philosophy. You put a WAF at the edge, and assume everything inside the Virtual Private Cloud (VPC) is safe. However, in a multi-tenant Nomad cluster, microservices are constantly scheduled and rescheduled across different nodes.

If a single internal service is compromised, it can sniff traffic or impersonate other services. We needed cryptographic proof of identity for every request.

Why Go?

I chose Go for Phalanx for three specific reasons:

1. Concurrency: Go's goroutines make handling thousands of concurrent proxy connections trivial.
2. Memory Footprint: Unlike JVM-based proxies, Phalanx compiles to a single static binary running on less than 15MB of RAM, making it perfect for a sidecar architecture.
3. Standard Library: Go's crypto/tls package is battle-tested and provides low-level control over the TLS handshake process.

Implementing the mTLS Handshake

To enforce zero-trust, Phalanx intercepts outbound traffic, attaches a client certificate, and routes it to the destination. The receiving Phalanx sidecar verifies the certificate against our trusted internal Certificate Authority (CA).

Here is a simplified look at how Phalanx configures the strict TLS listener:

go
package proxy

import (
	"crypto/tls"
	"crypto/x509"
	"log"
	"net"
)

// StartSecureListener initializes the mTLS proxy server
func StartSecureListener(port string, caCertPool *x509.CertPool, serverCert tls.Certificate) {
	tlsConfig := &tls.Config{
		Certificates: []tls.Certificate{serverCert},
		ClientCAs:    caCertPool,
		// Force the client to provide a valid certificate
		ClientAuth:   tls.RequireAndVerifyClientCert,
		MinVersion:   tls.VersionTLS13,
	}

	listener, err := tls.Listen("tcp", port, tlsConfig)
	if err != nil {
		log.Fatalf("Failed to start Phalanx listener: %v", err)
	}
	defer listener.Close()

	log.Printf("🛡️ Phalanx mTLS proxy listening on %s", port)

	for {
		conn, err := listener.Accept()
		if err != nil {
			log.Printf("Connection failed: %v", err)
			continue
		}
		
		// Handle the securely authenticated connection
		go handleConnection(conn)
	}
}

Performance & Takeaways

By enforcing tls.RequireAndVerifyClientCert, we ensure that only explicitly authorized services can communicate.

The performance impact? Minimal. By utilizing TLS 1.3 and Go's optimized cryptography suite, the P99 latency overhead introduced by the sidecar handshake is under 3 milliseconds.

Building Phalanx reinforced a critical engineering principle for me: security shouldn't be bolted on as an afterthought. It should be built into the network primitive itself.