Kubernetes – Connect to Hidden KubeAPI using Comprehensive Shell Script

KubeAPI

It is enticing to publicly expose your KubeAPI endpoint. For ease of access, using cloud based automation services (i.e GitHub Workflows) or just plain laziness. Authentication exists for sure, but what about possible exploits or ddos? If public access is not necessary, it is best to protect the perimeter.

KubeAPI <3 LB

Below is an example of a three node, all roles, cluster where the KubeAPI is load-balanced using a Nginx VM:

[Nginx VM] ---[RRD CONN]--->>> [NODE1, NODE2, NODE3]

The Nginx VM has a configuration using the Nginx stream module to round robin the KubeAPI connection (6443/tcp) from 127.0.0.1 Localhost on the Nginx VM to the Kubernetes nodes.

#load_module /usr/lib/nginx/modules/ngx_stream_module.so;

events {
 worker_connections 8192;
}

stream {
 upstream k8sapi {
  server NODE01:6443 max_fails=3 fail_timeout=2s;
  server NODE02:6443 max_fails=3 fail_timeout=2s;
  server NODE03:6443 max_fails=3 fail_timeout=2s;
 }

 server {
  listen 127.0.0.1:6443;
  listen SHARED_K8S_NETWORK_IP:6443;
  proxy_pass k8sapi;
 }
}

The SHARED_K8S_NETWORK_IP is an IP address from a special shared network where all of the nodes and the Nginx VM reside. I do this because the Kubernetes nodes are bootstrapped to the Nginx VM.

Kubeconfig and SSH Tunnel

Using the above load balancer setup, what is left to do is to edit your clients Kubeconfig. Specifically the server section:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [CA-DATA]
    server: https://127.0.0.1:6443    <-------
  name: kluster
...

And a SSH Tunnel can be executed and managed from the client to the Nginx VM using the following shell script

function kubetun() {
 local ssh_command="ssh [Nginx VM] -L 6443:localhost:6443 -N"
 local pid
  case $1 in
   start)
    echo "Starting SSH tunnel..."
    eval $ssh_command &!
   ;;
   status)
    pid=$(pgrep -f "$ssh_command")
     if [ -n "$pid" ]; then
      echo "SSH tunnel is up. PID: $pid"
     else
      echo "SSH tunnel is down."
     fi
   ;;
   stop)
     pid=$(pgrep -f "$ssh_command")
      if [ -n "$pid" ]; then
       echo "Stopping SSH tunnel..."
       kill $pid
      else
       echo "SSH tunnel is not running."
      fi
   ;;
   restart)
    kubetun stop
    kubetun start
   ;;
   test)
    curl --insecure -m 1 https://localhost:6443/livez > /dev/null 2>&1 && echo "OK - KubeAPI Reachable" || echo "FAIL - KubeAPI Unreachable"
   ;;
   *)
    echo "Usage: kubetun [start|status|stop|restart|test]"
   ;;
  esac
}

Place the above script in .zshrc for example.

Command Line

To start the tunnel

kubetun start

To stop the tunnel

kubetun stop

To restart the tunnel

kubetun restart

Check if (ssh PID) tunnel is active

kubetun status

Test connectivity to KubeAPI

kubetun test