An Egress Gateway specifies the exit node/interface for a set of pods. In its most basic setup, this allows you to pick any of your nodes to act as the exit point for pods.

Instead of leaving the node on which the container is running on through its own interface, Cilium routes the packets to a different node in the same cluster and sends it along its way from there.

The examples below use example-pod as the pod to hook into the VPN. There's nothing special about this pod so I won't describe it in detail. If you just want to see this working, use any container image containing an interactive environment with curl or wget.

First, let's make sure the egress gateway functionality is working. The following resource configures egress traffic from pods with the name=example-pod to exit the cluster through node-b:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: basic-gateway
spec:
  destinationCIDRs:
  - 0.0.0.0/0
  egressGateway:
    nodeSelector:
      matchLabels:
        kubernetes.io/hostname: node-b
  egressGateways: []
  selectors:
  - podSelector:
      matchLabels:
        name: example-pod

Before applying this, make note of your external IP by running curl icanhazip.com in the pod. Apply the CiliumEgressGatewayPolicy and check again. You don't need to restart the pod. You should now see a different IP returned by the command!

If you want all traffic from this node to use a VPN, it should be enough to setup the VPN connection as usual and make sure it sets the VPN as the default Gateway.

In my case however, I only want specific traffic to use the VPN. Everything else should just use the regular uplink.

Here's a short refresher on how packets reach their destination: Each packet is addressed to a specific IP address. Usually you're not directly connected to your target though. Instead, you'll send your packet to an intermediary that is connected to other hosts. This process repeats until the packet reaches its destination. To find the next hop, the networking stack consults the route table. It is a directory of networks and next hops. You can dump this information using the ip route show command:

default via 192.168.42.1 dev enp0s31f6 proto static metric 100
10.0.0.0/24 via 10.0.7.7 dev cilium_host proto kernel src 10.0.7.7 mtu 1230
10.0.1.0/24 via 10.0.7.7 dev cilium_host proto kernel src 10.0.7.7 mtu 1230
10.0.2.0/24 via 10.0.7.7 dev cilium_host proto kernel src 10.0.7.7 mtu 1230
10.0.3.0/24 via 10.0.7.7 dev cilium_host proto kernel src 10.0.7.7 mtu 1230
10.0.5.0/24 via 10.0.7.7 dev cilium_host proto kernel src 10.0.7.7 mtu 1230
10.0.7.0/24 via 10.0.7.7 dev cilium_host proto kernel src 10.0.7.7
10.0.7.7 dev cilium_host proto kernel scope link
192.168.42.0/24 dev enp0s31f6 proto kernel scope link src 192.168.42.100 metric 100

With the target 10.0.5.100, the packet will be sent to 10.0.7.7 on the cilium_host interface. If no rule matches (as is the case with most internet traffic), the packet is sent to the default gateway (192.168.42.1 in this example).

To test which route will be used, you can use ip route get 1.1.1.1:

1.1.1.1 from 192.168.42.100 via 192.168.42.1 dev enp0s31f6 uid 0
    cache

Usually, VPN setups add another default route with higher priority to make sure traffic passes through them. This is not what we want.

Luckily, we're not restricted to a single route table! Using NetworkManager, you can setup a second route table for this connection:

nmcli con modify <connection-name> ipv4.route-table 6969

With this in place, the VPN default route is configured in a completely separate table. You can view this table using ip route show table 6969

default via 10.10.112.1 dev tun0 proto static metric 50
10.10.112.0/24 dev tun0 proto kernel scope link src 10.10.112.188 metric 50

The last step of the puzzle is to configure the table to be used. To do this, you need to tell the networking stack to use this route table for packets originating from the VPN interface IP address:

ip -4 rule add from <addr> table 6969

It is also possible to persist this using network manager but in my case the VPN interface IP Address is not static (not even the network 😞) so I have to do this dynamically. For this I built a nm-dispatch script that adds the rule whenever the interface goes up and deletes it when it goes down again. You can find this script in my infrastructure repository.

Testing this again, the VPN gateway is chosen for traffic with the VPN IP as a source 🎉

# ip route get 1.1.1.1 from 10.6.112.172
1.1.1.1 from 10.6.112.172 via 10.6.112.1 dev tun0 table 6969 uid 1000
    cache

The hard part is done! The only thing to be done on the Cilium side is to configure the CiliumEgressGatewayPolicy to use the VPN interface to find its IP address:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: basic-gateway
spec:
  destinationCIDRs:
  - 0.0.0.0/0
  egressGateway:
    interface: tun0
    nodeSelector:
      matchLabels:
        kubernetes.io/hostname: node-b
  egressGateways: []
  selectors:
  - podSelector:
      matchLabels:
        name: example-pod

And that's it! Assuming you've set up everything correclty (& on the correct nodes), the example-pod will now have all its egress traffic sent through the VPN!