IT Ghost

Expose your kubernetes sevices using gateway-api with envoy-gateway.

Keresztes Peter-Zoltan -

I wanted to see what benefits the gateway-api has in comparison with other service types on kubernetes that each has their own use case with their pros and cons. After I have research different providers I decided to use envoy-gateway as it was the most stable and complete integration. I never used envoy in the past therefore it had it's own struggles to figure out how they work. After all everything came together nicely and I decided to write this post to share my experience with others.

First I deployed the envoy proxy via helm chart. At this stage I am assuming you are familiar how the helm works therefore I will not show how to install the helm on your computer. First I have installed the CRD's and the envoy-gateway:

#helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.1.2 -n envoy-gateway-system --create-namespace

After a bit of time the helm-gateway will be up and running in the envoy-gateway-system namespace.

#kubectl get pods -n envoy-gateway-system                                       
NAME                                                              READY   STATUS    RESTARTS   AGE
envoy-gateway-655745744f-fj8jb                                    1/1     Running   0          12h

Once the CRD's and envoy-gateway is deployed we need to start configuring it. The configuration is done by creating different custom resources. First we need to create a gatewayClass. In the parametersRef I am setting up the name and type of the configuration where the gatewayClass should apply the config from to the gateway using the name common-external-config and kind EnvoyProxy. Note different gateway implementation will use different naming:

#cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: external-gateway-class
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: common-external-config
    namespace: envoy-gateway-system
EOF

Different providers use different annotations for their loadbalancers to configure the property of the loadbalancer. Since I am using VPSie platform I need to configure the annotations and the services for the envoy gateway loadbalancers. This way I also configure the horizontal autoscaling policies. This can be done using Envoy Proxy custom resource:

#cat << EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: common-external-config
  namespace: envoy-gateway-system
spec:
  logging:
    level:
      default: warn
  provider:
    kubernetes:
      envoyHpa:
        maxReplicas: 10
        metrics:
          - resource:
              name: cpu
              target:
                averageUtilization: 60
                type: Utilization
            type: Resource
        minReplicas: 2
      envoyService:
        annotations:
          service.beta.kubernetes.io/vpsie-lb-protocol: tcp
          service.beta.kubernetes.io/vpsie-loadbalancer-plan: basic
          service.beta.kubernetes.io/vpsie-private-loadbalancer: 'false'
          service.beta.kubernetes.io/vpsie-vpc-name: NY2-k8s-test
        externalTrafficPolicy: Cluster
        type: LoadBalancer
    type: Kubernetes
EOF

This way I have configured to have a number of minimum 2 replicas and do the horizonta autoscaling based on the CPU if the pod CPU load is above 60% and have the maximum number of pods not higher than 10. The annotations tells that the loadbalance type is TCP, what payment plan to be used for the loadbalancer, if the loadbalancer is a public or private, and the VPC name where the internal interface would need to listen to.

Now that I have the configuration it is time to configure the Gateway custom resource that will create the LoadBalancer type of service:

#cat << EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: external-gateway
  namespace: envoy-gateway-system
spec:
  gatewayClassName: external-gateway-class
  listeners:
    - allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              shared-gateway-access: 'true'
      name: http
      port: 80
      protocol: HTTP
EOF

This tells that the gateway will be controlled by the external-gateway-class gatewayClass. It has one listener on port 80 that uses the http protocol called http and the gateway is usable by any namespace that has the shared-gateway-access=true label. Since my nginx app is runnig on the default namespace I need to label it:

#kubectl label Namespace default shared-gateway-access=true

After a short while if all was configured correctly you can see that the gateway was approved and has a public ip address assigned to it.

#kubectl -n envoy-gateway-system get gateways
NAME               CLASS                    ADDRESS         PROGRAMMED   AGE
external-gateway   external-gateway-class   162.xx.xx.xx   True         17h

Now that from the infrastructure point of view everything is setup. I am going to deploy an nginx app and then setup a HTTPRoute type of custom resource that would expose the service to the internet.

I have the nginx app setup using the app=nginx matchLabel also a service called web that listens on port 8080:

#kubectl get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
web                           ClusterIP   10.111.105.145   <none>        8080/TCP            16h

To add the HTTPRoute CR I am running the following command:

#apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web
  namespace: default
spec:
  hostnames:
    - test.zozoo.io
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: external-gateway
      namespace: envoy-gateway-system
      sectionName: http
  rules:
    - backendRefs:
          kind: Service
          name: web
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /

In this object I am setting the domain name based on which the routing to happen, the gateway name that will be used to route this request, the namespace where the gateway was deployed to and if it has more sections then the sectionName. This only have one section for now but I am planning on adding another https section that would terminate the TLS connections and use an ssl certificate provided by the certmanager. You can check this link on how to set that up.

To test if the connection work I am going to run a curl request on the configured domain name:

#╰─ curl -IL http://test.zozoo.io
HTTP/1.1 200 OK
server: nginx/1.27.1
date: Thu, 03 Oct 2024 14:42:22 GMT
content-type: text/html
content-length: 615
last-modified: Mon, 12 Aug 2024 14:21:01 GMT
etag: "66ba1a4d-267"
accept-ranges: bytes

You can do more than just hostname based routing using gateway-api. You can do path based routing, header based routing. You can expose TCP ports and then use TCPRoute to send the traffic to a specific service based on the port that you connecting to.