Advanced Deployments¶

Use Cases¶

When you have a new containerized image to deploy you can indeed just patch the deployment and do a RollingUpdate to force it out. The Kubernetes documentation has examples that show how to do it.

However, you might want to try it out on a subset of users. Say, give it to 10% of your user population and see if there are problems. This is typically described as a "canary deployment." You might want to increase the percentage until most of the population is using it, and then pass over control completely to the new group.

Or you may want to set up the deployment, and swap it over knowing you can swap it back quickly. That's a "red-blue deployment."

The LiteSpeed Ingress Controller supports both types of advanced deployment methodologies. And it does it with standard Kubernetes objects, no extensions or sidecars are required. You can even use regular expressions to limit the number of specifications you have to create and change.

How To¶

We recommend that you set up your deployment in advance and give it a try, but you can patch it in for immediate deployment if you prefer.

The LiteSpeed Ingress Controller advanced deployment technique relies on the fact that you can configure an ingress spec with two rules for the same domain and path.

To set up an advanced deployment, you would follow these steps:

Add two rules to your ingress for the same domain and path, giving them each a unique service: name.
Optionally port, or any other ingress values within the definition, can vary between rules.
Set up annotations to describe which rule is to be used, based on service: name and weight. (A weight is a percentage of traffic that goes to a given service.)

Examples¶

Routing domain.com to service1 or service2

Say you have an ingress that you route by the domain domain.com, and you wish to split traffic equally between service1 or service2. You would create an ingress definition like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: route
  annotations: 
    kubernetes.io/ingress.class: litespeedtech.com/lslbd
    litespeedtech.com/host.service.weight1: domain.com/service1/50/service2/50
spec:
  rules:
    - host: "domain.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service1
                port:
                  number: 8080

    - host: "domain.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service2
                port:
                  number: 8080

Assuming that service1 and service2 are properly defined with deployments, this would result in all traffic being routed equally amongst all of the deployed pods, even without the annotation. But the annotation is where you get the control.

The annotation title is litespeedtech.com/host.service.weight. In the example above it is litespeedtech.com/host.service.weight1 with the 1 suffix used to keep it unique, but any suffix text could be used so long as it used the litespeedtech.com/host.service.weight prefix and it's unique in the ingress.

The important values are separated by slashes.

domain.com: The domain to apply the annotation to.
service1: The first service to apply a weight to.
50: This service will get 50% of the traffic.
service2: The second service to apply a weight to.
50: This service will get 50% of the traffic as well.

The two weights must add up to 100. You can even skip the last slash and second weight and it will accomplish the same thing: domain.com/service1/50/service2.

Red deployment

Say you now want to phase out service1 entirely and have all traffic routed to service2. You can do this with a single command:

kubectl annotate ingress route litespeedtech.com/host.service.weight1=domain.com/service2 --overwrite

The command is simpler since you've only specified a single service, LiteSpeed will assume it gets 100% of traffic.

The --overwrite qualifier is necessary to overwrite the existing annotation.

You can do this with traffic running and LiteSpeed will finish any running traffic to service1, but all new traffic will be routed to service2.

Blue deployment

Change your mind and want it all back to service1? Use this command:

kubectl annotate ingress route litespeedtech.com/host.service.weight1=domain.com/service1 --overwrite

Canary deployment

Now you want to canary deploy only 10% of traffic to service2? Try this:

kubectl annotate ingress route litespeedtech.com/host.service.weight1=domain.com/service1/90/service2 --overwrite

Verifying Annotations¶

Kubernetes does no error checking with annotations, other than simple command line syntax settings. To verify that the annotation has been applied to the LiteSpeed, you can check the LiteSpeed's pod log looking for entries that begin with Using weighted annotation. For example in these testing servers:

I0518 19:57:00.515560       1 controller.go:1294] Using weighted annotation.  Domain: route-1-only, service: route-1-only, weight: 100
I0518 19:57:00.515744       1 controller.go:1294] Using weighted annotation.  Domain: route-1-only, service: route-1-2-only, weight: 0
I0518 19:57:00.515892       1 controller.go:1294] Using weighted annotation.  Domain: route-2-only, service: route-2-2-only, weight: 0
I0518 19:57:00.516014       1 controller.go:1294] Using weighted annotation.  Domain: route-2-only, service: route-2-only, weight: 100

Or you can look for annotation errors specifically using the prefix: Routing Annotation Error:

I0518 19:57:00.515295       1 controller.go:1257] Routing Annotation Error: First weight 100 + second weight 100 must total 100 in litespeedtech.com/host.service.weight4:route-1-invalid/route-1-invalid/100/route-2-invalid/100

You can also look in the LiteSpeed WebAdmin Console, real-time stats. Its configuration is described here.

Regular Expressions in Annotations¶

If you wish, you can use a regular expression in an annotation, which will allow you to refer to a number of domains or services in a single annotation.

There is a different annotation title prefix for annoations which use regular expressions: litespeedtech.com/hostx.servicex.weight - the difference being an x after host and service to help you remember where you can use regular expressions. Just like with non-regular expression annotations, the title is a prefix and you will need to add add a suffix to specify multiples of these. You can have regular expression annotations and non-regular expression annovations in the same ingress.

Example of Regular Expression Annotation¶

This simple example is a contrived case to showcase the feature, but it may give you ideas of how to use it in your environment. If you have 2 domains: routex-domain-1 with two services routex-svc-1-1, routex-svc-1-2; and routex-domain-2 with two services routex-svc-2-1, routex-svc-2-2. The services are meant to represent two different versions of software for each domain. In the base example below all traffic for domain routex-domain-1 is routed to routex-svc-1-1 and domain routex-domain-2 is routed to routex-svc-2-1.

This is because there is a single annotation which specifies the domain name with wildcards and the service with wildcards:

When selecting domain it matches domains routex-domain-1 and routex-domain-2 with routex-domain-[0-9]. The wildcard is the [0-9] which is a regular expression wildcard which will accept any single numeric digit.
When selecting service is matches routex-domain-1 with service routex-svc-1-1 and routex-domain-2 with service routex-svc-2-1. It uses the wildcard specification routex-svc-[0-9]-1 which again accepts a single numeric digit.
With no weight specified, all traffic goes to the service (default 100)

If you're unfamilar with regular expressions, they are described here and there are many references and tutorials on the internet.

This is the .yaml file which would be used for the ingress described in the example above.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: route
  annotations: 
    kubernetes.io/ingress.class: litespeedtech.com/lslbd
    litespeedtech.com/hostx.servicex.weight1: 'routex-domain-[0-9]/routex-svc-[0-9]-1'
spec:
  rules:
    - host: "routex-domain-1"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: routex-svc-1-1
                port:
                  number: 8080
    - host: "routex-domain-1"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: routex-svc-1-2
                port:
                  number: 8080
    - host: "routex-domain-2"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: routex-svc-2-1
                port:
                  number: 8080
    - host: "routex-domain-2"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: routex-svc-2-2
                port:
                  number: 8080

When you decide to change to a different version, first make sure that it has services named routex-svc-1-2 and routex-svc-2-2. They should have the appropriate deployment definitions that contain the images for your new version. You may want to verify the services using a non-production ingress to make sure it is correct.

If you'd like to perform a canary deployment and wanted to expose only 10% of the traffic to the new version, you would specify:

kubectl annotate ingress route 'litespeedtech.com/hostx.servicex.weight1=routex-domain-[0-9]/routex-svc-[0-9]-1/90/routex-svc-[0-9]-2' --overwrite

When you're ready to deploy it completely, specify:

kubectl annotate ingress route 'litespeedtech.com/hostx.servicex.weight1=routex-domain-[0-9]/routex-svc-[0-9]-2' --overwrite

Last update: October 20, 2023