Getting Started with VMware Dispatch on Kubernetes · TheHumbleLab

Getting Started with VMware Dispatch on Kubernetes

· Read in about 11 min · (2238 words) ·

VMware Dispatch - An Open Source Platform for Serverless Applications

Late last week, VMware released Dispatch, an open source Serverless framework. You can check out the blog post here I’ve been in contact with the team behind Dispatch, a really great group of folks who roll up to our Office of the CTO internally. The team is doing some really incredible stuff as part of our internal xLabs program. Dispatch I’m sure is just the start of what’s to come!

VMware Dispatch

Serverless has been getting a lot of attention lately universally. For me, I moved this blog to a Serverless configuration within Amazon Web Services last year, I also have been shouting about OpenFaaS for quite some time. I think the concept of “Serverless” is going to be one of those foundational game changers (one might argue it already is…) within our IT landscape. I don’t believe that everything will move to a Serverless model; but there is a lot to be said for stateless applications that spin up their platforms only when needed. Stateless, and lifecycle completely managed means that administrators get to focus on problems that really matter - instead of managing “pet” servers in an environment.

Function as a Service

Dispatch, in its current form, is heavily focused on the concept of “Function as a Service”; a topic I’ve covered in great deal over the past few months in posts around OpenFaaS. In fact, Dispatch leverages the OpenFaaS “engine”, with some magic sauce sprinkled on top to extend and enhance. As a refresher, function as a service is the idea of building larger “automation blocks” into functions - and then bolting them onto an API endpoint to be able to be called via a single URL.

VMware Dispatch

A practical example is this: Say you wanted to build some sort of data report that contacted vRealize Automation, pulled out information about builds, and then hit vRealize Operations to return any alert data about those specific machines. I recogize this is a bit of a contrived example - however, play along. Both of those products have rich REST API’s. You’d pull up your favorite IDE, and maybe throw together a script; using PowerCLI, Python, JavaScript, or GO, and hammer out a series of commands that would return that data to you. Then you’d run that script and call get your results. Maybe you’d commit that script to an internal repo so you could share with others on your team and have them run it. The downside to this - is that any dependencies you’ve included need to be installed on that users system. You also need to ensure access to run those reports for anyone who might be running them. It can quickly turn into an onion with a ton of layers.

With Function as a Service, you could capture those individual calls, build them into a larger function - and bind them to an API endpoint. That way when you run it - you’re returned the data you want. You can hand that URL out to anyone and have them access it. You can also setup security on that API endpoint to protect it if necessary. There’s a world options. You’re now a hero. Promotions galore. vBrownBag will want you to share your story. You’ll get a Lambo. Well, maybe not ALL of that - but it WOULD be pretty cool.

What Are We going to Accomplish?

With Dispatch being brand new, and really not even at a 1.0 point - the installation is really focused on a very specific set of usecases currently. Being the avid labber that I am; I’m always interested in finding ways to “square peg == round hole” things into my lab. The Dispatch team was a HUGE help in pointing me in the right directions to get up and running.

We’re going to stand up a single node Kubernetes cluster using Kubeadm. From there, we’ll get it all configured for our future Dispatch install. The biggest item here is going to be the configuration of the local storage provider. When I first deployed; Dispatch failed every time during the final steps. RabbitMQ requires a persistent volume store and it turns out that Kubeadm doesn’t ship with one. To use the local storage provisioner (host path) we needed to apply the manifest to enable the storage provisioner, then update our kube-controller-manager manifest to enable hostpath provisioning. We’ll cover that later.

Once our Kubernetes cluster is prepped, we’ll install and configure Helm; the Kubernetes package manager. Then we’ll install Dispatch and make our FaaS call!

Requirements

  • This guide is written for use with CentOS 7
  • External internet access for the CentOS Server
  • Docker Hub account

Let’s Get Started!

Getting Started

To make things easier, I’ve stashed my “Single Node Kubernetes Install” script on GitHub for everyones use. I set it up with the expectation that its run as root. At a high level the process the script follows is…

  • Exports all PATH’s (just to be safe…)
  • Turn swap off
  • Install IPtables, disable and stop firewalld, and then make iptables default
  • Setup systemd driver for docker consumption
  • Install and configure docker
  • Install/configure Kubernetes REPO
  • Disable selinux
  • Install Kubelet, Kubeadm, and Kubectl
  • Enable and start Kubelet
  • Use Kubeadm to initialize a pod
  • Copy over all the admin files so kubectl will work
  • Add “Flannel” networking driver
  • Untaint the node so it will allow containers to be scheduled on it
  • Apply RBAC permissions for Kubernetes to allow Dispatch to install

This script can be dropped into vRealize Automation as a software component, or you can just run it on a CentOS 7 machine directly. When you run it; it takes a little bit to fire off - so you’ll likely see something similar to below as you wait.

Kubeadm Init

Once this script finishes, you should have a functional single node Kubernetes cluster. You should see something like the below

Kubeadm Init Complete

There’s a couple of things we’ll want to do that I’ve left as manual for now. First, let’s make sure that everything is running correctly, go ahead and run a kubectl cluster-info to make sure everything is solid. Assuming you get a return like the below…

Cluster Info

You’re solid!

Pro-Tip

Frequently in this post you’ll see me run the full kubectl –kubeconfig=/etc/kubernetes/admin.conf whatever command. I do this to illustrate the importance of leveraging the administrative conf file. If you want to make your commands shorter, you can do this in linux by running

EXPORT KUBECONFIG=~/etc/kubernetes/admin.conf

Which will then have that applied as an environment variable for your session. The script I provided to install everything will place this in your root, and should pick it up when you log in - but only on a fresh login. Exporting the value is easy :)

Enabling “Host Path” Provisioner

First, we need to enable a default “storageclass”. To do this, we’re going to use one of the many addons Kubernetes provides, specifically the one for local storage

kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/storage-class/local/default.yaml

From there we need to edit our controller’ yaml file to reflect enabling this storage, then redeploy it

vi /etc/kubernetes/manifests/kube-controller-manager.yaml

This should present you with a rather long yaml file. The spot we specifically care about it the below “spec:” snippet

spec:
  containers:
  - command:
    - kube-controller-manager
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --controllers=*,bootstrapsigner,tokencleaner
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --address=127.0.0.1
    - --leader-elect=true
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16
    - --node-cidr-mask-size=24

we need to update this to include the --enable-hostpath-provisioner setting. We’ll simply add this under the kube-controller-manager section as follows

spec:
  containers:
  - command:
    - kube-controller-manager
    - --enable-hostpath-provisioner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --controllers=*,bootstrapsigner,tokencleaner
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --address=127.0.0.1
    - --leader-elect=true
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.244.0.0/16
    - --node-cidr-mask-size=24

Once that’s done, save your settings with a :wq! and run a

kubectl --kubeconfig=/etc/kubernetes/admin.conf apply -f /etc/kubernetes/manifests/kube-controller-manager.yaml 

Which will deploy the updated manager. We should be in business! To confirm we can run a

kubectl --kubeconfig=/etc/kubernetes/admin.conf get sc 

And confirm the storage class has loaded.

StorageClass

Our storageclass is enabled, and we can move on!

Installing Helm

Helm is a package manager for Kubernetes. It streamlines installation of Kubernetes deployments. Dispatch leverages Helm to do the heavy lifting parts of the installation; so we’ll need to install that on our cluster.

Helm has a pretty solid quickstart guide that covers the majority of the steps. Since we’re using Linux - the install is especially easy, since there’s a script version available.

(from Helm’s guide…)

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

Once that completes, run a helm version to ensure you get a successful return of both the server and client. As long as you do; you’re good to move forward!

##Installing Dispatch

The team deploying Dispatch at VMware has done a great job at the initial documentation for the platform; however, being that my lab is setup a specific way - and I’m doing a single node Kubernetes deployment instead of Minikube - there was a few changes I needed to make.

I went ahead and created 2 A Records in my DNS for my lab. My servers IP address is 10.0.0.23. The 2 records I created were…

  • dispatch.humblelab.com
  • api.humblelab.com

Setting them to the IP address I mentioned above.

We’ll pull down the necessary binary for the install next;

curl -OL https://github.com/vmware/dispatch/releases/download/v0.1.2/dispatch-linux
chmod +x dispatch-linux

Eventually, we’ll be using GitHub’s OAuth app integration to do authentication, so its a good idea to follow the guide the Dispatch team has provided here to get that setup, so you can provide your oath information in the yaml for the next step.

We’re going to slightly change the yaml that the Dispatch team provides. We’re updating the hostnames to match those records we created above, and we are going to feed in our OpenFaaS repository details. Since we’re using Docker Hub, I’ve put in some sample values below that should help you figure out what to put in. Specifically host should be your Docker Hub username, and the rest is your standard details.

cat << EOF > config.yaml
apiGateway:
  hostname: api.humblelab.com
dispatch:
  hostname: dispatch.humblelab.com
  image:
    host: vmware
    tag: v0.1.2
  debug: true
  openfaasRepository:
    host: codydearkland
    username: codydearkland
    email: codydearkland@humblelab.com
    password: VMware123!
  oauth2Proxy:
    clientID: <oauth2-client-id>
    clientSecret: <oauth2-client-secret>
EOF

And with that set, we can do our install. Run the command below to get rolling:

./dispatch-linux install --file config.yaml --service all --debug

When this completes, if we do a cat ~/.dispatch/config.json we’ll see our configuration file. We need to make a couple of quick changes to it in order to move forward. Run a vi against it, and change insecure and skipauth to True. Things won’t always be this way; but we are where we are at :)

Also, as mentioned in the Dispatch documentation - note the port; this is going to be used at a later date! Update the OAuth application in the GitHub developer section to reflect that port, as mentioned in the Dispatch documentation. This isn’t critical for our process - but you’ll want it done eventually.

Now, we’ll clone the Dispatch Repo so we can get to our example code.

yum install -y git
git clone https://github.com/vmware/dispatch.git

Upon completion, we will change into the examples directory and create our images from the seed file

cd /dispatch/examples/
./dispatch-linux create --file seed.yaml
./dispatch-linux get images

This should return something similar to the below, and we’re in business! Continuing to follow the Dispatch guide on GitHub, if we run

./dispatch-linux exec hello-py --input '{"name": "Jon", "place": "Winterfell"}' --wait

We will be presented with a nice JSON response of

{
    "blocking": true,
    "executedTime": 1515624222,
    "finishedTime": 1515624222,
    "functionId": "5138d918-e78f-41d6-aece-769addd3eed7",
    "functionName": "hello-py",
    "input": {
        "name": "Jon",
        "place": "Winterfell"
    },
    "logs": null,
    "name": "b5b3c1f5-fa8a-4b38-b7d1-475c44b76114",
    "output": {
        "myField": "Hello, Jon from Winterfell"
    },
    "reason": null,
    "secrets": [],
    "status": "READY"
}

This proves that our function is actually working as intended! Whats effectively happening is the python code is taking in a JSON input, running a simple python script which returns those back in “myField”, as a string sentence. This is spinning up a container on our kubernetes cluster for the sole purpose of running this script. Imagine this being orchestrated for 100 different functions; all interacting with parts of your environment. Glorious.

Now, this is just executing locally - but what about the API endpoint goodness I talked about before?

If we do a

./dispatch-linux create api --https-only --method POST --path /hello post-hello hello-py

We’ll see that an actual API endpoint is created for us, we’ll need to get Kong (a fantastic container based API proxy, which is used with Dispatch) to return our nodeport information. Run a…

kubectl get service api-gateway-kongproxy -n kong

which should return to you the ports being used. From here, we can call our API url we setup earlier (https://api.humblelab.com for me) and actually make the same API call.

curl -k "https://api.humblelab.com:32383/hello" -H "Content-Type: application/json" -d '{"name": "Jon", "place": "winterfell"}'

Which returns…

{"myField":"Hello, Jon from winterfell"}

Excellent! Our mission is accomplished. Your life is enriched.

Conclusion

As you can see, it’s incredibly easy to get started using Dispatch. These are some very basic examples, but the possibilities are endless when we start digging into what could come next. I’m going to be working on converting my OpenFaaS functions to work with Dispatch directly - which should be a really easy task to put together. As these are completed I’ll be sharing them here on the blog.

I hope you enjoyed digging into Dispatch; I look forward to sharing more soon!