We did k8s on prem: both Red Hat Openshift 3 and Red Hat Openshift 4. We were able to make Openshift 3 work on VMware with Terraform, the vSphere provider, a relatively simple Ansible inventory script to read from Terraform’s online state (stored in the cloud), and Red Hat’s official Ansible Openshift playbook.

I’m not paid by Red Hat and I’ll be frank: installing Kubernetes with Ansible was awful. But then Openshift 4 came out and 99% of the Ansible went away. I’m pretty dumb and I was able to get an Openshift 4 cluster up and running on VMware, on my own, in two weeks of exploring. I just had to stand up an auxiliary webserver for masters and worker nodes to grab their ignition configs.

You can do it and it was working pretty well for us. Everything in Red Hat Openshift comes downstream from OKD if you want to run completely open source. What I’m not sure of is the security bits. Red Hat’s security implementation is actually really good. OAuth and LDAP are built in. Pod security policies and security context constraints are well thought-out.

A lot of companies rush into the cloud and don’t think through the fact that they’re effectively introducing a new datacenter into their network. I’m all cloud, but I’m also somewhat fascinated by integrating cloud networks and on-prem networks correctly. That’s hard to do! Running workloads on prem might be a viable solution for quite a few companies even for a couple years, while they work out larger migration plans to cloud computing.