Rancher 2.0 Setup (with Kubernetes engine)

Introduction

Nomenclature (Kubernetes like)

Rancher 2.0 Guide to provisioning a Kubernetes cluster which uses the Kubernetes container-orchestration system to

Name Kubernetes concepts
Container Pod (Simplest Kubernetes object representing a set of containers on the cluster)
Services Workload (Units of work that are running on the cluster, these can be pods or deployments)
Load Balancer Ingress
Stack Namespace (A virtual cluster)
Environment Project (Administration)/Cluster (Compute machines that run containerized applications)
Host Node (Physical or virtual machines making up the cluster)
Catalog Helm

A. Cluster Description

Rancher Deployment Diagram

B. Lets Encrypt on Ubuntu 16.04

sudo netstat -peanut | grep ":80" 

sudo certbot certonly --standalone --dry-run \
   --cert-name loup.ece.ucsb.edu -d loup.ece.ucsb.edu

C. Master Rancher 2.0

Install/Startup Rancher: https://rancher.com/docs/rancher/v2.x/en/installation/single-node/ - Rancher etcd data persisted at /var/lib/rancher - Since port 80 is occupied by rancher/rancher, a rancher/rancher-agent cannot be run on this node.

docker run -d --restart=unless-stopped \
  -p 8080:80 -p 8443:443 \
  -v /var/log/rancher/auditlog:/var/log/auditlog \
  -v /host/rancher:/var/lib/rancher \
  -e AUDIT_LEVEL=1 \
  rancher/rancher:stable

Rancher main Container

Migration CLI
migration-tools export --url <RANCHER_URL> --access-key <RANCHER_ACCESS_KEY> \
 --secret-key <RANCHER_SECRET_KEY> --export-dir <EXPORT_DIR>

./migration-tools parse --docker-file compose/docker-compose.yml \
 --rancher-file compose/rancher-compose.yml

D. Setup Cluster RKE/custom-nodes

Port requirements

Open up ports based on the CNI provider requirements - Use Canal as the provider in this case

# API/UI Clients
sudo ufw allow 22,80,443/tcp
# Etcd Plane Nodes
sudo ufw allow 2379,2380,9099,6443/tcp && sudo ufw allow 8472/udp
# Control Plane Nodes
sudo ufw allow 2379,2380,10250,6443,9099,10254/tcp && sudo ufw allow 8472/udp
# Worker Plane Nodes
sudo ufw allow 6443,9099,10254/tcp && sudo ufw allow 8472/udp
# Workload
sudo ufw allow 30000:32767/tcp && sudo ufw allow 30000:32767/udp
# Others 
sudo ufw allow  2376/tcp

Ubuntu ufw status

Create cluster
sudo docker run -d --privileged --restart=unless-stopped --net=host \
 -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run \
  rancher/rancher-agent:v2.1.6 --server https://loup.ece.ucsb.edu:8443 \
  --token 7z2ncgjj4482m48fpsj7xjmc8lc9n6bsxh7qcjrsr6rcxrzhzl6prz \
  --ca-checksum d522680b13d7aabe4dc57bb2776e28759852c336d0cf0e0f9fed5d3fb7b495e8 \
  --etcd --controlplane --worker
Create a namespace bqdev within this cluster

Bisque Development environment where workloads are deployed and tested

E. Setup Volume

# Create the path on host system
sudo mkdir /opt/bisque/ -p && \
sudo mkdir /opt/bisque/data -p && \
sudo mkdir /opt/bisque/local/workdir -p

# Allow other users to edit this
sudo chown -R nobody:nogroup /opt/bisque/

# Access from specific machines
sudo ufw allow from 192.168.1.129 to any port nfs
sudo ufw allow from 192.168.1.133 to any port nfs

# Specific ports in case above doesnt work
sudo ufw allow 32768:65535/tcp && sudo ufw allow 32768:65535/udp
sudo ufw allow 2049/tcp && sudo ufw allow 2049/udp
sudo ufw allow 111/tcp && sudo ufw allow 111/udp

/opt/bisque     192.168.1.129(rw,sync,no_root_squash,no_subtree_check)
/opt/bisque     192.168.1.133(rw,sync,no_root_squash,no_subtree_check)

sudo systemctl restart nfs-kernel-server

sudo mount 192.168.1.123:/opt/bisque/ /run/bisque/

AND https://www.claudiokuenzler.com/blog/786/rancher-2.0-create-persistent-volume-from-nfs-share

Rancher NFS persistent volume addition

Rancher workload volumes

F. Setup Workload (on the cluster)

Bisque Test environment where workloads are deployed with open NodePort https://rancher.com/managing-kubernetes-workloads-with-rancher-2-0/

Test workload configuration

Workload ports

Finally we can see the overall state of pods in the workload within the clusters Workload pods

Environment Configuration
      BISQUE_USER= bisque
      BISQUE_BISQUE_ADMIN_EMAIL= admin@loup.ece.ucsb.edu
      BISQUE_BISQUE_BLOB_SERVICE_STORES= blobs,local
      BISQUE_BISQUE_STORES_BLOBS_MOUNTURL= file://$$datadir/blobdir/$$user/
      BISQUE_BISQUE_STORES_BLOBS_TOP= file://$$datadir/blobdir/
      BISQUE_BISQUE_STORES_LOCAL_MOUNTURL= file://$$datadir/imagedir/$$user/
      BISQUE_BISQUE_STORES_LOCAL_READONLY= true
      BISQUE_BISQUE_STORES_LOCAL_TOP= file://$$datadir/imagedir/
      BISQUE_DOCKER_DOCKER_HUB= biodev.ece.ucsb.edu:5000
      BISQUE_SECRET= bq123
      BISQUE_UID= 12027
      BISQUE_RUNTIME_STAGING_BASE= /run/bisque/data/staging
      BQ__BISQUE__IMAGE_SERVICE__WORK_DIR= /run/bisque/local/workdir
      BQ__BISQUE__PATHS__DATA= /run/bisque/data
      MAIL_SERVER= dough.ece.ucsb.edu
      BISQUE_DBURL=postgresql://postgres:postgres@10.42.0.15:5432/postgres

      DEBIAN_FRONTEND=noninteractive
      IMGCNV=imgcnv_ubuntu16_2.4.3

We should see the overview of workloads deployed as below Workload Dashboard

G. Load Balancing (using L7 Ingress)

H. Monitoring/Debugging

# Fetch namespaces
kubectl get pods --all-namespaces
kubectl get pods -n bqdev 

# Fetch logs on a pod/container
kubectl logs postgres-564d9f79d5-z2sxl  -n bqdev

I. Uninstall Rancher

# Master: for the rancher server container
docker stop $(docker ps -a -q --filter ancestor=rancher/rancher:stable --format="{{.ID}}")
# Workers: for all k8s containers 
docker stop $(docker ps -f name=k* --format="{{.ID}}")

docker rm -f $(docker ps -a -f name=k* --format="{{.ID}}")
docker rmi -f $(docker images -q "rancher/*")
docker volume rm $(docker volume ls -q)

# Unmount directories
for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do sudo umount $mount; done

# Clean the directories
sudo rm -rf /etc/ceph \
       /etc/cni \
       /etc/kubernetes \
       /opt/cni \
       /opt/rke \
       /run/secrets/kubernetes.io \
       /run/calico \
       /run/flannel \
       /var/lib/calico \
       /var/lib/etcd \
       /var/lib/cni \
       /var/lib/kubelet \
       /var/lib/rancher/rke/log \
       /var/log/containers \
       /var/log/pods \
       /var/run/calico
# Mounted host directories
sudo rm -rf /host/rancher/
sudo rm -rf /var/log/rancher/auditlog

ip address show
ip link delete <interface_name>

Additional References

==TODO==

1.) Mail server setup

https://www.linuxbabe.com/mail-server/ubuntu-16-04-iredmail-server-installation

2.) Migration from Rancher 1.x to 2.x

3.) Reference on Ingress Controllers

4.) PostgreSQL server