Skip to main content
Version: 0.9.1

Getting started enterprise

Creating your first CAPD Cluster

If you've followed the Upgrade steps in the Installation guide you should have:

  1. Weave GitOps Enterprise installed
  2. A CAPI provider installed (With support for ClusterResourceSets enabled).

Next up we'll add a template and use it to create a cluster.

Directory structure

Let's setup a directory structure to manage our clusters

mkdir -p clusters/bases \
clusters/management/capi/templates \
clusters/management/capi/bootstrap \
clusters/management/capi/profiles

Now we should have:

.
└── clusters
├── bases
└── management
└── capi
├── bootstrap
├── profiles
└── templates

This assumes that we've configured flux to reconcile everything in clusters/management into our management cluster.

To keep things organized we've created some subpaths for the different resources:

  • bases for any common resources between clusters like RBAC and policy.
  • templates for CAPITemplates
  • bootstrap for ClusterBootstrapConfig, ClusterResourceSet and the ConfigMap they reference
  • profiles for the HelmRepository of the profiles for the newly created clusters

Lets grab some sample resources to create our first cluster!

Add common RBAC to the repo

When a cluster is provisioned, by default it will reconcile all the manifests in ./clusters/<cluster-namespace>/<cluster-name> and ./clusters/bases.

To display Applications and Sources in the UI we need to give the logged in user permissions to inspect the new cluster.

Adding common rbac rules to ./clusters/bases/rbac is an easy way to configure this!

clusters/bases/rbac/wego-admin.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: wego-test-user-read-resources-cr
subjects:
- kind: User
name: wego-admin
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: wego-admin-cluster-role
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: wego-admin-cluster-role
rules:
- apiGroups: [""]
resources: ["secrets", "pods"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list"]
- apiGroups: ["kustomize.toolkit.fluxcd.io"]
resources: ["kustomizations"]
verbs: ["get", "list", "patch"]
- apiGroups: ["helm.toolkit.fluxcd.io"]
resources: ["helmreleases"]
verbs: ["get", "list", "patch"]
- apiGroups: ["source.toolkit.fluxcd.io"]
resources: ["buckets", "helmcharts", "gitrepositories", "helmrepositories"]
verbs: ["get", "list", "patch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "watch", "list"]
- apiGroups: ["pac.weave.works"]
resources: ["policies"]
verbs: ["get", "list"]

Add a template

See CAPI Templates page for more details on this topic. Once we load a template we can use it in the UI to create clusters!

Download the template below to your config repository path, then commit and push to your git origin.

clusters/management/capi/templates/capd-template.yaml
apiVersion: capi.weave.works/v1alpha1
kind: CAPITemplate
metadata:
name: cluster-template-development
namespace: default
spec:
description: A simple CAPD template
params:
- name: CLUSTER_NAME
required: true
description: This is used for the cluster naming.
- name: NAMESPACE
description: Namespace to create the cluster in
- name: KUBERNETES_VERSION
description: Kubernetes version to use for the cluster
options: ["1.19.11", "1.21.1", "1.22.0", "1.23.3"]
- name: CONTROL_PLANE_MACHINE_COUNT
description: Number of control planes
options: ["1", "2", "3"]
- name: WORKER_MACHINE_COUNT
description: Number of control planes
resourcetemplates:
- apiVersion: gitops.weave.works/v1alpha1
kind: GitopsCluster
metadata:
name: "${CLUSTER_NAME}"
namespace: "${NAMESPACE}"
labels:
weave.works/capi: bootstrap
spec:
capiClusterRef:
name: "${CLUSTER_NAME}"
- apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: "${CLUSTER_NAME}"
namespace: "${NAMESPACE}"
labels:
cni: calico
spec:
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
serviceDomain: cluster.local
services:
cidrBlocks:
- 10.128.0.0/12
controlPlaneRef:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
name: "${CLUSTER_NAME}-control-plane"
namespace: "${NAMESPACE}"
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerCluster
name: "${CLUSTER_NAME}"
namespace: "${NAMESPACE}"
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerCluster
metadata:
name: "${CLUSTER_NAME}"
namespace: "${NAMESPACE}"
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
metadata:
name: "${CLUSTER_NAME}-control-plane"
namespace: "${NAMESPACE}"
spec:
template:
spec:
extraMounts:
- containerPath: /var/run/docker.sock
hostPath: /var/run/docker.sock
- apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
name: "${CLUSTER_NAME}-control-plane"
namespace: "${NAMESPACE}"
spec:
kubeadmConfigSpec:
clusterConfiguration:
apiServer:
certSANs:
- localhost
- 127.0.0.1
- 0.0.0.0
controllerManager:
extraArgs:
enable-hostpath-provisioner: "true"
initConfiguration:
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
kubeletExtraArgs:
cgroup-driver: cgroupfs
eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
joinConfiguration:
nodeRegistration:
criSocket: /var/run/containerd/containerd.sock
kubeletExtraArgs:
cgroup-driver: cgroupfs
eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
machineTemplate:
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
name: "${CLUSTER_NAME}-control-plane"
namespace: "${NAMESPACE}"
replicas: "${CONTROL_PLANE_MACHINE_COUNT}"
version: "${KUBERNETES_VERSION}"
- apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
metadata:
name: "${CLUSTER_NAME}-md-0"
namespace: "${NAMESPACE}"
spec:
template:
spec: {}
- apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: "${CLUSTER_NAME}-md-0"
namespace: "${NAMESPACE}"
spec:
template:
spec:
joinConfiguration:
nodeRegistration:
kubeletExtraArgs:
cgroup-driver: cgroupfs
eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
- apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
name: "${CLUSTER_NAME}-md-0"
namespace: "${NAMESPACE}"
spec:
clusterName: "${CLUSTER_NAME}"
replicas: "${WORKER_MACHINE_COUNT}"
selector:
matchLabels: null
template:
spec:
bootstrap:
configRef:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: "${CLUSTER_NAME}-md-0"
namespace: "${NAMESPACE}"
clusterName: "${CLUSTER_NAME}"
infrastructureRef:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DockerMachineTemplate
name: "${CLUSTER_NAME}-md-0"
namespace: "${NAMESPACE}"
version: "${KUBERNETES_VERSION}"

Automatically install a CNI with ClusterResourceSets

We can use ClusterResourceSets to automatically install CNI's on a new cluster, here we use calico as an example.

Add a CRS to install a CNI

Create a calico configmap and a CRS as follows:

clusters/management/capi/boostrap/calico-crs.yaml
apiVersion: addons.cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
name: calico-crs
namespace: default
spec:
clusterSelector:
matchLabels:
cni: calico
resources:
- kind: ConfigMap
name: calico-crs-configmap

The full calico-crs-configmap.yaml is a bit large to display inline here but make sure to download it to clusters/management/capi/bootstrap/calico-crs-configmap.yaml too, manually or with the above curl command.

Profiles and clusters

WGE can automatically install profiles onto new clusters

Add a helmrepo

A profile is an enhanced helm chart. When publishing profiles to helm repositories make sure to include the weave.works/profile in Chart.yaml. These annotated profiles will appear in WGE

annotations:
weave.works/profile: nginx-profile

Download the profile repository below to your config repository path then commit and push. Make sure to update the url to point to a Helm repository containing your profiles.

clusters/management/capi/profiles/profile-repo.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
creationTimestamp: null
name: weaveworks-charts
namespace: flux-system
spec:
interval: 1m
url: https://my-org.github.io/profiles
status: {}

Add a cluster bootstrap config

Create a cluster bootstrap config as follows:

 kubectl create secret generic my-pat --from-literal GITHUB_TOKEN=$GITHUB_TOKEN

Download the config with

Then update the GITOPS_REPO variable to point to your cluster

clusters/management/capi/boostrap/capi-gitops-cluster-bootstrap-config.yaml
apiVersion: capi.weave.works/v1alpha1
kind: ClusterBootstrapConfig
metadata:
name: capi-gitops
namespace: default
spec:
clusterSelector:
matchLabels:
weave.works/capi: bootstrap
jobTemplate:
generateName: "run-gitops-{{ .ObjectMeta.Name }}"
spec:
containers:
- image: ghcr.io/fluxcd/flux-cli:v0.29.5
name: flux-bootstrap
resources: {}
volumeMounts:
- name: kubeconfig
mountPath: "/etc/gitops"
readOnly: true
args:
[
"bootstrap",
"github",
"--kubeconfig=/etc/gitops/value",
"--owner=$GITHUB_USER",
"--repository=fleet-infra",
"--path=./clusters/{{ .ObjectMeta.Namespace }}/{{ .ObjectMeta.Name }}",
]
envFrom:
- secretRef:
name: my-pat
restartPolicy: Never
volumes:
- name: kubeconfig
secret:
secretName: "{{ .ObjectMeta.Name }}-kubeconfig"

Add Monitoring Dashboards to your cluster

In order to add dashboards to your cluster, you'll need to use metadata annotations following the below pattern.

apiVersion: gitops.weave.works/v1alpha1
kind: GitopsCluster
metadata:
annotations:
metadata.weave.works/dashboard.grafana: https://grafana.com/
metadata.weave.works/dashboard.prometheus: https://prometheus.io/

Test

You should now be able to create a new cluster from your template and install profiles onto it with a single Pull Request via the WGE UI!