GitOps

Introduction

Since we moved code to version control, it seems useful to also move configuration code to it as well. Since most things are declarative now and are described by files, storing configuration in VCS is much more appealing. The advantages are

• clear paper trail
• centralized location
• version control

By storing configuration in a repository, pull-based deployment becomes possible as well, which is really cool! Bare with me for a minute.

K8 and Pull-based Deployments

Usually in your pipelines you might do something along these lines

lint -> format -> unit tests -> tests -> build-image -> deploy

Where deploying involves running kubectl apply commands manually. One obvious downside is that there is potential for drift as state is changing inside the pipelines and this state is not exactly tracked anywhere. There is also the fact that running kubectl commands in pipelines is not exactly clean.

But with pull-based deployment we change the last step to actually create a commit against a separate repository which hosts the manifest files. This is generally as simple as just changing the image tag. With this there are no kubectl commands and the state is stored somewhere visible.

Then you could have something like ArgoCD or flux observe the repository and then reconcile differences to get the current deployment to matched the newly updated manifest files.

Directory Structure

The directory structure for the manifests repository generally looks like this (for each project, you can probably store multiple ones here)

base/
  service.yml
  deployment.yml
  kustomization.yml
overlays/
  dev/
    patches.yml
    .env
    kustomization.yml
  prod/
    patches.yml
    .env
    kustomization.yml

Pretty simple, base/kustomization.yml contains the shared components. And the kustomization file in the ~overlays directory than call refer back to the base using something like

resources:
  - ../../base

Along with any other required stuff. In the pipelines you might directly make commits changing the image tag of files inside dev. And to promote you can have a workflow which copies required files from dev to prod.

Making Commits

You will probably need an access token and then use ssh in the pipelines to clone the project and make commits. Example of a pipeline implementation: GitOps Pipelines Example.

Secrets

Since we are commiting manifests, which includes kubernetes Secrets and therefore passwords and such. Luckily, I kind of know that these should not be commited as is. So there are a couple of ways to handle secrets:

1. outside of cluster: using something like KSOPS and SOPs so they are encrypted on the repo but gets decrypted before it gets sent to the cluster
2. inside of cluser: the secret gets decoded or fetched after it arrives at the cluster

ArgoCD recommends secret decoding inside of the cluster, so it doesn't have to handle secrets¹. In that case, there are a couple of solutions: External Secret Operator (ESO) or CSI allows you to store secrets elsewhere, like using Hashicorp Vault or its open source cousin OpenBao, and then have the cluster fetch it, or Bitnami SealedSecrets, which uses GPG keys to encrypt the secrets (which you commit) and then send to the cluster to have them decrypted.

References

https://www.digitalocean.com/community/tutorials/how-to-deploy-to-kubernetes-using-argo-cd-and-gitops