Scalable multi-environment directory structure for a kubernetes project using kustomize, helm and ArgoCD

Personally I find ArgoCD works perfectly fine for kustomize and helm charts. Each tool is used in the following manner:

1. argoCD for configuring kubernetes deployments, keeping Applications in sync with the state of the git repo (GitOps)
2. helm charts we use it to install typical third party charts such as argoCD, external-secrets, cert-manager , istiod, kyverno, etc.
3. kustomize is great a customizing the different values in k8s config files between different environments.

Here is the Github repo with example that you can use to structure your project or at least as a reference.

From now own we will consider {env} as dev or prod . you can call your environments whatever you want but those two are typical bare minimum environments in multiple companies.

Here is a top level of how I currently organize my projects:

```
├─ infra           # Infrastructure as Code 
└─ kubernetes      # Code for configuring GKE and custom k8s resources
    ├─ kustomize
    |   ├─ ApplicationSet.{env}.yaml # The base AppSet per environment
    |   ├─ base       # base k8s resources for all environments
    |   └─ overlays    # kustomize overlays for customizing different environments
    |
    └─ helm
       ├─ charts  # Helm charts downloaded by using `helm pull {REPO}/{NAME} --untar` 
       └─ values  # values for customizing helm charts per environment
```

helm subdirectory

Lets start with the helm/ subdirectory at available here. This subdir just contains dummy files. The important thing to notice here is that helm/charts/ is where you download and untar the charts from third parties such as the argocd helm chart, or the cert-manager helm chart. From this chart you will copy and paste the values.yaml file into helm/values/{chart}/{env}.values.yaml, where env will equal to a short prefix of your environment.

Including the helm charts in your repo has the following advantages:

• Allows you to easily compare your custom helm/values/{chart}/{env}.values.yaml file with the original untouched values.yaml file at helm/charts/{chart} . This is extremely useful when you want to see what was changed for your environment.
• Allows you to install directly from your own repo, which gives you more consistent builds, since you can't warranty third party repos to be available, to remain in same domain or to respect semantic versioning.
• Allows developers to check easily and fast what resources will be installed by the helm chart at path helm/charts/{chart}

The main disadvantage is:

• Every time you add a new chart it increases the Pull Request size making it harder to code review.

In my opinion the advantages here clearly outweighs the disadvantages.

kustomize subdirectory

Now lets proceed with subdir kustomize available here.

At the top level you will notice we have our ArgoCD ApplicationSet one per environment at path: kustomize/ApplicationSet.{env}.yaml

Each of these ApplicationSet will configure two ArgoCD "App of Apps":

1. helm-apps: App of Apps for all helm charts
2. custom-apps: App of Apps for all non helm charts, basically everything that is your custom code.

# kustomize/ApplicationSet.dev.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: appset-dev
  namespace: argocd
spec:
  goTemplate: true # activate Go templating
  generators:
  - list:
      elements:
        # App of apps for all helm charts
        - appName: helm-apps
          appType: "helm"
          path: kubernetes/kustomize/overlays/dev/helm-apps
          syncWave: 0
        # App of apps for all non helm charts
        - appName: custom-apps
          appType: "custom"
          path: kubernetes/kustomize/overlays/dev/custom-apps
          syncWave: 1
  template:
    # Application Metadata
    metadata:
      name: "{{.appName}}"
      namespace: argocd
      finalizers:
        # needed so k8s resources are deleted when app is deleted as well
        - resources-finalizer.argocd.argoproj.io
      labels:
        appType: "{{.appType}}"
      annotations:
        # order of deployment based on syncWave
        argocd.argoproj.io/sync-wave: "{{.syncWave}}"
    spec:
      # spec same as it (spec field) would be in a
      # typical ArgoCD Application.yaml, just replace 
      # generator values defined above.
      project: default
      source:
        repoURL: git@github.com:my-org/my-example-repo.git
        targetRevision: dev
        path: "{{.path}}" # path must point to kustomize overlays        
      destination:
        server: 'https://kubernetes.default.svc'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
          allowEmpty: true
        syncOptions:
          - PruneLast=true
          - CreateNamespace=true
          - PrunePropagationPolicy=foreground
        retry:
          limit: 2

Lets continue diving into kustomize sub directory it has two sub directories:

    └─ kustomize
     ├─ base       # base k8s resources for all environments
     └─ overlays    # kustomize overlays for customizing different 

Both kustomize/base and kustomize/overlays have helm-apps sub directory and a custom-apps sub directory.

• helm-apps contains the argoCD Application configuration pointing to helm/charts/{chart} and the helm/values/{env}.values.yaml . Check base kustomization file for ArgoCD kustomize/base/helm-apps/argocd-app.yaml and overlay for dev environment at kustomize/overlays/dev/helm-apps/argocd-app.yaml for reference of how this works.
• custom-apps contains a simple busybox http server I have added for your reference make sure to check respective base and overlay directories to understand how to structure custom resources.

Once you have everything in place you can scale this dir structure which works well for multiple of applications in same cluster. You might want to refactor and break down custom-apps into smaller chunks once you reach a significant number of apps.