Introduction
Deploying and managing applications in Kubernetes environments can be challenging, especially when dealing with complex configurations and multiple environments. Imagine a scenario where an operator needs to be deployed across development, staging, and production environments, each requiring unique configurations and settings. Without a structured approach, handling this for each environment can quickly become error-prone and time-consuming.
In this blog, the focus will be on simplifying and streamlining AWS ROSA (RedHat OpenShift Service on AWS) OpenShift Operator deployments using Kustomize and RedHat OpenShift GitOps (ArgoCD). By implementing GitOps methodologies, architects & engineers can alleviate the burden of managing multiple configurations while ensuring that deployments are consistent and reproducible. With GitOps, Git repository becomes the single source of truth for all configuration changes, making deployments predictable and auditable.
The exploration will cover the benefits of Kustomize, highlighting its seamless integration with Kubernetes and OpenShift, including support for both oc and kubectl commands. Additionally, the advantages of using Argo CD for continuous deployment will be discussed, showcasing how it enhances the deployment process by providing automated synchronization with Git repositories.
Why Kustomize?
Kustomize is a tool that enables configuration management through a "base and overlay" model, allowing to manage common and environment-specific configurations without duplicating YAML files.
Native Kubernetes Integration: Kustomize is built into both kubectl and oc (OpenShift CLI), offering a seamless experience.
Base and Overlay Structure: Define a common configuration (base) and apply environment-specific customizations (overlays) using patches.
Declarative Configurations: By adhering to declarative principles, Kustomize simplifies version control and collaboration.
Benefits of GitOps with OpenShift GitOps (ArgoCD)
OpenShift GitOps (based on Argo CD) is a RedHat native continuous deployment tool that synchronizes the cluster's state with the configurations stored in Git. Here’s how it helps:
Automated Continuous Delivery: Monitor Git repositories for changes, and apply updates automatically to the OpenShift cluster.
Seamless Kustomize Integration: Manage complex, environment-specific configurations without duplicating YAML files.
Declarative and Auditable: Track every change made to the cluster through Git, offering full traceability.
High-level flow deploying an Operator using OpenShift GitOps
This section tees up the steps for deploying an operator that is commonly used in the industry. Dynatrace is a well-known observability platform that enables organizations to monitor the performance of their applications and infrastructure effectively. In developing this blog, the decision to showcase the Dynatrace Operator stemmed from its widespread use and recognition within the industry. The focus here is not to provide a comprehensive guide to fully configuring Dynatrace, which involves additional steps like handling subscription keys, but rather to highlight the operator deployment process. This approach emphasizes the importance of automating the deployment in a consistent manner. For those interested in fully configuring Dynatrace, additional information can be found @ https://www.redhat.com/en/blog/partner-showcase-openshift-app-observability-with-dynatrace-operator.
Below are the overarching setup process to deploy the Dynatrace Operator using Kustomize and OpenShift GitOps.
Step 1: Set Up the Folder Structure
Create a structured folder layout for managing the Dynatrace Operator deployment using Kustomize. This structure allows for clear organization and easy management of base and environment-specific configurations.
code├── base
│ ├── kustomization.yaml
│ └── dynatrace-operator-subscription.yaml
├── overlays
│ ├── development
│ │ └── kustomization.yaml
│ ├── staging
│ │ └── kustomization.yaml
│ └── production
│ └── kustomization.yaml
└── argocd
└── application.yaml
└── kustomization.yaml
Base Folder: Contains the base configuration for the Dynatrace Operator subscription. This folder holds the essential deployment YAML that can be reused across different environments.
Overlays Folder: Each environment (development, staging, production) has its own folder containing its specific Kustomization file. Overlays allow for customization of the base configuration without duplicating it.
ArgoCD Folder: Contains the Argo CD application manifest and its own Kustomization file that defines how Argo CD will manage the deployment of the Dynatrace Operator using Kustomize.
Step 2: Define the Base Configuration
In the base folder, the configuration files specify the core settings for the Dynatrace Operator. This includes deployment manifests and services that are common across all environments. PATCH-ME placeholders provides a convenient way to replace with environment specific values.
Example: base/dynatrace-operator-subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
labels:
operators.coreos.com/dynatrace-operator.openshift-operators: ""
name: dynatrace-operator
namespace: openshift-operators
spec:
channel: PATCH-ME
installPlanApproval: PATCH-ME
name: dynatrace-operator
source: certified-operators
sourceNamespace: openshift-marketplace
Step 2: Create the base Kustomization configuration
This file specifies that the Dynatrace Operator subscription defined in dynatrace-operator-subscription.yaml is a resource that Kustomize should apply.
Example: base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- dynatrace-operator-subscription.yaml
Step 3: Create Environment-Specific Overlays
This file references the base configuration and applies the necessary changes specific to the development environment. The inline patch specifies what values will replace the PATCH-ME placeholders in the base YAML.
Example: overlays/development/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- target:
version: v1
kind: Subscription
name: dynatrace-operator
patch: |
- op: replace
path: /spec/channel
value: alpha # Dev-specific channel
- op: replace
path: /spec/installPlanApproval
value: Automatic # Dev-specific install plan approval
Step 4: Deploy using OpenShift GitOps(ArgoCD)
Create an Argo CD Application manifest that points to the appropriate overlay for deployment. This configuration informs Argo CD to sync the specified environment's configuration to the OpenShift cluster automatically.
Example: Argo CD Application Manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: dynatrace-operator
namespace: openshift-gitops
spec:
project: default
source:
repoURL: 'https://github.com/your-repo/dynatrace-operator'
targetRevision: HEAD
path: overlays/development
kustomize:
namePrefix: dev-
destination:
server: 'https://kubernetes.default.svc'
namespace: dynatrace-operators
syncPolicy:
automated:
prune: true
selfHeal: true
Source Path: By setting path: overlays/development, Argo CD knows to apply the development overlay and deploy the development-specific configuration.
Destination Namespace: The namespace: dynatrace-operators defines where the operator will be deployed. This can be different for each environment.
Automated Sync: The syncPolicy ensures that Argo CD automatically applies changes when they are pushed and merge to Git repo, keeping the cluster state in sync with the desired state in Git.
Why GitOps (Argo CD) Simplifies Management
GitOps, particularly via OpenShift GitOps (Argo CD), offers a superior method of managing operator deployments for several reasons:
Centralized Source of Truth: The Git repository serves as the single source of truth for all configurations. This ensures consistency across environments and makes it easy to audit changes.
Automated Deployments: Argo CD automatically syncs any changes in the Git repository to the Kubernetes cluster, ensuring that the cluster state always matches the desired state in Git.
Environment Control: By using overlays, we can easily customize deployments for different environments (dev, staging, production), while still maintaining a common base configuration.
Operator-Driven Approach: Since GitOps itself is managed by an operator in ROSA, the operational burden is reduced. However, deploying GitOps operator to the cluster must be done manually or via automation (e.g., Terraform) to begin with, ensuring the operator is in place to manage future deployments.
Enabling OpenShift GitOps
To enable OpenShift GitOps (Argo CD) in the AWS ROSA cluster, we need to deploy the OpenShift GitOps Operator. This can be done manually through the OpenShift web console or automated using tools like Terraform or Helm. We recommend installing the OpenShift GitOps operator as soon as the cluster is provisioned, treating it as a day 0 activity. From that point forward, GitOps(ArgoCD) should manage all operators and application deployments.
Alternatives to Kustomize and GitOps
While Kustomize and GitOps provide a streamlined approach to deploying OpenShift Operators, other methods also exist for managing configurations and deployments. Terraform is a powerful tool for provisioning and managing infrastructure, making it an excellent choice for cloud resources. However, when it comes to managing Kubernetes configurations, Terraform might not be the best fit due to its complexity in handling Kubernetes manifests and the potential for configuration drift in dynamic environments.
Another option is Helm, which is a popular package manager for Kubernetes. Helm works well with OpenShift GitOps and can simplify the deployment of complex applications through templating. However, for the purposes of this blog, Kustomize was chosen for its straightforward approach to managing configurations directly within Kubernetes without the need for additional templating.
Conclusion
Using Kustomize and GitOps with the OpenShift GitOps Operator for deploying OpenShift Operators on AWS ROSA provides a scalable, maintainable, and consistent approach to managing Kubernetes configurations. Leveraging the power of Kustomize's native Kubernetes support along with OpenShift GitOps' continuous deployment capabilities facilitates efficient and reliable application deployments. This integration ensures that all changes are tracked in version control, allowing for better collaboration and traceability within teams. By automating deployment processes and managing configurations effectively, DevOps practices become smoother and more robust, leading to improved productivity and operational excellence. With the Red Hat OpenShift GitOps Operator, organizations can confidently streamline their workflows while ensuring consistency across different environments, ultimately enhancing the overall performance and reliability of their Kubernetes applications.
\The code snippets provided in this blog server as a framework for understanding the deployment logic. They have been sanitized of real data and should be used for basic guidelines only.*
