# Rollback Strategy ## Q. Tell me how you handle rollback ? with example ? Here’s a **clear, interview-ready answer** for handling rollbacks in your Kubernetes/CI-CD setup, with an example specific to your fintech wealth advisory platform: --- ### **Q: How do you handle rollbacks?** **Answer:** We handle rollbacks at **both CI/CD and Kubernetes levels** using Helm and Argo CD, ensuring minimal downtime and fast recovery. --- ### **1️⃣ Helm Rollback (Namespace-specific)** * Every microservice is deployed via **Helm charts** into its environment-specific namespace (`auth-service-qa`, `auth-service-dev`, `auth-service-prod`). * Helm keeps a **release history** per namespace. * If a deployment fails or a bug is detected after deployment, we can **rollback to a previous working release** with a single command: ```plaintext # Check release history helm history auth-service --namespace auth-service-qa # Rollback to previous version helm rollback auth-service 2 --namespace auth-service-qa ``` * `2` refers to the previous release version. * This immediately restores the **previous stable image/configuration** without affecting other services or namespaces. --- ### **2️⃣ Argo CD Rollback (GitOps Approach)** * All deployments are GitOps-managed via **Argo CD**. * If a release introduces issues, we can **revert the Git commit** in the CD repo (Helm chart or values file) to the last stable version: ```plaintext # Sync Argo CD application after revert argocd app sync auth-service-qa --prune ``` * Argo CD applies the previous Helm chart state to the namespace. * This ensures the **cluster state matches the repository**. --- ### **3️⃣ CI/CD Rollback Triggers** * CI pipelines are configured to **tag Docker images with build numbers**. * If a rollback is needed, we can trigger the Helm chart to deploy a **specific previous image tag**: ```plaintext helm upgrade --install auth-service auth-service/helm-chart \ --namespace auth-service-qa \ --set image.tag=45 # previous stable build ``` * This allows **fast rollback without rebuilding images**. --- ### **4️⃣ Benefits of Our Rollback Strategy** ✅ Namespace-level isolation prevents impact on other services. ✅ Multiple rollback options: Helm release, Argo CD GitOps, CI image tag. ✅ Zero-downtime rollback using **rolling updates** in Helm + Kubernetes. ✅ Ensures production parity with DR clusters, so DR can also be rolled back in sync. --- ### **Example Scenario in QA Environment:** 1. QA team tests a new feature in `auth-service-qa`. 2. Deployment introduces a bug in authentication. 3. Pipeline or Argo CD triggers rollback to **previous stable release**: ```plaintext helm rollback auth-service 2 --namespace auth-service-qa ``` 4. QA can immediately resume testing without waiting for a new build. --- ## Q. So we don’t use commit style here right ? and how its diff with this and which is better approach according to you for our fintech app ? Exactly! Let me explain the **difference between commit-style (GitOps) rollback vs. Helm/Argo CD release rollback** and which approach fits your **wealth advisory fintech platform**. --- ## **1️⃣ Commit-Style Rollback (GitOps)** **How it works:** * In GitOps, your **cluster state is a reflection of the Git repository** (CD repo). * Rollback happens by **reverting the Git commit** that introduced the bad change. * Argo CD or Flux detects the repo change and **syncs the cluster back** to the previous state. **Example:** ```plaintext # Revert bad commit in CD repo git revert git push origin qa # Argo CD sync applies previous state argocd app sync auth-service-qa ``` **Pros:** * **Full history of changes** in Git. * Enforces **audit/compliance**, ideal for regulated fintech apps. * Guarantees **cluster state matches repository**. **Cons:** * Requires **Git operations** for rollback (slightly slower). * Not ideal for **emergency quick rollback** unless pipeline is automated. --- ## **2️⃣ Helm/Argo CD Release Rollback** **How it works:** * Helm keeps **release history per namespace**. * Rollback is done via **Helm rollback command**, optionally via Argo CD UI. **Example:** ```plaintext helm rollback auth-service 2 --namespace auth-service-qa ``` **Pros:** * **Instant rollback** without touching Git. * Works even if GitOps repo is behind or has pending commits. * Good for **hotfixes in QA or dev**, faster recovery. **Cons:** * Less **audit-friendly** than Git revert (need to check Helm history separately). * Manual rollback might cause divergence if Git repo is not updated. --- ## **3️⃣ Which is Better for Our Fintech Wealth Advisory Platform?** | Aspect | Commit-Style (GitOps) | Helm/Release Rollback | | --- | --- | --- | | **Speed** | Slower (requires Git revert) | Fast (single command) | | **Audit & Compliance** | Excellent (Git history) | Moderate (Helm history only) | | **Use Case** | Production, regulated changes | QA/dev, hotfixes, emergency rollback | | **Cluster-State Consistency** | Perfect, always matches repo | Needs manual check | **Our Strategy:** * **Dev/QA/Stage:** Use **Helm/Release rollback** for speed and agility. * **Prod/DR:** Use **GitOps commit-style rollback** for audit, compliance, and full traceability. **Rationale:** * Fintech apps require **strict audit and compliance** in production. * Non-prod environments need **rapid testing and rollback** without waiting for Git operations. * This hybrid approach balances **speed, safety, and compliance**. --- ## Q. So which one is better ? For your **wealth advisory fintech platform**, the “better” approach depends on **environment and goals**: --- ### **Recommendation:** * **Dev / QA / Stage:** **Helm/Release rollback** * **Reason:** Fast, simple, allows rapid iteration and testing. * Developers and QA can quickly revert a failed deployment without touching Git. * **Production / DR:** **Commit-style GitOps rollback** * **Reason:** Full audit trail, compliance-friendly, ensures cluster state exactly matches repo. * Slower than Helm rollback but critical for **regulated fintech environments**. --- ✅ **Summary:** * **Speed-focused rollback:** Helm rollback → good for dev/QA. * **Compliance-focused rollback:** GitOps commit revert → mandatory for prod/DR. **Hybrid approach** gives you **both agility in testing** and **regulatory safety in production**. --- ## Q. So I can say Commit based right ? and for `qa/stage` and `production` it gets deployed when we create tag so we use tag rollback version right give example of command right from creating tag and then rolling back ? **( Note : We use shell script to update image inside the deployment i.e. helm manifest )** --- Got it! Here’s a **modified interview-ready answer** incorporating your **shell script that updates the image in Helm manifests**, along with the commit/tag-based deployment and rollback workflow: --- ## **Commit/Tag-Based Deployment & Rollback Workflow with Shell Script** ### **1️⃣ Create a Tag for Deployment** * When code in a branch (e.g., `qa` or `prod`) is ready to deploy, we **create a Git tag**: ```plaintext git checkout prod git tag -a v1.2.0 -m "Release v1.2.0 for production" git push origin v1.2.0 ``` * Tags are **immutable** and serve as the deployment reference. --- ### **2️⃣ CI/CD Updates Helm Manifest Using Shell Script** * Our CI/CD pipeline has a **shell script** that updates the Docker image tag inside the Helm `values.yaml` or manifest before deployment. **Example shell script (**[`update-helm-image.sh`](http://update-helm-image.sh)): ```plaintext #!/bin/bash # Usage: ./update-helm-image.sh SERVICE=$1 NAMESPACE=$2 IMAGE_TAG=$3 HELM_PATH="./$SERVICE/helm-chart" echo "Updating image tag for $SERVICE to $IMAGE_TAG in namespace $NAMESPACE" sed -i "s|repository:.*|repository: 123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/$SERVICE|" $HELM_PATH/values.yaml sed -i "s|tag:.*|tag: $IMAGE_TAG|" $HELM_PATH/values.yaml echo "Deploying via Helm..." helm upgrade --install $SERVICE $HELM_PATH \ --namespace $NAMESPACE \ --create-namespace ``` * Pipeline calls this script automatically after building and pushing the Docker image. ```plaintext ./update-helm-image.sh auth-service auth-service-qa v1.2.0 ``` * This ensures **image is updated in Helm chart** before deployment, without manually editing manifests. --- ### **3️⃣ Rollback to Previous Tag** * If QA or production finds an issue, rollback is simple: ```plaintext # Use previous stable tag ./update-helm-image.sh auth-service auth-service-qa v1.1.5 ``` * Or, in production with GitOps, revert the Git commit for the Helm values.yaml: ```plaintext git checkout prod git revert git push origin prod argocd app sync auth-service-prod ``` * Cluster now runs **previous stable version**, maintaining **zero downtime** with rolling updates. --- ### **4️⃣ QA/Stage Deployments** * QA and Stage also follow **tagged deployments**, with **shell script updating Helm manifests** automatically: ```plaintext ./update-helm-image.sh payments-service payments-service-stage v1.2.0 ``` * Each microservice deployed to **its own namespace** (`auth-service-qa`, `payments-service-stage`) → isolates tests for functional, integration, or regression testing. --- ### **5️⃣ Key Points for Interview** * **Shell script automates image updates** in Helm manifests → reduces human error. * Tags ensure **immutable, traceable releases**. * CI/CD automatically triggers deployment to dev/qa/stage; prod is **manual approval**. * Rollback is simple via **script with previous tag** or **GitOps revert**. * **Namespace isolation** allows multiple test types in QA/stage clusters. --- If you want, I can also **draw a diagram showing shell script → Helm manifest → CI/CD → Argo CD → namespace → rollback** to make this extremely clear for interview explanations.