Rollback Strategy

Q. Tell me how you handle rollback ? with example ?

Here’s a clear, interview-ready answer for handling rollbacks in your Kubernetes/CI-CD setup, with an example specific to your fintech wealth advisory platform:

Q: How do you handle rollbacks?

Answer:

We handle rollbacks at both CI/CD and Kubernetes levels using Helm and Argo CD, ensuring minimal downtime and fast recovery.

1️⃣ Helm Rollback (Namespace-specific)

Every microservice is deployed via Helm charts into its environment-specific namespace (auth-service-qa, auth-service-dev, auth-service-prod).
Helm keeps a release history per namespace.
If a deployment fails or a bug is detected after deployment, we can rollback to a previous working release with a single command:

# Check release history
helm history auth-service --namespace auth-service-qa

# Rollback to previous version
helm rollback auth-service 2 --namespace auth-service-qa

2 refers to the previous release version.
This immediately restores the previous stable image/configuration without affecting other services or namespaces.

2️⃣ Argo CD Rollback (GitOps Approach)

All deployments are GitOps-managed via Argo CD.
If a release introduces issues, we can revert the Git commit in the CD repo (Helm chart or values file) to the last stable version:

# Sync Argo CD application after revert
argocd app sync auth-service-qa --prune

Argo CD applies the previous Helm chart state to the namespace.
This ensures the cluster state matches the repository.

3️⃣ CI/CD Rollback Triggers

CI pipelines are configured to tag Docker images with build numbers.
If a rollback is needed, we can trigger the Helm chart to deploy a specific previous image tag:

helm upgrade --install auth-service auth-service/helm-chart \
  --namespace auth-service-qa \
  --set image.tag=45  # previous stable build

This allows fast rollback without rebuilding images.

4️⃣ Benefits of Our Rollback Strategy

✅ Namespace-level isolation prevents impact on other services.
✅ Multiple rollback options: Helm release, Argo CD GitOps, CI image tag.
✅ Zero-downtime rollback using rolling updates in Helm + Kubernetes.
✅ Ensures production parity with DR clusters, so DR can also be rolled back in sync.

Example Scenario in QA Environment:

QA team tests a new feature in auth-service-qa.
Deployment introduces a bug in authentication.
Pipeline or Argo CD triggers rollback to previous stable release:
```
 helm rollback auth-service 2 --namespace auth-service-qa
```
QA can immediately resume testing without waiting for a new build.

Q. So we don’t use commit style here right ? and how its diff with this and which is better approach according to you for our fintech app ?

Exactly! Let me explain the difference between commit-style (GitOps) rollback vs. Helm/Argo CD release rollback and which approach fits your wealth advisory fintech platform.

1️⃣ Commit-Style Rollback (GitOps)

How it works:

In GitOps, your cluster state is a reflection of the Git repository (CD repo).
Rollback happens by reverting the Git commit that introduced the bad change.
Argo CD or Flux detects the repo change and syncs the cluster back to the previous state.

Example:

# Revert bad commit in CD repo
git revert <commit-hash>
git push origin qa

# Argo CD sync applies previous state
argocd app sync auth-service-qa

Pros:

Full history of changes in Git.
Enforces audit/compliance, ideal for regulated fintech apps.
Guarantees cluster state matches repository.

Cons:

Requires Git operations for rollback (slightly slower).
Not ideal for emergency quick rollback unless pipeline is automated.

2️⃣ Helm/Argo CD Release Rollback

How it works:

Helm keeps release history per namespace.
Rollback is done via Helm rollback command, optionally via Argo CD UI.

Example:

helm rollback auth-service 2 --namespace auth-service-qa

Pros:

Instant rollback without touching Git.
Works even if GitOps repo is behind or has pending commits.
Good for hotfixes in QA or dev, faster recovery.

Cons:

Less audit-friendly than Git revert (need to check Helm history separately).
Manual rollback might cause divergence if Git repo is not updated.

3️⃣ Which is Better for Our Fintech Wealth Advisory Platform?

Aspect	Commit-Style (GitOps)	Helm/Release Rollback
Speed	Slower (requires Git revert)	Fast (single command)
Audit & Compliance	Excellent (Git history)	Moderate (Helm history only)
Use Case	Production, regulated changes	QA/dev, hotfixes, emergency rollback
Cluster-State Consistency	Perfect, always matches repo	Needs manual check

Our Strategy:

Dev/QA/Stage: Use Helm/Release rollback for speed and agility.
Prod/DR: Use GitOps commit-style rollback for audit, compliance, and full traceability.

Rationale:

Fintech apps require strict audit and compliance in production.
Non-prod environments need rapid testing and rollback without waiting for Git operations.
This hybrid approach balances speed, safety, and compliance.

Q. So which one is better ?

For your wealth advisory fintech platform, the “better” approach depends on environment and goals:

Recommendation:

Dev / QA / Stage: Helm/Release rollback
- Reason: Fast, simple, allows rapid iteration and testing.
- Developers and QA can quickly revert a failed deployment without touching Git.
Production / DR: Commit-style GitOps rollback
- Reason: Full audit trail, compliance-friendly, ensures cluster state exactly matches repo.
- Slower than Helm rollback but critical for regulated fintech environments.

✅ Summary:

Speed-focused rollback: Helm rollback → good for dev/QA.
Compliance-focused rollback: GitOps commit revert → mandatory for prod/DR.

Hybrid approach gives you both agility in testing and regulatory safety in production.

Q. So I can say Commit based right ? and for `qa/stage` and `production` it gets deployed when we create tag so we use tag rollback version right give example of command right from creating tag and then rolling back ?

( Note : We use shell script to update image inside the deployment i.e. helm manifest )

Got it! Here’s a modified interview-ready answer incorporating your shell script that updates the image in Helm manifests, along with the commit/tag-based deployment and rollback workflow:

Commit/Tag-Based Deployment & Rollback Workflow with Shell Script

1️⃣ Create a Tag for Deployment

When code in a branch (e.g., qa or prod) is ready to deploy, we create a Git tag:

git checkout prod
git tag -a v1.2.0 -m "Release v1.2.0 for production"
git push origin v1.2.0

Tags are immutable and serve as the deployment reference.

2️⃣ CI/CD Updates Helm Manifest Using Shell Script

Our CI/CD pipeline has a shell script that updates the Docker image tag inside the Helm values.yaml or manifest before deployment.

Example shell script (update-helm-image.sh):

#!/bin/bash

# Usage: ./update-helm-image.sh <service> <namespace> <image-tag>
SERVICE=$1
NAMESPACE=$2
IMAGE_TAG=$3

HELM_PATH="./$SERVICE/helm-chart"

echo "Updating image tag for $SERVICE to $IMAGE_TAG in namespace $NAMESPACE"
sed -i "s|repository:.*|repository: 123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/$SERVICE|" $HELM_PATH/values.yaml
sed -i "s|tag:.*|tag: $IMAGE_TAG|" $HELM_PATH/values.yaml

echo "Deploying via Helm..."
helm upgrade --install $SERVICE $HELM_PATH \
  --namespace $NAMESPACE \
  --create-namespace

Pipeline calls this script automatically after building and pushing the Docker image.

./update-helm-image.sh auth-service auth-service-qa v1.2.0

This ensures image is updated in Helm chart before deployment, without manually editing manifests.

3️⃣ Rollback to Previous Tag

If QA or production finds an issue, rollback is simple:

# Use previous stable tag
./update-helm-image.sh auth-service auth-service-qa v1.1.5

Or, in production with GitOps, revert the Git commit for the Helm values.yaml:

git checkout prod
git revert <commit-hash-for-v1.2.0>
git push origin prod
argocd app sync auth-service-prod

Cluster now runs previous stable version, maintaining zero downtime with rolling updates.

4️⃣ QA/Stage Deployments

QA and Stage also follow tagged deployments, with shell script updating Helm manifests automatically:

./update-helm-image.sh payments-service payments-service-stage v1.2.0

Each microservice deployed to its own namespace (auth-service-qa, payments-service-stage) → isolates tests for functional, integration, or regression testing.

5️⃣ Key Points for Interview

Shell script automates image updates in Helm manifests → reduces human error.
Tags ensure immutable, traceable releases.
CI/CD automatically triggers deployment to dev/qa/stage; prod is manual approval.
Rollback is simple via script with previous tag or GitOps revert.
Namespace isolation allows multiple test types in QA/stage clusters.

If you want, I can also draw a diagram showing shell script → Helm manifest → CI/CD → Argo CD → namespace → rollback to make this extremely clear for interview explanations.

Rollback Strategy

Q. Tell me how you handle rollback ? with example ?

Q: How do you handle rollbacks?

1️⃣ Helm Rollback (Namespace-specific)

2️⃣ Argo CD Rollback (GitOps Approach)

3️⃣ CI/CD Rollback Triggers

4️⃣ Benefits of Our Rollback Strategy

Example Scenario in QA Environment:

Q. So we don’t use commit style here right ? and how its diff with this and which is better approach according to you for our fintech app ?

1️⃣ Commit-Style Rollback (GitOps)

2️⃣ Helm/Argo CD Release Rollback

3️⃣ Which is Better for Our Fintech Wealth Advisory Platform?

Q. So which one is better ?

Recommendation:

Q. So I can say Commit based right ? and for `qa/stage` and `production` it gets deployed when we create tag so we use tag rollback version right give example of command right from creating tag and then rolling back ?

Commit/Tag-Based Deployment & Rollback Workflow with Shell Script

1️⃣ Create a Tag for Deployment

2️⃣ CI/CD Updates Helm Manifest Using Shell Script

3️⃣ Rollback to Previous Tag

4️⃣ QA/Stage Deployments

5️⃣ Key Points for Interview

Comments

More from this blog

Python Project setup for production and how secrets are handled right from code to Dockerfile and K8S manifest files./ SA + IRSA + RoleBinding + RBAC

frontend+Backend+Containerization+K8S Deploy+ Service Mesh Flow

Containerizing and deploying on K8S confusion solved

Frontend - Backend - DB

DR FLOW - Final

Command Palette

Q. Tell me how you handle rollback ? with example ?

Q: How do you handle rollbacks?

1️⃣ Helm Rollback (Namespace-specific)

2️⃣ Argo CD Rollback (GitOps Approach)

3️⃣ CI/CD Rollback Triggers

4️⃣ Benefits of Our Rollback Strategy

Example Scenario in QA Environment:

Q. So we don’t use commit style here right ? and how its diff with this and which is better approach according to you for our fintech app ?

1️⃣ Commit-Style Rollback (GitOps)

2️⃣ Helm/Argo CD Release Rollback

3️⃣ Which is Better for Our Fintech Wealth Advisory Platform?

Q. So which one is better ?

Recommendation:

Q. So I can say Commit based right ? and for qa/stage and production it gets deployed when we create tag so we use tag rollback version right give example of command right from creating tag and then rolling back ?

Commit/Tag-Based Deployment & Rollback Workflow with Shell Script

1️⃣ Create a Tag for Deployment

2️⃣ CI/CD Updates Helm Manifest Using Shell Script

3️⃣ Rollback to Previous Tag

4️⃣ QA/Stage Deployments

5️⃣ Key Points for Interview

Comments

More from this blog

Q. So I can say Commit based right ? and for `qa/stage` and `production` it gets deployed when we create tag so we use tag rollback version right give example of command right from creating tag and then rolling back ?