Introduction
In today’s fast-paced world of software development, building scalable, reliable, and automated infrastructure is very crucial. This blog summarizes creating a multi-cloud, fully automated e-commerce platform from Infrastructure setup to deployment. By leveraging Terraform, I provisioned essential infrastructure components across Google Cloud Platform (GCP) and Microsoft Azure. The sample application itself is an e-commerce solution, featuring services for authentication, payments, product management, and order processing.
To ensure seamless deployments, I integrated Jenkins for running CI/CD pipeline, enabling continuous integration and delivery of the application across environments. This project showcases a robust workflow combining the power of Infrastructure as Code (IaC), containerized microservices, CI/CD and cloud automation.
Let’s dive into the technical details.
This will be broken down into three parts:
- Infrastructure provisioning using Terraform
- Application deployment to Azure Kubernetes Service (AKS)
- Automated deployment using Jenkins CI/CD
Prerequisites
1. Azure CLI installed
2. kubectl installed
3. Docker installed
4. Node.js 18+ installed
5. Azure subscription
6. Google Cloud subscription
7. Google Cloud CLI
8. Terraform
Infrastructure
- Azure Kubernetes Service (AKS)
- Azure Container Registry (ACR)
- Azure Key Vault
- Azure Application Gateway
- Azure DNS
- Google Cloud VM Instance
- Jenkins
- Other components provisioned and configured which will be later used in our App but not used in this specific project: Microsoft Azure PostgreSQL – Flexible Server, Azure Cache for Redis
Local Development Setup
# Clone the repository
git clone <GitHub URL>
cd ecomm-app# Install dependencies for all services
npm install# Set up environment variables
cp .env.example .env# Start all services locally – from the root folder
npm run devStep 1: Infrastructure setup using Terraform
Jenkins Server on GCP VM Instances
We will setup Jenkins server on Google Cloud (Ok, you may be wondering why not use Azure platform, since other components will be there, well, I wanted to leverage on the Google Cloud free credits, and I intend to run the Jenkins server for a little bit longer).
Authenticate to Google Cloud, there are a couple of methods, the common ones are:
- Authenticate using personal account
gcloud auth login
gcloud config set project PROJECT_ID2. Authenticate to Google cloud using ADC
gcloud auth application-default loginBelow is the terraform script – main.tf, you can download the rest from here.
# Create a Google Cloud Network
resource "google_compute_network" "jenkins_nkw" {
name = "jenkins-nkw"
auto_create_subnetworks = false
}
resource "google_compute_subnetwork" "jenkins_subnet" {
name = "jenkins-subnet"
ip_cidr_range = "10.0.0.0/24"
network = google_compute_network.jenkins_nkw.self_link
region = var.region
}
# Create a Firewall Rule to allow Jenkins port
resource "google_compute_firewall" "jenkins_firewall" {
name = "jenkins-firewall"
network = google_compute_network.jenkins_nkw.id
allow {
protocol = "tcp"
ports = ["22", "8080"]
}
source_ranges = ["0.0.0.0/0"]
target_tags = ["jenkins"]
}
# Create a Google Compute Instance
resource "google_compute_instance" "jenkins_server" {
name = "jenkins-server"
machine_type = var.machine_type
tags = ["jenkins"]
boot_disk {
initialize_params {
image = "ubuntu-os-cloud/ubuntu-2004-lts"
size = 50
}
}
network_interface {
subnetwork = google_compute_subnetwork.jenkins_subnet.self_link
access_config {
// Ephemeral public IP
}
}
metadata_startup_script = file("${path.module}/scripts/install_jenkins.sh")
service_account {
email = google_service_account.jenkins_sa.email
scopes = ["cloud-platform"]
}
}
resource "google_service_account" "jenkins_sa" {
account_id = "jenkins-service-account"
display_name = "Jenkins Service Account"
}
resource "google_project_iam_member" "jenkins_storage_admin" {
project = var.project_id
role = "roles/storage.admin"
member = "serviceAccount:${google_service_account.jenkins_sa.email}"
}}
First, create bucket on google cloud to store terraform state
gsutil mb -p PROJECT_ID gs://BUCKET_NAMERun the below commands to provision the server:
Initialize and apply Terraform:
terraform init
terraform plan -out=tfplan
terraform applyOutputs:
jenkins_public_ip = “<public-ip-jenkins>”
jenkins_url = “http://<public-ip-jenkins>:8080”
- Retrieve the password
- Install suggested Plugin.
Setup AKS Cluster, Application Gateway, Redis, PostgreSQL Infra on Azure
Provision the AKS cluster using terraform. You can view and download the terraform scripts from Here:
Authenticate to Azure
Login to Azure cli:
az loginTo explicitly login with Service Principal:
az login --service-principal \
--username APP_ID \
--password PASSWORD \
--tenant TENANT_IDFirst, create the storage account for Terraform state:
az group create --name terraform-state-rg --location uksouthaz storage account create --name tfstatedev24 --resource-group terraform-state-rg --location eastus --sku Standard_LRSaz storage container create --name tfstate --account-name tfstatedev24Run the terraform scripts by navigating to folder environment/dev, to download all the scripts, visit the GitHub link, clone or download to your machine.
Here is a sample config from one of the modules:
# modules/aks/main.tf
resource "azurerm_kubernetes_cluster" "aks" {
name = "${var.prefix}-aks"
location = var.location
resource_group_name = var.resource_group_name
dns_prefix = "${var.prefix}-aks"
kubernetes_version = var.kubernetes_version
default_node_pool {
name = "system"
node_count = var.system_node_count
vm_size = var.system_node_vm_size
vnet_subnet_id = var.subnet_id
enable_auto_scaling = true
min_count = var.system_node_min_count
max_count = var.system_node_max_count
}
identity {
type = "SystemAssigned"
}
key_vault_secrets_provider {
secret_rotation_enabled = var.enable_secret_rotation
#secret_rotation_interval = "2m" # Optional: Configure rotation interval
}
# Enable OIDC and Workload Identity
oidc_issuer_enabled = true
workload_identity_enabled = true
network_profile {
network_plugin = "azure"
network_policy = "calico"
load_balancer_sku = "standard"
service_cidr = "172.16.0.0/16"
dns_service_ip = "172.16.0.10"
}
oms_agent {
log_analytics_workspace_id = var.log_analytics_workspace_id
}
tags = var.tags
}
# On-demand node pool for critical workloads
resource "azurerm_kubernetes_cluster_node_pool" "ondemand" {
name = "ondemand"
kubernetes_cluster_id = azurerm_kubernetes_cluster.aks.id
vm_size = var.ondemand_node_vm_size
vnet_subnet_id = var.subnet_id
priority = "Regular"
enable_auto_scaling = true
min_count = var.ondemand_node_min_count
max_count = var.ondemand_node_max_count
node_labels = {
"nodepool-type" = "ondemand"
"environment" = var.environment
}
node_taints = []
tags = var.tags
}
# Spot instance node pool for cost optimization
resource "azurerm_kubernetes_cluster_node_pool" "spot" {
name = "spot"
kubernetes_cluster_id = azurerm_kubernetes_cluster.aks.id
vm_size = var.spot_node_vm_size
vnet_subnet_id = var.subnet_id
priority = "Spot"
eviction_policy = "Delete"
spot_max_price = var.spot_price_max
enable_auto_scaling = true
min_count = var.spot_node_min_count
max_count = var.spot_node_max_count
node_labels = {
"nodepool-type" = "spot"
"environment" = var.environment
}
node_taints = [
"kubernetes.azure.com/scalesetpriority=spot:NoSchedule"
]
tags = var.tags
}
# Role assignment for Key Vault access
resource "azurerm_role_assignment" "aks_keyvault" {
scope = var.key_vault_id
role_definition_name = "Key Vault Secrets User"
principal_id = azurerm_kubernetes_cluster.aks.key_vault_secrets_provider[0].secret_identity[0].object_id
depends_on = [
azurerm_kubernetes_cluster.aks
]
}
# Additional role assignment for CSI Driver
resource "azurerm_role_assignment" "aks_csi_keyvault" {
scope = var.key_vault_id
role_definition_name = "Key Vault Secrets User"
principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
depends_on = [
azurerm_kubernetes_cluster.aks
]
}Other modules present for this project are: Network, acr, app-gw, keyvault, monitoring, redis, postgreSQL.
Run the below commands to provision all the required resources:
cd environments/dev
terraform init
terraform plan -out=tfplan
terraform apply (Or run: terraform apply -auto-approve) Step 2: Application deployment to Azure Kubernetes Service (AKS)
Our application has a frontend and backend services which we are going to deploy to AKS:
Note that in this implementation, I have used two namespaces – ecomm-frontend and ecomm-backend.
Further enhancement could involve using separate namespaces for each service in production environment (Auth, catalog, orders and payment). This makes it modular, easily managed, secure and scalable.
### Microservices
1. Auth Service
2. Catalog Service
3. Payment Service
4. Orders Service
5. Frontend
You can view and download all the manifest files from Here:
Before applying our Kubernetes configurations, we need build our images and push them to ACR.
Build and Push Docker Images
Login to ACR
# Login to ACR
az acr login --name ecomdevacrYou can go into the folder directly to build each image:
Build the image
docker build -t ecomdevacr.azurecr.io/frontend:v1 .Push to ACR
# Push to ACR
docker push ecomdevacr.azurecr.io/frontend:v1For Services:
cd services/auth
docker build -t ecomdevacr.azurecr.io/auth:v1 .
docker push ecomdevacr.azurecr.io/auth:v1Do same for other service
Or build while pointing to the folder, for example:
docker build -t ecommacr.azurecr.io/frontend:v1 ./frontend
docker build -t ecomm.azurecr.io/orders-service:v1 ./services/orders
Keyvault
Go to Azure Portal to add your env variables in keyvault or you could use CLI to add it from your local machine. Ensure you are authenticated to Azure before addition.
az keyvault secret set --vault-name "ecom-dev-kv216" \
--name "STRIPE-SECRET-KEY" \
--value "<STRIPE_SECRET_KEY>"
az keyvault secret set \
--vault-name ecom-dev-kv216 \
--name "REDIS-CONNECTION" \
--value "your-connection-string"Add ALL the required envs.
Get AKS Credentials
Let us get AKS credentials which retrieves the kubeconfig file, that is, retrieves access credentials for the Kubernetes API server of the specified AKS cluster and allows you to interact with the AKS cluster using Kubectl.
az aks get-credentials \
--resource-group ${RESOURCE_GROUP} \
--name ${AKS_CLUSTER} \
--overwrite-existingApply K8 Configs
# Create namespace
kubectl apply -f k8s/namespace.yaml# Apply individual configs
kubectl apply -f k8s/ServiceAccount.yaml
kubectl apply -f k8s/auth-service.yaml
kubectl apply -f k8s/ingress.yaml# Apply all configs at once
Or to apply all at once
kubectl apply -f .
or the directory
kubectl apply -f K8/Check deployment status
kubectl get deployments -n ecomm-backend
Check pods
kubectl get pods -n ecomm-backend
Check services
kubectl get services -n ecomm-backend
Check ingress
kubectl get ingress -n ecomm-backend
Describe a deployment
kubectl describe deployment auth-service -n ecomm-backend
Additional step to use KeyVault with our cluster
I performed this additional setup to connect AKS with Azure Keyvault to download environment variables from Keyvault into our cluster which will be used by our application.
If you check the terraform scripts for AKS cluster, you will notice that we have enabled CSI driver add-on.
key_vault_secrets_provider {
secret_rotation_enabled = var.enable_secret_rotation
#secret_rotation_interval = "2m" # Optional: Configure rotation interval
}
# Enable OIDC and Workload Identity
oidc_issuer_enabled = true
workload_identity_enabled = true
If you are using this method, you also have to configure ServiceAccount and SecretProviderClass which will map our objects pointing to the keyvault, keyvault name and IDs to enable the communication. Please check the K8 manifest files for the config – you would have applied them in the previous step.
Configure workload identity
az identity federated-credential create --name $FEDERATED_IDENTITY_NAME --identity-name $UAMI --resource-group $RESOURCE_GROUP --issuer ${AKS_OIDC_ISSUER} --subject system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}Ensure to export your variables before running the above commands, for example:
export AKS_OIDC_ISSUER="$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query "oidcIssuerProfile.issuerUrl" -o tsv)"
export SERVICE_ACCOUNT_NAME="workload-identity-sa" or "workload-identity-sa-fe"
export SERVICE_ACCOUNT_NAMESPACE="ecomm-backend" or "ecomm-frontend"
export RESOURCE_GROUP=<Your-resource-group>
export UAMI=azurekeyvaultsecretsprovider-ecom-dev-aks
export FEDERATED_IDENTITY_NAME="aksfederatedidentity"Check if the CSI driver addon is enabled with managed identity:
az aks show -g hub-ecommdev-rg -n ecom-dev-aks --query "addonProfiles.azureKeyvaultSecretsProvider.identity"To confirm that your csi driver-secret store provider is enabled and running:
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)'
NAME READY STATUS RESTARTS AGE
aks-secrets-store-csi-driver-wrwkw 3/3 Running 0 10h
aks-secrets-store-csi-driver-zr2gq 3/3 Running 0 10h
aks-secrets-store-provider-azure-5vdmc 1/1 Running 0 10h
aks-secrets-store-provider-azure-b8dvf 1/1 Running 0 10hCONFIGURE DNS
If you are using Azure DNS, this is a brief summary of how you can set it up.
Retrieve your App GW Public IP, or visit Azure Portal to get it.
Create DNS Zone
az network dns zone create --resource-group hub-ecommdev-rg --name example.co.ukVerify creation
az network dns zone show --resource-group hub-ecommdev-rg --name example.co.ukCreate DNS records using A-records
az network dns record-set a add-record --resource-group hub-ecommdev-rg \
--zone-name "example.co.uk" \
--record-set-name "@" \
--ipv4-address x.x.x.104Also create similar record for your api. subdomain.
Create self-signed certificate or import your SSL certificate (Note that you can use other methods)
az network application-gateway ssl-cert create \
--gateway-name "ecom-dev-appgw" \
--name "ecomm-cert" \
--resource-group hub-ecommdev-rg \
--cert-file "../cert.pfx" \
--cert-password "<your-password>"Go to the site of your domain name host provider and add Azure nameservers in your DNS settings.
After the completion of the setup and there are no errors, all the pods should be running like below output.
Checks
$ kubectl get pods -n ecomm-backend
NAME READY STATUS RESTARTS AGE
auth-service-69f4dc4695-2tt8r 1/1 Running 0 34m
auth-service-69f4dc4695-z8m56 1/1 Running 0 34m
catalog-service-6c66bbd588-gwlgw 1/1 Running 0 20m
catalog-service-6c66bbd588-xmz92 1/1 Running 0 20m
orders-service-8c5f8656c-2k7vb 1/1 Running 0 4h11m
orders-service-8c5f8656c-kpbmk 1/1 Running 0 4h11m
payment-service-76f49d4d9-fgkms 1/1 Running 0 4m30s
payment-service-76f49d4d9-lp9mb 1/1 Running 0 4m30s
$ kubectl get pods -n ecomm-frontend
NAME READY STATUS RESTARTS AGE
frontend-744c4474c5-9lx6r 1/1 Running 0 130m
frontend-744c4474c5-g4v4t 1/1 Running 0 130m
$ kubectl get ingress -n ecomm-frontend
NAME CLASS HOSTS ADDRESS PORTS AGE
frontend-ingress <none> <domain-name> <app-gw-pip> 80 150mTroubleshooting tips
kubectl logs <pod-name> -n ecomm-backend
kubectl get pods -n ecomm-backend
kubectl logs orders-service-b578c578f-5pj68 -n ecomm-backend --previous
kubectl describe pods orders-service-b578c578f-mkrjs -n ecomm-backendCluster-wide resource usage
kubectl top nodes
kubectl top pods --all-namespacesCheck ingress status
kubectl get ingress -ACheck AGIC pods
kubectl get pods -n kube-system -l app=ingress-appgwStep 3: Automated deployment using Jenkins CI/CD
Earlier we provisioned Jenkins server on Google cloud VM instance, logged in and installed suggested plugins.
Next, we will install additional plugins and configure some tools.
Plugins
- Azure credentials
- Azure cli
- Pipeline stage view
- NodeJS
- Docker pipeline
- parameterized trigger
- Image tag parameter
Configure Tools
In Dashboard –> Manage Jenkins –> Tools: Configure git, NodeJS, JDK (JAVA_HOME)
In Dashboard –> Manage Jenkins –> Credentials: Add all the required credentials to enable authentication and to hide our secrets – envs.
Add the following:
- AKS Credentials: (Kind: Azure Service Principal)
- Azure Container Registry Credentials (Kind: Username with password – SP ID/Secret)
- Registry name (Kind: secret)
- Resource group (Kind: secret)
- CSI_CLIENT_ID (Kind: secret)
- TENANT_ID (Kind: secret)
- KeyVault name (Kind: secret)
- AKS_CLUSTER (Kind: secret)
Note: I had to install additional packages (Azure cli, git, terraform, npm) on Jenkins server to support our build, you can include them as part of the installation steps in the start-up script – install_jenkins.sh.
Lets automate the provisioning and deployment process using Jenkins CI/CD solutions
We are going for Pipeline for the sake of practice otherwise you can configure Jenkins webhook in your GitHub to automatically trigger pipeline build when changes are made to the codes/scripts in the repo.
Infrastructure pipeline setup
Go to Dashboards –> New Item –> Enter an item name –> Pipeline –> Ok.
Go to Pipeline –> select “Pipeline script” and paste in content of the Jenkinsfile –> Save.
Now you can manually start the build. As it is a parameterized build, you choose apply to run the scripts and provision resources or choose destroy to delete all resources.
pipeline {
agent any
parameters {
choice(name: 'ACTION', choices: ['apply', 'destroy'], description: 'Select Terraform action')
}
environment {
TERRAFORM_DIR = 'Infrastructure-app-terraform/environments/dev'
AZURE = credentials('aks-cred')
}
stages {
stage('Checkout Infrastructure') {
steps {
checkout([$class: 'GitSCM',
branches: [[name: '*/master']],
userRemoteConfigs: [[
url: 'https://github.com/<user-name>/ecomm-hub-infra.git',
credentialsId: 'github'
]]
])
}
}
stage('Terraform Operations') {
steps {
withCredentials([azureServicePrincipal('aks-cred')]) {
dir(TERRAFORM_DIR) {
script {
// Set Azure credentials for all Terraform operations
def azureEnv = """
export ARM_CLIENT_ID=\${AZURE_CLIENT_ID}
export ARM_CLIENT_SECRET=\${AZURE_CLIENT_SECRET}
export ARM_SUBSCRIPTION_ID=\${AZURE_SUBSCRIPTION_ID}
export ARM_TENANT_ID=\${AZURE_TENANT_ID}
"""
// Terraform Init
stage('Terraform Init') {
sh """
${azureEnv}
terraform init
"""
}
if (params.ACTION == 'destroy') {
// Terraform Destroy
stage('Terraform Destroy') {
sh """
${azureEnv}
terraform plan -destroy -out=tfplan
terraform apply -auto-approve tfplan
"""
}
} else {
// Terraform Plan
stage('Terraform Plan') {
sh """
${azureEnv}
terraform plan -out=tfplan
"""
}
// Terraform Apply
stage('Terraform Apply') {
//sh """
//${azureEnv}
//terraform apply -auto-approve tfplan
//"""
sh "ls -lrth"
}
}
}
}
}
}
}
}
post {
always {
sh '''
# Logout from Azure
#az logout
# Clear Azure CLI cache
rm -rf ~/.azure
'''
cleanWs()
}
success {
echo 'Infrastructure deployment successful!'
}
failure {
echo 'Infrastructure deployment failed!'
}
}
}
Application pipeline setup
Go to Dashboards > New Item > Enter an item name > Pipeline > Ok
Go to Pipeline –> select Pipeline script and paste in content of the Jenkinsfile –> Save.
Now, manually start the build.
Our pipeline finally got completed after dealing with some Jenkins errors.
Conclusion
In this blog, we worked on building and deploying a cloud-native e-commerce platform using Azure Kubernetes Service (AKS). By leveraging Terraform for infrastructure provisioning and Jenkins CI/CD for deployment automation, we successfully demonstrated the power of combining modern DevOps practices with scalable cloud solutions.
We have explored how to streamline the deployment of a microservices-based e-commerce application together with integrated CI/CD pipelines and robust infrastructure as code, this project highlights how to achieve operational efficiency, scalability, and maintainability in cloud-native application development.
