Jaliya's Blog: Azure Kubernetes Service: Running MongoDB as a StatefulSet with Azure Files as PersistentVolume

In this post let's see how we can run MongoDB as a StatefulSet in AKS and maintain its storage outside of the pods so data is safe from the ephemeral nature of pods. We are going to maintain the database storage in File Shares in an Azure Storage Account.

Preparation

In Azure, I already have an AKS created.

And then I have created a simple ASP.NET Core Minimal API which reads and writes data from/to a MongoDB. We will use this API to test the functionality.

using KubeStorage.Mongo.Api.Models;
using KubeStorage.Mongo.Api.Services;
using Microsoft.AspNetCore.Http.HttpResults;
 
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
 
builder.Services.Configure<CustomerDatabaseSettings>(builder.Configuration.GetSection("CustomersDatabase"));
builder.Services.AddSingleton<CustomersService>();
 
WebApplication app = builder.Build();
 
app.UseHttpsRedirection();
 
app.MapGet("/customers/{id}", async Task<Results<Ok<Customer>, NotFound>> (CustomersService customersService, string id) =>
{
    Customer? customer = await customersService.GetAsync(id);
    if (customer == null)
    {
        return TypedResults.NotFound();
    }
 
    return TypedResults.Ok(customer);
});
 
app.MapGet("/customers", async (CustomersService customersService) =>
{
    return TypedResults.Ok(await customersService.GetAsync());
});
 
app.MapPost("/customers", async (CustomersService customersService, Customer customer) =>
{
    await customersService.CreateAsync(customer);
 
    return TypedResults.Created($"/customers/{customer.Id}", customer);
});
 
app.Run();

I have containerized this and have it available as a Docker image.

Now let's run MongoDB as a StatefulSet in AKS, maintain its storage in Azure File Shares, and use the above API to consume the database.

Start

First, let's start by creating a K8s namespace for the demo.

apiVersion: v1
kind: Namespace
metadata:
  name: demo-mongodb

Now let's create a K8s StorageClass (SC). This will be used to dynamically provision storage.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: sc-azurefile-mongo
provisioner: file.csi.azure.com # replace with "kubernetes.io/azure-file" if aks version is less than 1.21
reclaimPolicy: Retain # default is Delete
allowVolumeExpansion: true
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=0
  - gid=0
  - mfsymlinks
  - cache=strict
  - actimeo=30
parameters:
  skuName: Standard_LRS
  location: eastus2

Here you can customize dynamic provisioning parameters and those are listed here: Create and use a volume with Azure Files in Azure Kubernetes Service (AKS): Dynamic provisioning parameters.

And another important thing to note here, SCs are cluster-scoped resources.

The next step is creating a StatefulSet and its wrapper Service for MongoDB.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
  namespace: demo-mongodb
spec:
  selector:
    matchLabels:
      app: mongodb
  serviceName: mongodb
  replicas: 3
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongodb
          image: mongo:latest
          command:
            - mongod
            - "--bind_ip_all"
            - "--replSet"
            - rs0
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: vol-azurefile-mongo
              mountPath: /data/db
  volumeClaimTemplates:
    - metadata:
        name: vol-azurefile-mongo
      spec:
        storageClassName: sc-azurefile
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
 
---
 
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  namespace: demo-mongodb
spec:
  clusterIP: None
  ports:
    - name: tcp
      port: 27017
      protocol: TCP
  selector:
    app: mongodb

Here, I am creating a StatefulSet of mongo:latest with 3 replicas. One of the advantages of StatefulSet is, for a StatefulSet with n replicas, when Pods are being deployed, they are created sequentially, ordered from {0..n-1}. And then I am setting up a ReplicaSet named rs0 by running the command section. The important section is the volumeClaimTemplates, where we will create storage using PersistentVolumes provisioned by our storage class sc-azurefile.

And then I have a headless service wrapping the MongoDB StatefulSet.

Now let's apply all these configurations to our AKS.

k apply

And let's make sure everything is created.

k get all --namespace demo-mongodb

And now if I check the node resource group for my AKS, I can see a new Storage Account is provisioned and Azure Files are created for each replica.

Azure File Shares Created For Each Replica

MongoDB Files

Now since MongoDB replicas are running, let's configure the ReplicaSet. I am shelling into the primary relica.

kubectl exec --namespace demo-mongodb mongo-0 --stdin --tty  -- mongosh

Mongo Shell

Now run the following set of commands in the Mongo Shell.

rs.initiate()
var cfg = rs.conf()
cfg.members[0].host="mongo-0.mongodb:27017"
rs.reconfig(cfg)
rs.add("mongo-1.mongodb:27017")
rs.add("mongo-2.mongodb:27017")
rs.status()

And note here, hostnames must follow below.

<mongo-pod-name>.<mongodb-headless-service>:<mongodb-port>

And make sure rs.status() is all successful. Something like below.

rs0 [direct: primary] test> rs.status()
{
  ...
  members: [
    {
      _id: 0,
      name: 'mongo-0.mongodb:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
      ...
    },
    {
      _id: 1,
      name: 'mongo-1.mongodb:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      syncSourceHost: 'mongo-0.mongodb:27017',
      ...
    },
    {
      _id: 2,
      name: 'mongo-2.mongodb:27017',
      health: 1,
      state: 2,
      stateStr: 'SECONDARY',
      ...
    }
  ],
  ok: 1,
  ...
}

Let's exit from the Shell.

Finally, let's create a deployment and a service for our test API.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-bridge-mongodb
  namespace: demo-mongodb
spec:
  selector:
    matchLabels:
      app: api-bridge-mongodb
  template:
    metadata:
      labels:
        app: api-bridge-mongodb
    spec:
      containers:
        - name: api-bridge-mongodb
          image: myacr.azurecr.io/demo/mongodb/api:dev
          imagePullPolicy: Always
          env:
            - name: CustomersDatabase__ConnectionString
              value: mongodb://mongo-0.mongodb:27017,mongo-1.mongodb:27017,mongo-2.mongodb:27017?replicaSet=rs0
 
---
 
apiVersion: v1
kind: Service
metadata:
  name: api-bridge-mongodb
  namespace: demo-mongodb
spec:
  type: LoadBalancer
  ports:
    - port: 5051
      targetPort: 80
      protocol: TCP
  selector:
    app: api-bridge-mongodb
    
---