In this post let's see how we can run MongoDB as a StatefulSet in AKS and
maintain its storage outside of the pods so data is safe from the ephemeral nature of pods. We are going to maintain the database
storage in File Shares in an Azure Storage Account.
Preparation
In Azure, I already have an AKS created.
And then I have created a simple ASP.NET Core Minimal API which reads and
writes data from/to a MongoDB. We will use this API to test the functionality.
using KubeStorage.Mongo.Api.Models;
using KubeStorage.Mongo.Api.Services;
using Microsoft.AspNetCore.Http.HttpResults;
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
builder.Services.Configure<CustomerDatabaseSettings>(builder.Configuration.GetSection("CustomersDatabase"));
builder.Services.AddSingleton<CustomersService>();
WebApplication app = builder.Build();
app.UseHttpsRedirection();
app.MapGet("/customers/{id}", async Task<Results<Ok<Customer>, NotFound>> (CustomersService customersService, string id) =>
{
Customer? customer = await customersService.GetAsync(id);
if (customer == null)
{
return TypedResults.NotFound();
}
return TypedResults.Ok(customer);
});
app.MapGet("/customers", async (CustomersService customersService) =>
{
return TypedResults.Ok(await customersService.GetAsync());
});
app.MapPost("/customers", async (CustomersService customersService, Customer customer) =>
{
await customersService.CreateAsync(customer);
return TypedResults.Created($"/customers/{customer.Id}", customer);
});
app.Run();
I have containerized this and have it available as a Docker image.
Now let's run MongoDB as a StatefulSet in AKS, maintain its storage in Azure File Shares, and use the above API to
consume the database.
Start
First, let's start by creating a K8s namespace for the demo.
apiVersion: v1
kind: Namespace
metadata:
name: demo-mongodb
Now let's create a K8s StorageClass (SC). This will be used to dynamically provision storage.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: sc-azurefile-mongo
provisioner: file.csi.azure.com # replace with "kubernetes.io/azure-file" if aks version is less than 1.21
reclaimPolicy: Retain # default is Delete
allowVolumeExpansion: true
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Standard_LRS
location: eastus2
And another important thing to note here, SCs are cluster-scoped resources.
The next step is creating a StatefulSet and its wrapper Service for MongoDB.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongo
namespace: demo-mongodb
spec:
selector:
matchLabels:
app: mongodb
serviceName: mongodb
replicas: 3
template:
metadata:
labels:
app: mongodb
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongodb
image: mongo:latest
command:
- mongod
- "--bind_ip_all"
- "--replSet"
- rs0
ports:
- containerPort: 27017
volumeMounts:
- name: vol-azurefile-mongo
mountPath: /data/db
volumeClaimTemplates:
- metadata:
name: vol-azurefile-mongo
spec:
storageClassName: sc-azurefile
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: mongodb
namespace: demo-mongodb
spec:
clusterIP: None
ports:
- name: tcp
port: 27017
protocol: TCP
selector:
app: mongodb
Here, I am creating a StatefulSet of mongo:latest with 3 replicas. One of the advantages of StatefulSet is, for a StatefulSet with n replicas, when Pods are being deployed, they are created sequentially, ordered from
{0..n-1}. And then I am setting up a ReplicaSet named rs0 by running the command section. The important section is the volumeClaimTemplates, where we will create storage using PersistentVolumes provisioned by our
storage class sc-azurefile.
And then I have a headless service wrapping the MongoDB StatefulSet.
Now let's apply all these configurations to our AKS.
|
| k apply |
And let's make sure everything is created.
|
|
k get all --namespace demo-mongodb
|
And now if I check the
node resource group for my AKS, I can see
a new Storage Account is provisioned and Azure Files are created for each
replica.
|
|
Azure File Shares Created For Each Replica
|
|
| MongoDB Files |
Now since MongoDB replicas are running, let's configure the ReplicaSet. I am
shelling into the primary relica.
kubectl exec --namespace demo-mongodb mongo-0 --stdin --tty -- mongosh
|
| Mongo Shell |
Now run the following set of commands in the Mongo Shell.
rs.initiate()
var cfg = rs.conf()
cfg.members[0].host="mongo-0.mongodb:27017"
rs.reconfig(cfg)
rs.add("mongo-1.mongodb:27017")
rs.add("mongo-2.mongodb:27017")
rs.status()
And note here, hostnames must follow below.
<mongo-pod-name>.<mongodb-headless-service>:<mongodb-port>
And make sure rs.status() is all successful. Something like below.
rs0 [direct: primary] test> rs.status()
{
...
members: [
{
_id: 0,
name: 'mongo-0.mongodb:27017',
health: 1,
state: 1,
stateStr: 'PRIMARY',
...
},
{
_id: 1,
name: 'mongo-1.mongodb:27017',
health: 1,
state: 2,
stateStr: 'SECONDARY',
syncSourceHost: 'mongo-0.mongodb:27017',
...
},
{
_id: 2,
name: 'mongo-2.mongodb:27017',
health: 1,
state: 2,
stateStr: 'SECONDARY',
...
}
],
ok: 1,
...
}
Let's exit from the Shell.
Finally, let's create a deployment and a service for our test API.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-bridge-mongodb
namespace: demo-mongodb
spec:
selector:
matchLabels:
app: api-bridge-mongodb
template:
metadata:
labels:
app: api-bridge-mongodb
spec:
containers:
- name: api-bridge-mongodb
image: myacr.azurecr.io/demo/mongodb/api:dev
imagePullPolicy: Always
env:
- name: CustomersDatabase__ConnectionString
value: mongodb://mongo-0.mongodb:27017,mongo-1.mongodb:27017,mongo-2.mongodb:27017?replicaSet=rs0
---
apiVersion: v1
kind: Service
metadata:
name: api-bridge-mongodb
namespace: demo-mongodb
spec:
type: LoadBalancer
ports:
- port: 5051
targetPort: 80
protocol: TCP
selector:
app: api-bridge-mongodb
---
Note here for our test API, I am using the MongoDB Cluster ConnectionString.
Now let's test things out. I am getting the IP for our Test API.
|
|
k get services --namespace demo-mongodb
|
And let's hit the endpoints.
|
| GET: /customers |
No errors at least. Let's create a Customer.
|
| POST: /customers |
And now let's get all Customers.
Happy Coding.
Regards,
Jaliya
