Azure Cosmos DB for MongoDB: Backup and Restore Databases using MongoDB Command Line Database Tools

In this post let's see how we can backup and restore databases in an Azure Cosmos DB for MongoDB account using MongoDB Command Line Database Tools.

I am using Windows and the first step is to download MongoDB Command Line Database Tools. As of today, the version is 100.6.1 and I have downloaded the zip version. You can even download the msi and install the tools.

Backup

You can run the mongodump.exe from the command prompt providing the connection string of your Azure Cosmos DB for MongoDB account, the database you want to backup and an output path to write the backup files to.

mongodump.exe --uri "<connectionString>" --db <databaseName> --out <backupDirectory> 

mongodump.exe

Restore

You can run the mongorestore.exe from the command prompt providing the connection string of your Azure Cosmos DB for MongoDB account, a target database name, and the directory containing the backup files of the database.

mongorestore.exe --uri "<connectionString>" --db <databaseName> --dir <backupDirectory\databaseName>

mongorestore.exe

I am yet to confirm whether this approach is recommended, but so far I haven't faced any issues with the restored database. Collections and their documents count look great.

If you want to copy a database from one Azure Cosmos DB for MongoDB account to other, you can try this out. I have tried, Copy data to or from Azure Cosmos DB for MongoDB using Azure Data Factory or Synapse Analytics, but unfortunately, it didn't work out well due to some issues, hence tried MongoDB Command Line Database Tools.

   Issue: Copy Activity: Copying data from and to Azure Cosmos DB for MongoDB: Failing due to incorrect Data Type 
   Issue: Copy Activity: Copying data from and to Azure Cosmos DB for MongoDB: "The retrieved type of data JObject is not supported yet"

Hope this helps. 

Please don't forget to leave a comment on whether this approach worked for you or not.

Happy Coding.

Regards,
Jaliya

____________________________________________________________________________________________________________________________

Update on 2023/02/08:

I have received some recommendations from Product Group: Azure Cosmos DB.

• Native MongoDB tools would be recommended for smaller workloads
• Azure Data Factory (ADF) for medium (<1TB)
• Spark if it’s a large dataset (>1 TB)

Azure DevOps Pipelines: Install npm Packages from an External Private Package Registry

In this post let's see how we can Install npm Packages from an external private package registry in an Azure DevOps Pipeline.

The first step is adding a Service Connection for Connection Type: npm. You can do it by going into your Project Settings -> Service Connections and then New service connection and choosing npm.

New npm service connection

Click on Next.

New npm service connection

Here you can put the details, give the Service Connection a name and save it (note down the name as we are going to need it in a future step). In my case, I want to consume npm packages from Form.io private package repository and I have only the Username and the Password. So I have filled those up.

Next, we need to add/update the .npmrc file and specify the private package registry URL.

# Some other Azure Artifact Packages
registry=https://pkgs.dev.azure.com/{some-organization}/{some-project}/_packaging/{some-feed}/npm/registry/
 
# Form.io Premium Packages
@formio:registry=https://pkg.form.io/
 
always-auth=true

And now we can modify the DevOps pipeline to install the private packages.

trigger:
  branches:
    include:
      - main
      - feature/*
      - version/*
 
pool:
  vmImage: 'ubuntu-latest'
 
name: $(Build.SourceBranchName).$(Build.BuildId)
 
steps:
  - task: npmAuthenticate@0
    displayName: "Authenticate for FormIo Premium Packages"
    inputs:
      workingFile: .npmrc
      customEndpoint: FormIo-Packages # Important: Name of the npm Service Connection created

  - task: NodeTool@0
    displayName: 'Install Node.js'
    inputs:
      versionSpec: '16.x'
      checkLatest: true
 
  - task: Npm@1
    displayName: 'Install Formio Premium Packages'
    inputs:
      command: custom
      workingDir: $(RootPath)
      verbose: false
      customCommand: 'install @formio/premium --registry https://pkg.form.io --force'

Here the important step is using npmAuthenticate@0 task to provide npm credentials to the .npmrc file for the scope of the build. So it will get used when we are doing the npm install.

And that should do.

Hope this helps.

Happy Coding.

Regards,

Jaliya

C# 11.0: Newlines in String Interpolation Expressions

In this post let's have a look at another feature that is available with C# 11.0. And that is support for Newlines in String Interpolation Expressions.

Consider the following example code written in C# 10.0.

int age = 60;
string ageCategory = age switch
{
    < 1 => "Infant",
    < 12 => "Child",
    < 17 => "Adolescent",
    < 65 => "Adult",
    _ => "Older adult"
};
string message = $"Based on the Age of {age}, you are a(n) {ageCategory}.";

Here it would have been nice if I can include the logic to determine the age category within the interpolated string. But Newlines inside a non-verbatim interpolated string are not supported in C# 10.0.

With C# 11.0, we now have the support for Newlines in String Interpolation Expressions. So I can simplify the code as follows.

int age = 60;
string message = $"Based on the Age of {age}, you are a(n) {age switch
{
    < 1 => "Infant",
    < 12 => "Child",
    < 17 => "Adolescent",
    < 65 => "Adult",
    _ => "Older adult"
}}.";

Isn't it nice? There is no pleasure like having readable simplified code.

Read more of C# features here.
   What's new in C# 11

Happy Coding.

Regards,

Jaliya

Visual Studio 2022: Sending HTTP Requests with .http Files

Do you know Visual Studio has this nice feature to send HTTP Requests using .http files. It's actually pretty neat.

We just have to create a file with .http extension.

.http Files in Action

We can do all the methods and the output is displaying nicely.

Disclaimer: I am not sure when this feature got added, I am on Microsoft Visual Studio Enterprise 2022 (64-bit) - Version 17.5.0 Preview 4.0 and I don't have a specific extension installed in order to enable this functionality.

Hope this helps!

Happy Coding.

Regards,

Jaliya

Visual Studio 2022 Version 17.5 Preview 2: Text Visualizer Improvements

In this post let's explore some nice improvements that got added to Text Visualizer in Visual Studio. This got shipped with Visual Studio 2022 Version 17.5 Preview 2. 

I actually found this accidentally and it's so handy!

Say I have a string variable that holds a JWT token and I want to decode it for some debugging purpose. Usually what I would do is copy-pasting the token into https://jwt.io or https://jwt.ms or some external JWT decoding tool. What if we can see the decoded token without leaving Visual Studio? And that's exactly what the new Text Visualizer brings to the table.

Say I want to see the decoded token of the following.

Debugging a string

I need to click on the View button. And then I am presented with the following.

Text Visualizer

I can just select JWT Decode and just like that I can see the decoded token. 

Along that, you can see there are more options to manipulate the string such as Base64 Encode & Decode,  Url Encode & Decode etc.

Isn't that great? I just love this. Remember, if you want to try this out, you need to install the latest Visual Studio 2022 Preview.

Hope this helps.

Happy Coding.

Regards,

Jaliya

Azure Durable Functions in Azure Functions .NET Isolated Worker Process

Azure Durable Functions now supports .NET 7.0 running in the Isolated worker process. It's still in it's preview stage, but it's super exciting.

In this post, let's see how we can get ourselves started in Azure Durable Functions in Azure Functions .NET Isolated Worker Process.

First, after creating a Function App that targets .NET 7 Isolated functions worker, we need to install the NuGet package: Microsoft.Azure.Functions.Worker.Extensions.DurableTask. Make sure you have checked Included prerelease checkbox if you are trying to install the package through Visual Studio NuGet Package Manager.

Microsoft.Azure.Functions.Worker.Extensions.DurableTask

After installing the package, I have updated Function1.cs as follows adding a simple Durable Function.

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.Extensions.Logging;
 
namespace FunctionApp1;
 
public class Function1
{
    [Function(nameof(TriggerHello))]
    public async Task<HttpResponseData> TriggerHello([HttpTrigger(AuthorizationLevel.Anonymous, "get")] HttpRequestData req,
        [DurableClient] DurableClientContext durableContext,
        FunctionContext executionContext)
    {
        ILogger logger = executionContext.GetLogger(nameof(Function1));
 
        string instanceId = await durableContext.Client.ScheduleNewOrchestrationInstanceAsync(nameof(HelloPersonOrchestrator));
        logger.LogInformation("Created new orchestration with instance ID = {instanceId}", instanceId);
 
        return durableContext.CreateCheckStatusResponse(req, instanceId);
    }
 
    [Function(nameof(HelloPersonOrchestrator))]
    public static async Task<string> HelloPersonOrchestrator([OrchestrationTrigger] TaskOrchestrationContext context,
        FunctionContext executionContext)
    {
        ILogger logger = executionContext.GetLogger(nameof(HelloPersonOrchestrator));
 
        string result = await context.CallActivityAsync<string>(nameof(SayHello), "John Doe") + " ";
        result += await context.CallActivityAsync<string>(nameof(SayHello), "Jane Doe") + " ";
        result += await context.CallActivityAsync<string>(nameof(SayHello), "Joe Bloggs") + " ";
        result += await context.CallActivityAsync<string>(nameof(SayHello), "Fred Bloggs");
 
        logger.LogInformation("HelloPersonOrchestrator says: {output}", result);
 
        return result;
    }
 
    [Function(nameof(SayHello))]
    public static string SayHello([ActivityTrigger] string name, FunctionContext executionContext)
    {
        ILogger logger = executionContext.GetLogger(nameof(SayHello));
        logger.LogInformation("Saying hello to {name}", name);
        return $"Hello {name}!";
    }
}

And then we can call the HTTP  trigger to start the orchestration. And it works like a charm.

Output

Can't wait for this to go under GA.

Do try this out and if you see any issues, please do not hesitate to log an issue here: microsoft/durabletask-dotnet

Hope this helps.

Happy Coding.

Regards,

Jaliya

ASP.NET Core Web API: Exception Handling

In this post, let's see how we can provide a common approach for handling exceptions in ASP.NET Core Web APIs in the Development environment as well as in Production environments.

It's quite easy, basically, we can introduce UseExceptionHandler Middleware to handle exceptions.

Consider the following code.

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
 
// Add services to the container.
builder.Services.AddControllers();
 
WebApplication app = builder.Build();
 
// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/error-development");
}
else
{
    app.UseExceptionHandler("/error");
}
 
app.UseHttpsRedirection();
 
app.UseAuthorization();
 
app.MapControllers();
 
app.Run();

Here I have added UseExceptionHandler passing in different routes based on the environment. Now we need to define controller actions to respond to /error-development and /error routes.

using Microsoft.AspNetCore.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using System.Net;
 
namespace WebApplication1.Controllers;
 
[ApiController]
[ApiExplorerSettings(IgnoreApi = true)]
public class ErrorController : ControllerBase
{
    private readonly IHostEnvironment _hostEnvironment;
 
    public ErrorController(IHostEnvironment hostEnvironment)
    {
        _hostEnvironment = hostEnvironment;
    }
 
    [Route("/error-development")]
    public IActionResult HandleErrorDevelopment()
    {
        if (!_hostEnvironment.IsDevelopment())
        {
            return NotFound();
        }
 
        IExceptionHandlerFeature exceptionHandlerFeature = HttpContext.Features.Get<IExceptionHandlerFeature>()!;
 
        if (exceptionHandlerFeature == null)
        {
            return Problem(
                title: $"'{nameof(IExceptionHandlerFeature)}' not found.");
        }
 
        return exceptionHandlerFeature.Error switch
        {
            NotImplementedException notImplementedException => Problem(
                title: notImplementedException.Message,
                detail: notImplementedException.StackTrace,
                statusCode: (int)HttpStatusCode.NotImplemented),
 
            _ => Problem(
                title: exceptionHandlerFeature.Error.Message,
                detail: exceptionHandlerFeature.Error.StackTrace)
        };
    }
 
    [Route("/error")]
    public IActionResult HandleError() => Problem();
}

Note: The actions aren't attributed with HttpVerbs and the controller is attributed with [ApiExplorerSettings(IgnoreApi = true)] to exclude from OpenAPI specification (if there's any).

For the Development environment, based on the type of the exception, I am returning different statusCodes and I am including the StackTrace to troubleshoot the issue easily. 

For an example, consider the following action.

[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get()
{
    throw new NotImplementedException("Not implemented.");
}

If we call the above action when running on a Development environment, I will be getting a response like below. It's the standard RFC 7807-compliant Problem Detail.

Development: StatusCode: 501

And now if I change the endpoint to throw a different exception,

[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get()
{
    throw new Exception("Something happened.");
}

I am getting the default Internal Server Error status code with the StackTrace.

Development: StatusCode: 500

And when in a production environment, we just get the response hiding internal details.

Production: StatusCode: 500

More read:
   Handle errors in ASP.NET Core web APIs

Hope this helps.

Happy Coding.

Regards,

Jaliya

ASP.NET Core: Suppress Implicit Required Model Validation For Non Nullable Reference Types

In this post, let's see how we can disable implicit required model validation for non-nullable reference types when the nullable context is enabled in an ASP.NET Core Web Application.

Let's consider the following sample.

#nullable enable
 
using Microsoft.AspNetCore.Mvc;
 
WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
 
// Register Services.
builder.Services.AddControllers();
 
WebApplication app = builder.Build();
 
// Configure the HTTP request pipeline.
app.UseHttpsRedirection();
 
app.MapControllers();
 
app.Run();

[ApiController]
[Route("[controller]")]
public class EmployeesController : ControllerBase
{
    [HttpPost]
    public ActionResult<Employee> Create([FromBody] Employee employee)
    {
        return employee;
    }
}
 
public class Employee
{
    public string FirstName { get; set; }
 
    public string LastName { get; set; }
 
    public Employee(string firstName, string lastName)
    {
        FirstName = firstName;
        LastName = lastName;
    }
}

Here if we try to post a payload with null values for reference types that are not marked with nullable, ASP.NET Core by default is doing a model validation and throwing an error.

For example, here if we post a payload without setting the value for LastName, I am getting the following error.

ASP.NET Core Model Validation

But there are times, we need to offload the validation to a separate module. In that case, we can easily suppress this validation as follows.

builder.Services.AddControllers(x =>
{
    x.SuppressImplicitRequiredAttributeForNonNullableReferenceTypes = true;
});

And then, we won't be seeing that validation anymore.

ASP.NET Core: Suppress Implicit Required Attribute For Non Nullable Reference Types

Hope this helps.

Happy Coding.

Regards,

Jaliya

JsonExtensionDataAttribute in System.Text.Json.Serialization

In this post let's have a look at this nice attribute JsonExtensionDataAttribute that's available in System.Text.Json.Serialization Namespace.

I had a requirement where I have an integration API that connects a client application with a third-party API. The client application is sending some JSON that needs to be sent to the third-party API as is, and at the same time, within the integration API, I need to intercept and read some values in the JSON. The JSON schema can be huge, and the Integration API doesn't really care about the full JSON schema. So how do we define a POCO class having the properties I really care about and let other properties get deserialized/serialized without any data loss.

Let's go by an example.

Consider the following JSON schema.

string jsonString =
    """
    {
        "_id": "57d0a0a676f943a4007e1525",
        "modified": "2022-09-07T23:20:06.949Z",
        "title": "Register",
        "display": "form",
        "type": "form",
        "name": "register",
        "path": "register",
        "components": [
            {
                "label": "First Name",
                "key": "firstName",
                "type": "textfield",
                "placeholder": "",
                "prefix": "",
                "suffix": "",
                "multiple": false
            },
            {
                "label": "Last Name",
                "key": "firstName",
                "type": "textfield",
                "placeholder": "",
                "prefix": "",
                "suffix": "",
                "multiple": false
            }
        ],
        "tags": []
    }
    """;

Now let's say, I only care about _id, name and type properties.

I can create a POCO class, something like below.

public class Form
{
    [JsonPropertyName("_id")]
    public string Id { get; set; }
 
    public string Name { get; set; }
 
    public string Type { get; set; }
}

But then the issue is, upon deserialization, I am going to lose some data. And this is where JsonExtensionDataAttribute comes in handy.

public class Form
{
    [JsonPropertyName("_id")]
    public string Id { get; set; }
 
    public string Name { get; set; }
 
    public string Type { get; set; }
 
    [JsonExtensionData]
    public Dictionary<string, object> JsonExtensionData { get; set; }
}

Now when I deserialized the jsonString to Form what's going to happen is, the properties with matching members will get their values and the properties that do not have a matching member will get added to the JsonExtensionData dictionary.

JsonSerializerOptions jsonSerializerOptions = new()
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase
};
 
Form form = JsonSerializer.Deserialize<Form>(jsonString, jsonSerializerOptions);

Here if I have a look at the form, it's going to look like this.

Deserialization

You can see that properties with matching members got their values and the rest went into the Dictionary.

And now when I serialize the form, the values in the Dictionary are written back as expected. 

string serializedString = JsonSerializer.Serialize(form, jsonSerializerOptions);
//{
//    "_id": "57d0a0a676f943a4007e1525",
//    "name": "register",
//    "type": "form",
//    "modified": "2022-09-07T23:20:06.949Z",
//    "title": "Register",
//    "display": "form",
//    "path": "register",
//    "components": [
//        {
//            "label": "First Name",
//            "key": "firstName",
//            "type": "textfield",
//            "placeholder": "",
//            "prefix": "",
//            "suffix": "",
//            "multiple": false
//        },
//        {
//            "label": "Last Name",
//            "key": "firstName",
//            "type": "textfield",
//            "placeholder": "",
//            "prefix": "",
//            "suffix": "",
//            "multiple": false
//        }
//    ],
//    "tags": []
//}

Isn't it nice?

Happy Coding.

Regards,
Jaliya

C# 11.0: File-local Types

In this post let's have a look at one of the newest additions to C#, which is the file access modifier. 

For types that are declared with file access modifier, its visibility or scope is only limited to the file in which it's declared.

Consider the below code.

File1.cs

namespace MyNamespace;
 
file class MyClass
{
    public static void MyMethod()
    {
        //
    }
}

File2.cs

namespace MyNamespace;
 
file class MyClass
{
    public static void MyMethod()
    {
        //
    }
}

Here I have 2 classes with the same name and within the same namespace but in different files. With C# 11.0, this will build just fine. Prior to C# 11.0, we weren't able to do this.

file keyword can be applied only to the following types.

• class
• struct
• interface
• enum
• delegate
• record
• record struct

And it's allowed only at the top-level type. 

public class MyClass
{
    // Error: Not allowed as MyOtherClass is a nested type
    file class MyOtherClass
    { 
    
    }
}

It's very unlikely we will be using this file access modifier during business application development. But wanted to share it, because in case you see it, you know what it is.

More read:
   file (C# Reference)
   File-local types

Hope this helps.

Happy Coding.

Regards,
Jaliya