Ingressing Custom Text Log Files from an Azure Virtual Machine into Log Analytics

I recently had a requirement where I wanted to see some logs in an Azure Virtual Machine, but didn't want to SSH into the VM to see the logs. Instead, it would have been nice if I could see those in a Log Analytics workspace (LAW) in Azure. This gives a lot of advantages, like I can set up alerts on those logs if I want to.

So in this post, let's see how we can ingress some logs that is being written into some custom log file in an Azure Virtual Machine into LAW.

Assuming we already have a LAW created, the first step is creating a custom table there to ingress the logs into.

# Connect-AzAccount
$tableName = "<TableName>_CL"
$tableParams = @"
{
    "properties": {
        "schema": {
            "name": "$tableName",
            "columns": [
                {
                    "name": "TimeGenerated",
                    "type": "DateTime"
                },
                {
                    "name": "RawData",
                    "type": "String"
                },
                {
                    "name": "FilePath",
                    "type": "String"
                },
                {
                    "name": "Computer",
                    "type": "String"
                }
            ]
        }
    }
}
"@
 
Invoke-AzRestMethod `
    -Path "/subscriptions/<SUBSCRIPTION_ID>/resourcegroups/<RESOURCE_GROUP>/providers/microsoft.operationalinsights/workspaces/<LAW_NAME>/tables/$($tableName)?api-version=2021-12-01-preview" `
    -Method PUT `
    -payload $tableParams

Here I am using a default template as for the demo purposes I am not doing any transformations.

Make sure the table is created in your LAW.

Log Analytics Workspace

Next, I am going to Azure Monitor (search for Monitor in Azure Portal) and creating a Data Collection Endpoint (DCE).

Create data collection endpoint

It's pretty straightforward.

Now from Azure Monitor, I am creating a Data Collection Rule (DCR).

Create Data Collection Rule

My VM is on Linux so I am selecting Linux as the Platform Type, and I am selecting the DCE I created in the previous step as the Data Collection Endpoint.

Next on Resources, I am adding a new Resource and selecting the target VM and the DCE.

Create Data Collection Rule: Resources

Now going to Collection and deliver tab and selecting a Data Source of type Custom Text Logs.

Add data source: Custom Text Logs

Add data source: Custom Text Logs

Here, for the File pattern, I am giving the path of the file where I need the logs ingested from and for the Table name, giving the name of LAW table I created before. And for the Transform, I am leaving it as it is.

Now configuring the destination as Azure Monitor Logs as follows.

Add data source: Custom Text Logs

And that's it. I am creating the DCR and Azure will deploy Azure Monitor Agent to the target VM and configure the ingestion.

And once deployment is completed, after some time, I can see the logs in LAW.

Ingested Logs

More read:
   Data collection rules (DCRs) in Azure Monitor
   Collect data with Azure Monitor Agent
   Troubleshooting guidance for the Azure Monitor agent on Linux virtual machines and scale sets

Hope this helps.

Happy Coding.

Regards,

Jaliya