Monitor and Alert Data Factory by using Azure Monitor
Microsoft DP-200 exam is retired. A new replacement exam Data Engineering on Microsoft Azure (DP-203) is available.
In this we will learn about how to monitor and alert Data Factory by using Azure monitor. However, Monitors provide data to help ensure that your applications stay up and running in a healthy state. Monitors also help you avoid potential problems and troubleshoot past ones. You can use monitoring data to gain deep insights about your applications.
Further, Azure Monitor provides base-level infrastructure metrics and logs for most Azure services. Azure diagnostic logs are emitted by a resource and provide rich, frequent data about the operation of that resource. Azure Data Factory (ADF) can write diagnostic logs in Azure Monitor.
Keeping Azure Data Factory metrics and pipeline-run data
Data Factory stores pipeline-run data for only 45 days. Use Azure Monitor if you want to keep that data for a longer time. With Monitor, you can route diagnostic logs for analysis to multiple different targets.
- Firstly, Storage Account: Save your diagnostic logs to a storage account for auditing or manual inspection. You can use the diagnostic settings to specify the retention time in days.
- Secondly, Event Hub: Stream the logs to Azure Event Hubs. The logs become input to a partner service/custom analytics solution like Power BI.
- Then, Log Analytics: Analyze the logs with Log Analytics. The Data Factory integration with Azure Monitor is useful in the following scenarios:
- You want to write complex queries on a rich set of metrics that are published by Data Factory to Monitor.
- Or, you want to monitor across data factories. You can route data from multiple data factories to a single Monitor workspace.
Configure diagnostic settings and workspace
Create or add diagnostic settings for your data factory.
- Firstly, in the portal, go to Monitor. Select Settings > Diagnostic settings.
- Secondly, select the data factory for which you want to set a diagnostic setting.
- Thirdly, if no settings exist on the selected data factory, you’re prompted to create a setting. Select Turn on diagnostics. However, if there are existing settings on the data factory, you see a list of settings already configured on the data factory. Select Add diagnostic setting.
- Then, give your setting a name, select Send to Log Analytics, and then select a workspace from Log Analytics Workspace.
- In Azure-Diagnostics mode, diagnostic logs flow into the AzureDiagnostics table.
- If you select AllMetrics, various ADF metrics will be made available for you to monitor or raise alerts on, including the metrics for ADF activity, pipeline, and trigger runs, as well as for SSIS IR operations and SSIS package executions.
- Lastly, select Save.
Install Azure Data Factory Analytics solution from Azure Marketplace
This solution provides you a summary of overall health of your Data Factory, with options to drill into details and to troubleshoot unexpected behavior patterns. With rich, out of the box views you can get insights into key processing including:
- At a glance summary of data factory pipeline, activity and trigger runs
- Ability to drill into data factory activity runs by type
- Summary of data factory top pipeline, activity errors
- Firstly, go to Azure Marketplace, choose Analytics filter, and search for Azure Data Factory Analytics
- Secondly, Details about Azure Data Factory Analytics
- Lastly, select Create and then create or select the Log Analytics Workspace.
Set up diagnostic logs via the Azure Monitor REST API
Diagnostic settings
Use diagnostic settings to configure diagnostic logs for non-compute resources. The settings for a resource control have the following features:
- Firstly, they specify where diagnostic logs are sent. Examples include an Azure storage account, an Azure event hub, or Monitor logs.
- Secondly, they specify which log categories are sent.
- Thirdly, they specify how long each log category should be kept in a storage account.
- Next, a retention of zero days means logs are kept forever. Otherwise, the value can be any number of days from 1 through 2,147,483,647.
- However, if retention policies are set but storing logs in a storage account is disabled, the retention policies have no effect.
- Lastly, Retention policies are applied per day. The boundary between days occurs at midnight Coordinated Universal Time (UTC). At the end of a day, logs from days that are beyond the retention policy are deleted.
Monitor SSIS operations with Azure Monitor
To lift & shift your SSIS workloads, you can provision SSIS IR in ADF that supports:
- Firstly, running packages deployed into SSIS catalog (SSISDB) hosted by Azure SQL Database server/Managed Instance (Project Deployment Model)
- Secondly, running packages deployed into file system, Azure Files, or SQL Server database (MSDB) hosted by Azure SQL Managed Instance (Package Deployment Model)
However, once provisioned, you can check SSIS IR operational status using Azure PowerShell or on the Monitor hub of ADF portal. With Project Deployment Model, SSIS package execution logs are stored in SSISDB internal tables or views, so you can query, analyze, and visually present them using designated tools like SSMS. With Package Deployment Model, SSIS package execution logs can be stored in file system or Azure Files as CSV files that you still need to parse and process using other designated tools before you can query, analyze, and visually present them.
Now with Azure Monitor integration, you can query, analyze, and visually present all metrics and logs generated from SSIS IR operations and SSIS package executions on Azure portal. Additionally, you can also raise alerts on them.
Reference: Microsoft Documentation