Start preparing for your Next Exam | Use coupon – TOGETHER | Avail 30% discount

Implement Data Lake Storage monitoring

  1. Home
  3. Implement Data Lake Storage monitoring

Microsoft DP-200 exam is getting retired on June 30, 2021. A new replacement exam Data Engineering on Microsoft Azure Beta (DP-203) is available.

Go back to DP-200 Tutorials

In this we will learn to enable diagnostic logging for your Azure Data Lake Storage Gen1 account and how to view the logs collected for your account.

Organizations can enable diagnostic logging for their Azure DL Storage Gen1 account to collect data access audit trails that provides information such as list of users accessing the data, how frequently the data is accessed and how much data is stored in the account. When enabled, the diagnostics and/or requests are logged on a best-effort basis. Both Requests and Diagnostics log entries are created only if there are requests made against the service endpoint.

Enable diagnostic logging for your Data Lake Storage Gen1 account

  • Firstly, sign on to the new Azure portal.
  • Secondly, open your Lake Storage Gen1 account, and from your Data Lake Storage Gen1 account blade, click Diagnostic settings.
  • Thirdly, in the Diagnostics settings blade, click Turn on diagnostics.
  • Next, in the Diagnostics settings blade, make the following changes to configure diagnostic logging.
    • Further, for Name, enter a value for the diagnostic log configuration.
    • Then, you can choose to store/process the data in different ways.
      • select the option to Archive to a storage account to store logs to an Azure Storage account. You use this option if you want to archive the data that will be batch-processed at a later date.
      • Next, select the option to Stream to an event hub to stream log data to an Azure Event Hub.
      • Then, select the option to Send to Log Analytics to use the Azure Monitor service to analyze the generated log data.
    • After that, specify whether you want to get audit logs or request logs or both.
    • Then, specify the number of days for which the data must be retained. Retention is only applicable if you are using Azure storage account to archive log data.
    • Lastly, click Save.
DP-200 practice tests

View diagnostic logs for your Data Lake Storage Gen1 account

There are two ways to view the log data for your Data Lake Storage Gen1 account.

Using the Data Lake Storage Gen1 Settings view

  • Firstly, from your Data Lake Storage Gen1 account Settings blade, click Diagnostic Logs.
  • Secondly, in the Diagnostics Logs blade, you should see the logs categorized by Audit Logs and Request Logs.
    • Request logs capture every API request made on the Data Lake Storage Gen1 account.
    • Audit Logs are similar to request Logs but provide a much more detailed breakdown of the operations being performed on the Data Lake Storage Gen1 account.
  • Lastly, to download the logs, click the Download link against each log entry.

From the Azure Storage account that contains log data

  • Firstly, open the Azure Storage account blade associated with Data Lake Storage Gen1 for logging, and then click Blobs. The Blob service blade lists two containers.
    • The container insights-logs-audit contains the audit logs.
    • The container insights-logs-requests contains the request logs.
  • Secondly, within these containers, the logs are stored under the following structure.
log structure as it is stored in the container.
Image Source: Microsoft
Implement Data Lake Storage monitoring DP-200 Online course

Reference: Microsoft Documentation

Go back to DP-200 Tutorials

Menu