Databricks Unity Catalog Query Audit Logs

Native query audit for Unity Catalog captures user data access activity within Unity Catalog as Immuta audit logs. Immuta audits the activity of all Unity Catalog users and tables. Multiple access options are supported for audit:

• Cluster queries with the following supported languages: SQL, Scala, Python, and R.
• SQL Warehouse queries

Requirements

• Databricks Unity Catalog integration with native query audit enabled
• A Databricks deployment with system tables capabilities
• Provide the Immuta system account with access to audit-related Unity Catalog system tables

Audit frequency

Immuta collects audit records once an hour by default. This ingest frequency can be configured from the Immuta app settings page.

To manually prompt the native query audit, click Native Query Audit on the Immuta audit page.

Audit messages

Each audit message from the Immuta platform will be a one-line JSON object containing the properties listed below. These audit logs are stored with the recordType: nativeQuery.

Example audit record

{
  "id": "9f542dfd-5099-4362-a72d-8377306db3b8",
  "dateTime": "1686072470865",
  "month": 1481,
  "profileId": 999111223,
  "userId": "taylor@immuta.com",
  "dataSourceId": 21,
  "dataSourceName": "Immuta_Sample_Data",
  "count": 1,
  "recordType": "nativeQuery",
  "success": true,
  "component": "nativeSql",
  "accessType": "query",
  "query": "select * from samples_catalog.samples_schema.databricks_sample_data limit 52;\n",
  "queryId": "869500255746459",
  "extra": {
    "handler": "Databricks",
    "startTime": "2023-06-06 13:27:50.865",
    "endTime": "2023-06-06 13:27:51.678",
    "duration": "557",
    "nativeObject": "databricks_sample_data",
    "nativeObjectFullName": "samples_catalog.samples_schema.databricks_sample_data",
    "host": "immuta-example.cloud.databricks.com",
    "actionStatus": "SUCCESS",
    "actionStatusReason": null,
    "errorCode": null,
    "actionName": "runCommand",
    "accountId": "52e863bc-ea7f-46a9-8e17-6aed7541832d",
    "notebookId": "869500255746458",
    "clusterId": "0511-134653-3eykv2e7",
    "requestId": "6435b58c-9557-4395-ae73-dc8ebe83c15f",
    "sessionId": null,
    "warehouseId": null,
    "queryLanguage": "sql",
    "actorIp": "100.40.110.136",
    "actorEmail": "taylor@immuta.com",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36",
    "workspaceId": 8765531160949612
  },
  "createdAt": "2023-06-07T15:14:41.950Z",
  "updatedAt": "2023-06-07T15:14:41.950Z"
}

Limitations

• Enrichment of audit logs with Immuta entitlements information is not supported. While you will see these entitlements in the Databricks Spark audit logs, the following will not be in the native query audit for Unity Catalog:
  • Immuta policies information
  • User attributes
  • Groups
• Immuta determines unauthorized events based on error messages within Unity Catalog records. When the error messages contain expected language, unauthorized events will be available for native query audit for Unity Catalog. In other cases, it is not possible to determine the cause of an error.
• Audit for cluster queries do not support UNAUTHORIZED status. If a cluster query is unauthorized, it will show FAILURE.
• Data source information will be provided when available:
  • For some queries, Databricks Unity Catalog does not report the target data source for the data access operation. In these cases the activity is audited, yet the audit record in Immuta will not include the target data source information.
  • Data source information is not available for unauthorized queries and events.
• Column information from the query is not currently supported.
• The cluster for the Unity Catalog integration must always be running for Immuta to audit activity.
• Immuta audit records include unregistered data sources and users; however, activity from them will not appear in any governance reports.

Enable native query audit for Unity Catalog

To enable native query audit for Unity Catalog, complete the following steps before configuring the integration:

1. Enable a system schema where the <SCHEMA_NAME> is access.

2. Grant your Immuta system account user access to the Databricks Unity Catalog system tables. For Databricks Unity Catalog audit to work, Immuta must have, at minimum, the following access.

   • USE CATALOG on the system catalog
   • USE SCHEMA on the system.access schema

   • SELECT on the following system tables:

     • system.access.audit
     • system.access.table_lineage
     • system.access.column_lineage

3. Enable verbose audit logs in Unity Catalog.

4. Use the Databricks Personal Access Token in the configuration above for the account you just granted system table access. This account will be the Immuta system account user.