External Metadata Database Configuration

Audience: System Administrators

Content Summary: This page outlines how to configure an external metadata database for Immuta instead of using Immuta's built-in PostgreSQL Metadata Database that runs in Kubernetes.

Overview

You can replace Immuta's built-in PostgreSQL Metadata Database with an external PostgreSQL instance when deploying Immuta in Kubernetes. Running the metadata database outside of Kubernetes eliminates the variability introduced by Kubernetes scaling and scheduling events, and infrastructure administrators have greater control over the configuration of the database overall, such as options for high availability and disaster recovery.

Configuration

PostgreSQL Version 12

Use PostgreSQL version 12 for your external database to match the version used by Immuta. Immuta only supports PostgreSQL, not PostgreSQL abstraction layers such as AWS Aurora.

When configuring your Helm values, enable an external metadata database by following these steps:

Enable an external metadata database by setting database.enabled=false in the immuta-values.yaml file and passing the connection information for the PostgreSQL instance under the key externalDatabase.
Set backup.enabled=true and externalDatabase.backup.enabled=true.
If migrating from Immuta's built-in metadata database to your external PostgreSQL database, set backup.restore.enabled=true and externalDatabase.restore.enabled=true.

database:
  enabled: false
externalDatabase:
  host: replace-with-your-external-postgres.database.hostname
  password: replace-with-your-bometauserpassword
  superuser:
    username: postgres
    password: postgrespassword
  backup:
    enabled: true
  restore:
    enabled: true

Once set up and working, a backup restore creates a hook job. This hook job performs restores without needing to touch the external database.

External Database Restore

Backup Required

The following process will destroy the database and recreate it for a restore. A backup is necessary for this action to be completed.

In the event that the external database is no longer working and a fresh restore is needed, you can connect to the external database either via a tool or the platform UI and run the following:

drop database bometadata;
create database bometadata;

External Database Object

The externalDatabase object is detailed below and in the Immuta Helm Chart Options.

Property	Description	Default Value
`host` (required)	Hostname of the external PostgreSQL database instance.	`nil`
`port`	Port of the external PostgreSQL database instance.	`5432`
`sslmode` (required)	The mode for the database connection. Supported values are `disable`, `require`, `verify-ca`, and `verify-fully`.	`nil`
`superuser.username` (required)	Username for the superuser used to initialize the PostgreSQL instance.	`nil`
`superuser.password` (required)	Password for the superuser used to initialize the PostgreSQL instance.	`nil`
`username`	Username that Immuta creates and uses for the application.	`bometa`
`password` (required)	Password associated with `username`.	`nil`
`dbname`	Database name that Immuta uses.	`bometadata`
`backup.enabled`	Enable flag for external database backups. Only used when `backup.enabled=true`.	`true`
`restore.enabled`	Enable flag for the external database restore. Only used when `backup.restore.enabled=true`.	`true`

Additionally, it is possible to use existingSecret instead of setting externalDatabase.password and externalDatabase.superuser.password in the Helm values. These passwords map to the same keys that are used for the built-in database. For example,

apiVersion: v1
kind: Secret
metadata:
  name: immuta-secret
data:
  databasePassword: <password for externalDatabase.password>
  databaseSuperuserPassword: <password for externalDatabase.superuser.password>

Superuser User Requirements

There are two options for the user used for externalDatabase.superuser:

a user with superuser role attribute (such as the postgres user), or
a user with reduced privileges (this option requires preliminary setup).

A user with superuser permission is required to use the built-in backup functionality. It is possible to disable built-in backups for the external PostgreSQL instance when using a less-privileged user; however, care must be used to ensure that the Query Engine is in-sync with the external database if backups are restored. One situation in which built-in backups might be disabled for the external database is when database provider backups are used. For example, an AWS RDS instance might be configured to take backups, and the user would rely on those backups for the metadata database.

The bometadata database must exist ahead of time, and the built-in pgcrypto extension must be created in it. The user must have CREATEROLE permissions and have been granted CONNECT, CREATE, and TEMPORARY (ALL) on the bometadata database:

CREATE DATABASE bometadata;
\connect bometadata
CREATE EXTENSION pgcrypto;
CREATE USER example_immuta_admin WITH CREATEROLE ENCRYPTED PASSWORD '';
GRANT ALL ON DATABASE bometadata TO example_immuta_admin;
CREATE ROLE bometa WITH CREATEROlE LOGIN ENCRYPTED PASSWORD '';
GRANT ALL ON DATABASE bometadata TO bometa;
ALTER ROLE bometa SET search_path TO bometadata,public;

Migrating from Immuta's Built-In PostgreSQL Database to External Metadata Database

For existing deployments, you can migrate from the built-in database to an external database. To migrate, backups must be configured.

Follow step 1 (Capture a Complete Current Backup) of this process to back up your instance. Do not complete the other steps in that tutorial.
Set the external database Helm values outlined above and backup.restore.enabled=true in the immuta-values.yaml file.
Run helm upgrade.