Bridge Specifications
| Vendor | Databricks |
| Tool Name | Databricks Unity Catalog |
| Tool Version | 10.x to 16.x |
| Tool Web Site | https://docs.databricks.com/data-governance/unity-catalog/ |
| Supported Methodology | [Metadata Management] Multi-Model, Data Store (Physical Data Model) via REST API |
SPECIFICATIONS
Tool: Databricks / Databricks Unity Catalog version 10.x to 16.x via REST API
See https://docs.databricks.com/data-governance/unity-catalog/
Metadata: [Metadata Management] Multi-Model, Data Store (Physical Data Model)
Component: DatabricksUnityCatalogLineage version 11.2.0
DISCLAIMER
This import bridge requires internet access to download third-party libraries:
- such as https://repo.maven.apache.org/maven2/ to download open source third-party libraries,
- and more sites for other third-party software such as database specific JDBC drivers.
The downloaded third-party libraries are stored into $HOME/data/download/MIMB/
- If HTTPS fails, the import bridge then tries with HTTP.
- If a proxy is used to access internet, you must configure that proxy in the JRE (see the -java.parameters option in the Miscellaneous parameter).
- If the import bridge does not have full access to internet, that $HOME/data/download/MIMB/ directory can be copied from another server with internet access where the command $HOME/bin/MIMB.sh (or .bat) -d can be used to download all third-party libraries used by all bridges at once.
By running this import bridge, you hereby acknowledge responsibility for the license terms and any potential security vulnerabilities from these downloaded third-party software libraries.
OVERVIEW
This export bridge uses a REST API to export lineage metadata to Unity Catalog service.
REQUIREMENTS
n/a
FREQUENTLY ASKED QUESTIONS
Q: What supported tools can be imported as external metadata?
A: Any supported Business Intelligence tools sourcing from Databricks, including PowerBI, Tableau, Looker, Qlik and more:
https://www.metaintegration.net/Products/MIMB/SupportedTools.html?show=Business%20Intelligence%20(BI%20Reporting)%20only&in=All
Q: Why are some external metadata icons missing?
A: Unity Catalog provides a limited set of icons including POWER_BI, TABLEAU, LOOKER, other business intelligence tools have the OTHER icon
Q: Why are the external metadata names not matching the original tool ?
A: Unity Catalogs requires unique name for any report across any business intelligence tools, and such name cannot have upper case letters, space, underscore, or any special characters. This is why unique names are generated from the original names, and a mandatory “External name suffix” parameter must be provided, such as gold-powerbi.
Note that original names are also provided in businessName within Properties.
Q: Are external metadata from a tool connected to the databricks objects for data lineage?
A: Yes, all databricks tables and views used by the imported tool (such as PowerBI) are automatically searched in Unity Catalog and connected for data lineage.
Q: Can external metadata from a tool be updated (such reports edited, added, deleted) ?
A: Yes, this the Properties contain source tool metadata (fields: lastModificationTime, nativeID and Modelkey) that can be used to compare and update the Unity Catalog.
Q: Can external metadata from a tool be deleted?
A: Yes, using the Miscellaneous parameter “-databricks.cleanup” for the scope provided in on of the parameter “External name suffix” such as gold-powerbi.
LIMITATIONS
Refer to the current general known limitations at https://metaintegration.com/Products/MIMB/Help/#!Documents/mimbknownlimitations.html
SUPPORT
Provide a troubleshooting package with:
- the debug log (can be set in the UI or in conf/conf.properties with MIR_LOG_LEVEL=6)
- the metadata backup if available (can be set in the Miscellaneous parameter with -backup option, although this common option is not implemented on all bridges for technical reasons).
Bridge Parameters
| Parameter Name | Description | Type | Values | Default | Scope |
| Server hostname | Enter the DatabricksUnityCatalogLineage Server URL here. | STRING | https://localhost/ | Mandatory | |
| Account ID | Enter the Databricks Account ID. The import bridge will use to log in. | STRING | |||
| Client ID | Enter the Databricks Client ID. The import bridge will use to log in via OAuth. | STRING | |||
| Client Secret | Enter the Databricks Client Secret. The import bridge will use to log in via OAuth. | PASSWORD | |||
| Personal access token | Enter the Azure Databricks personal access token associated with the token which the import bridge will use to log in. | PASSWORD | |||
| External name suffix | Specify a suffix for the exported External metadata object name. This suffix will be appended to the end of the object name. | STRING | Mandatory | ||
| Miscellaneous | INTRODUCTION Specify miscellaneous options starting with a dash and optionally followed by parameters, e.g. -connection.cast MyDatabase1="MICROSOFT SQL SERVER" Some options can be used multiple times if applicable, e.g. -connection.rename NewConnection1=OldConnection1 -connection.rename NewConnection2=OldConnection2; As the list of options can become a long string, it is possible to load it from a file which must be located in ${MODEL_BRIDGE_HOME}\data\MIMB\parameters and have the extension .txt. In such case, all options must be defined within that file as the only value of this parameter, e.g. ETL/Miscellaneous.txt JAVA ENVIRONMENT OPTIONS -java.memory <Java Memory's maximum size> (previously -m) 1G by default on 64bits JRE or as set in conf/conf.properties, e.g. -java.memory 8G -java.memory 8000M -java.parameters <Java Runtime Environment command line options> (previously -j) This option must be the last one in the Miscellaneous parameter as all the text after -java.parameters is passed "as is" to the JRE, e.g. -java.parameters -Dname=value -Xms1G The following option must be set when a proxy is used to access internet (this is critical to access https://repo.maven.apache.org/maven2/ and exceptionally a few other tool sites) in order to download the necessary third-party software libraries. Note: The majority of proxies are concerned with encrypting (HTTPS) the outside (of the company) traffic and trust the inside traffic that can access proxy over HTTP. In this case, an HTTPS request reaches the proxy over HTTP where the proxy HTTPS-encrypts it. -java.parameters -java.parameters -Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=3128 -Dhttp.proxyUser=user -Dhttp.proxyPassword=pass DATABRICKS UNITY CATALOG -databricks.cleanup if this options is specified the bridge will delete all external metadata objects ending with the "External name suffix". Please note that if this option is used, the bridge will not export anything. |
STRING |
Bridge Mapping
Mapping information is not available