Meta Integration® Model Bridge (MIMB)
"Metadata Integration" Solution

Known Limitations as of 2017/03/31

Table of Contents

Overview

The Meta Integration® Model Bridge (MIMB) software provides solutions for:
  • Metadata Harvesting required for Metadata Management (MM) applications, including metadata harvesting from live databases (or big data), Data Integration (DI, ETL and ELT) , and Business Intelligence (BI) software.
  • Metadata Integration required for tool integration in the software development life cycle, including all necessary forward and reverse engineering steps to update each other tools: for example between Data Modeling (DM), Data Integration (DI), and Business Intelligence (BI) design tools.
  • Metadata Conversion required for legacy migration from an older design tool to a new one.

Copyright Notice

1 - Consistency Check of the Source Model

BACKGROUND PURPOSE:
Data Modeling, Data Integration (ETL/DI), Business Intelligence (BI) and other design tools supported by MIMB may not always save models which are valid. Indeed, some of these models may still be under development (not yet completed), or the model validation may be a separate step (design tool menu entry, separate tool, or part of the model publication). Therefore, MIMB import bridges may be dealing with semantically incorrect or incomplete models which the MIR repository can save, but they may break the export bridges. Having a centralized model validation (or consistency check) between the import and export phases allows to continuously improve the consistency check algorithms, and avoids having each export bridge retest for the same model validity issues.

INPUT PARAMETERS:
The user has the choice of 3 levels of consistency check:

OUTPUT RESULTS:
Consistency checks will report (in the log) "WARNINGS" on the source model, or "ERRORS" that are fatal and will prevent the export to any tool. The solution is to fix the original model, and then import it again.

2 - Metadata Import/Export is limited to the "Bridge Specifications"

The metadata mapping and transformations implemented by Meta Integration Model Bridge (MIMB) are limited to the published model mapping specifications for each integrated tool. Such specifications are part of the MIMB online help, and are also published on the web at http://www.metaintegration.net/Products/MIMB/SupportedTools.html (select/click on a bridge to get the full specifications including all the bridge parameters/options and detailed mapping specifications).

The metadata mapping specifications are intended to define the scope of metadata captured (by an import bridge) or populated (by an export bridge) in the 3rd party tool or format (tool and metamodel naming). Therefore, it can only be used to analyze the scope of the metadata covered by the bridge algorithms. However, it is not intended to be a complete documentation of all the mapping algorithms involved in migrating metadata from any tool to any other tool, as the algorithms includes expression parsing, expression conversions and other complex operations which may not be expressed as a mapping specification.

3 - Potential Loss of Metadata from Tool A to Tool B

There are 7 points of potential metadata loss from Tool A to Tool B:

  1. Tool A's Methodology/Metamodel Support & Implementation,
  2. Tool A's Export Capabilities,
  3. MIMB's Import Bridge for Tool A,
  4. MIR's Methodology/Metamodel Support & Implementation,
  5. MIMB's Export Bridge for Tool B,
  6. Tool B's Import Capabilities,
  7. Tool B's Methodology/Metamodel Support & Implementation.

 

MITI provides the best possible metadata mapping across methodologies and tools. However, some modeling concepts may not be available in Tool A, MIR, or Tool B. In such a case, the import & export bridges try to use the descriptions or notes to carry such modeling concepts all the way from the source to the target tool without any loss.

4 - Limitations on Import/Export Round-Trip

There are an additional considerations when performing what is often referred to as a "round-trip", or importing from tool A, exporting to tool B, editing the model (or not) in tool B, re-import this model in from tool B, and exporting back to tool A. In this case, the 7 points of potential metadata loss identified above are compounded so that there are now 7 additional points of loss:

The fidelity of some simple types of metadata may be well maintained, where there is a direct, one-to-one, translation from tool A --> MIR's methodology --> tool B and then back again. Examples include tools with the same methodology and metadata coverage (e.g., a data modeling and design tool to another data modeling or design tool), or simple elements like the name, description, layout, etc. This process is referred to as "round-tip migration", and may work successfully given these restrictions.

However, it is never recommended that a combination of import-> export-> edit-> export-> import (i.e., "round-trip engineering") be expected to retain fidelity when there is a migration from tool A to tool B (and vice-versa) of methodology, metadata coverage or any of the known limitations identified above. What round-trip engineering implies is a forward-engineering and then reverse engineering process, which has never been a reality in systems engineering. One should not expect the forward engineering of simple to complex transformations (e.g., forward engineering from a spreadsheet to an ETL tool and then reverse engineering from the ETL tool back to a spreadsheet) would be reversible, or even bear much resemblance to the original.

The proper process is to treat the forward-engineering step as "irreversible", treat the reverse engineering step as a separate process, and not expect to effectively compare the original will the "round-trip" result, nor re-use the result for further engineering of the metadata.

Some examples:

Although each of the above steps can use the same Excel format (MetaMap), the round-trip re-engineering is not recommended and is likely to produce results which are not useful for all of the reasons identified above. Furthermore, like most destination tools, Informatica PowerCenter, as an example, is unable to consume the exported metadata correctly, i.e., it is unable to compare/merge/integrate different versions of complex transformations. (Please see the section on Limitations in the Receiving Environment for Merge/Update of Metadata.)

5 - Limitations on Export Bridge's Merge/Update Capabilities

The export bridges generally depend upon the tool receiving the metadata to integrate the metadata correctly, i.e., provide compare, merge and integrate capabilities. Such capabilities are:

In order to accommodate these limitations in the destination tool, the following MIMB export bridges offer "limited" metadata update capabilities (Please see the bridge parameter tooltips for more details):

6 - API based Bridges

Some MIMB bridges rely on the tool's API to import/export metadata, such as:

Such MIMB bridges require these tools to be installed with their API properly setup the machine (PC) where the MIMB bridge is executed, which sometimes mean that the toll's client or desktop version needs to be installed with a valid license (e.g. Erwin Data Modeler) The bridges requiring the tool's SDK/API are annotated with a tool set icon on the published supported tools at: http://www.metaintegration.net/Products/MIMB/SupportedTools.html.

7 - Multi-file based Bridges

When multiple files are involved, all must be accessible to the MIMB bridge.

Some MIMB bridges have parameters of type "Directory" instead of just "File", and therefore require access to a directory of multiple files, for example:

Some MIMB bridges have parameters of type "File" which support the notion of "include" files, and therefore require access to all the specified include file paths, for example:

8 - Graphical Layout is carried only through some Bridges

Some MIMB bridges convert the graphical information between tools, including conversion of the model layout between various notations like IDEF1X data modeling, and UML object modeling. The following MIMB bridges carry graphical information:

However, some MIMB bridges do not transfer the graphical information of the model layout. The primary reason for this limitation is that the import / export capabilities of most tools do not provide graphical information. In other words, their published file formats and/or Application Programming Interface (API) cover the semantic (each of the modeling concepts), but not the graphical information (i.e. the concept's associated shape sizes and positions). Furthermore, when available, such graphical information is not easily reusable in the target tool. This problem is also true for tools sharing the same methodology (e.g. a tool may allow a graphical layout that is not graphically implementable in another equivalent tool). This problem is accentuated when crossing methodology boundaries (e.g. IDEF1X to UML).

9 - SQL Parsing of Transformation Expressions in Import Bridges

Most Database, Data Integration (DI), and Business Intelligence (BI) import bridges depend on a common "SQL Transformation Expression Parsing" library to extract the data flow lineage out of the many transformation expressions used by these technologies. In fact, each tool and technology has its expression syntax. DI and BI tools often defined their transformation expression languages with a syntax close to programming languages like Visual Basic or Java. However, most DI and BI tools have support for SQL override, allowing data transformations to be defined as the full SQL executed as is by the original database. Note that this SQL parsing does not support the syntax of special data stores such as MOLAP Cubes often seen in BI tools such as Oracle Hyperion Essbase.

The section explains the general limitations in expression parsing, in particular with respect to the SQL syntax variations from the SQL 99 standard to the limitations in proprietary variations and database extensions for each popular database such as IBM DB2, Microsoft SQL Server, Oracle, Teradata, etc.

Furthermore, the support for SQL use cases from Data Definition Language (DDL) to Data Manipulation Language (DML) is increasingly limited by the complexity as the SQL features:

  1. SQL Views
    SQL Views are defined by a single SQL statement. SQL Views are used to read databases and are therefore mostly based on a complex SELECT statement.
    SQL Views are well supported with only a few limitations based on the some proprietary extensions of some databases.
  2. SQL Operations (Stored Procedures, Functions and Triggers)
    SQL Operations are defined by multiple SQL statements that are pre-compiled and stored on the server for reuse, and can have parameters. SQL Operations often contain multiple DML statements (SELECT, INSERT, UPDATE, DELETE, etc.), and use control statements (IF, THEN, ELSE, CASE, FOR, etc.).
    Support for SQL Operations at this point is limited to functions and stored procedures. The additional limitations are the following:
    • Parsing of independent SQL Stored Procedures and Functions is currently limited to the following database technologies:
      (although some statements specific to database proprietary SQL syntax may not be supported)
      • Microsoft SQL Server Database
      • Oracle Database
      • PostgreSQL Database
      • Pivotal GreenPlum Database
      • Teradata Database
    • Most dynamic and chaining aspects of the operations are not supported such as:
      • operations calling other operations
      • operations that execute dynamic SQL (e.g. SQL text passed as parameter)
      • external operations (e.g. includes)
    • Some usage aspects of such operations by DI and BI tools might be limited:
      • using SQL operations to read data is supported in most DI and BI tool import bridges
      • using SQL operations to write data is supported only in DI tool import bridges such as Informatica PowerCenter, Microsoft SSIS and Talend
  3. SQL Scripts
    SQL Script files usually consist of multiple SQL statements and sometimes designed to be executed from outside the database, and therefore not necessarily compiled in the database. Such scripts are often handcrafted and sometimes generated or part of the database utilities (e.g. Teradata fastload, bulkload and multiload)
    Parsing of independent SQL script files (.sql) is currently limited to the following database technologies:
    • Oracle Database SQL Script (PL/SQL)
    • IBM DB2 Universal Database (UDB) SQL Script - Beta Bridge
    • Teradata Database SQL Script (BTEQ)
    • Apache (Cloudera, Hortonworks, MapR) Hadoop HiveQL Script

9.1 - SQL Expression Analyzer Based Import Bridges

MIMB bridges depending on "Transformation Expression Parsing" include the following types of import bridges:

9.2 - SQL Expression Analyzer intrinsic limitations

Unsupported features

9.2.1 - Limitations on SQL99 Keywords

Most popular SQL99 keywords are supported by the expression analyzer, including (but not limited to):

9.2.2 - Limitations on Database extensions

In addition to the list of SQL99 keywords, the expression analyzer has been enhanced to support common database specific constructs. The limitations are in the list below.

9.2.3 - Limitations on Data Type Support

The expression analyzer doesn't support user defined complex data types.

10 - Conversions of Transformation Expressions in Export Bridges

Some MIMB Export Bridges depend on "Transformation Expression Conversions" to restore the transformation semantic (ClassifierMap and FeatureMap). Such conversion might be limited by the complexity of the transformation expression. If the target tool accepts expressions in textual form, such expressions too complex to restore are converted only in textual form, as defined in the source tool language. If the expression can be successfully restored, it is transformed into the target tool language.

MIMB bridges depending on "Transformation Expression Conversions" include the following export bridges:

11 - Internationalization Aspects

The internationalization of the MIMB bridge names, parameters, tooltips and messages is managed by XML files within [MIMB-INSTALL-DIR]\conf\MIRI18n\

The support for transferring internationalized (e.g. multi-byte character sets) through MIMB is achieved via the UTF8 encoding in the MIR neutral metamodel. The MIMB import and export bridges which cannot automatically detect the encodings of the tools, offer a parameter called "Encoding" allowing the user to specify one. There are however a few minor limitations:

12 - Multi-Model Import Bridges for Incremental Metadata Harvesting

Import bridges can be used for metadata harvesting into the repository of a Metadata Management (MM) solution. The metadata to be harvested can be very large, especially from Business Intelligence (BI) environments which may contain thousands of reports. Consequently some import bridges are implemented as a "Multi Model" bridge managing the large amount of metadata in multiple Model files. This approach also allows for incremental harvesting where the import bridge can check what has changed since the last import date, and return an updated "table of contents" indicating which models have been added, modified or deleted and import only the new or updated models. Such feature is supported only by a limited amount of import bridges listed below:

12 - Linux vs. Windows Only Availability

Most bridges are written Java and can therefore run on both Windows and Linux platforms where a Java RuneTime Environment (JRE) is available. However, a significant amount bridges are written in C++ compiled only for Windows, especially when they depend on Windows only COM API, such as:

The bridges running only on Windows are annotated with a Windows icon on the published supported tools at: http://www.metaintegration.net/Products/MIMB/SupportedTools.html.

Copyright © Meta Integration Technology, Inc. 1997-2016 All Rights Reserved.

Meta Integration® is a registered trademark of Meta Integration Technology, Inc.
All other trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.