With the download of the generated Artifacts, a zip file was created in the default download location set for your Web Browser as:
Let's see what this zip file contains now.
Generated Artifacts content
The Generated Artifacts zip file contains the following:
- A Helpers folder
- A Jupyter folder
- A LoadControl folder
- A ParseError.log file (optional)
- The placeholder toolkit is in the replace_placeholders.ps1 file
- The placeholder configuration is in the replacement_config.json file
Helpers folder
The Helpers folder contains some Jupyter Notebook files helpful in loading data:
See Microsoft Fabric - Load data with a native load control.
Jupyter folder
The Jupyter folder contains 1 Jupyter Notebook file by Target Model Object to deploy and 1 Jupyter Notebook to execute the deployment based on the other files (Deployment.ipynb):
Each file contains all the needed code to create the Target Parquet file in the Target Lake House.
Example of a part of a Jupyter Notebook file:
...
"source" : [
"# HubLoader: Customer_Hub_Hub Loader_1\n",
"\n",
"operation_metrics_collection = {}\n",
"\n",
"result_df = spark.sql(f\"\"\"\n",
"INSERT\n",
"INTO `{fabric1.4.0datavaultandmart#rawvault#database_name}`.`{fabric1.4.0datavaultandmart#rawvault#schema_name}`.`rdv_hub_customer_hub` (\n",
" `bg_loadtimestamp`\n",
" ,`hub_hk`\n",
" ,`bg_sourcesystem`\n",
" ,`accountnumber`\n",
")\n",
"SELECT\n",
" {bg_loadtimestamp} AS `bg_loadtimestamp`\n",
" ,`bg_source`.`hub_hk` AS `hub_hk`\n",
" ,`bg_source`.`bg_sourcesystem` AS `bg_sourcesystem`\n",
" ,`bg_source`.`accountnumber` AS `accountnumber`\n",
"FROM `{fabric1.4.0datavaultandmart#rawvault#database_name}`.`{fabric1.4.0datavaultandmart#rawvault#schema_name}`.`rdv_hub_customer_hub_delta` AS `bg_source`\n",
"\"\"\")\n",
"from delta.tables import DeltaTable\n",
"table_file_location = spark.sql('describe detail `{fabric1.4.0datavaultandmart#rawvault#database_name}`.`{fabric1.4.0datavaultandmart#rawvault#schema_name}`.`rdv_hub_customer_hub`').select('location').collect()[0][0]\n",
"delta_table = DeltaTable.forPath(spark, table_file_location)\n",
"last_operation_history = delta_table.history(1)\n",
"operation_metrics = last_operation_history.select('operationMetrics').collect()[0][0]\n",
"operation_metrics_collection['inserttarget_{fabric1.4.0datavaultandmart#rawvault#database_name}_{fabric1.4.0datavaultandmart#rawvault#schema_name}_rdv_hub_customer_hub'] = operation_metrics\n",
"\n"
]
...
As you can observe, a placeholder is used for the target database and schema names.
Before you deploy the generated code in your target environment, please Replace the placeholders in the Generated Artifacts.
LoadControl folder
The LoadControl folder contains 1 JSON file by Target Model Object to deploy and a Python file:
Each JSON file contains all the necessary code to create the pipeline in Azure Data Factory.
Example of a part of a JSON file:
...
"policy": {
"timeout": "{fabric1.4.0datavaultandmart#datafactory_pipeline#timeout}",
"retry": {fabric1.4.0datavaultandmart#datafactory_pipeline#retry},
"retryIntervalInSeconds": {fabric1.4.0datavaultandmart#datafactory_pipeline#retry_interval_in_seconds},
"secureOutput": {fabric1.4.0datavaultandmart#datafactory_pipeline#secure_output},
"secureInput": {fabric1.4.0datavaultandmart#datafactory_pipeline#secure_input}
},...
The Python file (Fabric 1_4_0 Datavault and Mart_dag.py) contains all the code to create the Airflow pipelines to load the data creation script.
As you can observe, placeholders are used to configure Azure Data Factory behavior.
Before you deploy the generated code in your target environment, please Replace the placeholders in the Generated Artifacts.
ParseError.log file
If some errors occur during the generation of your Artifacts, the file ParseError.log will contain these errors.
Example of error:
ERROR LinkSourceView: Customer_Order_Link_link source view_1 ### TargetName: RDV_LNK_Customer_Order_Link_Source
Invalid amount of dataflows for ArchitectureModelObject name Customer_Order_Link_link source view_1
System.Exception: Invalid amount of dataflows for ArchitectureModelObject name Customer_Order_Link_link source view_1
at biGENIUS.Implementation.Base.DataflowBuilder.Build(ArchitectureModelObject architectureModelObject) in C:\.sources\bgaas\bigenius.implementation.base\src\biGENIUS.Implementation.Base\StatementBuilder\DataflowBuilder.cs:line 18
at biGENIUS.Implementation.Datavault.Mssql.SourceViewBuilder.Build(ArchitectureModelObject architectureModelObject) in C:\.sources\bgaas\bigenius.implementation.datavault.mssql\src\biGENIUS.Implementation.Datavault.Mssql\StatementBuilder\SourceViewBuilder.cs:line 22
at biGENIUS.Implementation.Datavault.Mssql.LinkSourceView.BuildScriptInternal() in C:\.sources\bgaas\bigenius.implementation.datavault.mssql\src\biGENIUS.Implementation.Datavault.Mssql\Template\LinkSourceView.cs:line 16
at biGENIUS.Implementation.Base.TemplateScriptBase.BuildScript(IGeneratorContext generatorContext, ArchitectureModelObject architectureModelObject, TemplateRenderingContext templateRenderingContext, StatementBuilderContext statementBuilderContext) in C:\.sources\bgaas\bigenius.implementation.base\src\biGENIUS.Implementation.Base\TemplateBase\TemplateScriptBase.cs:line 45
at biGENIUS.Implementation.Base.GeneratorHookBase.RenderTemplate(IGeneratorContext generatorContext, Assembly assembly, ArchitectureModelObject amo) in C:\.sources\bgaas\bigenius.implementation.base\src\biGENIUS.Implementation.Base\Hook\GeneratorHookBase.cs:line 169
In this case, no Dataflow was configured for the Link named Customer_Order_Link.
So, please now define the Dataflow and generate a new time.
If no errors occur during generation, the file ParseError.log will not exist in the Generated Artifact zip file.
Placeholder toolkit
It is composed when you deploy PowerShell script: replace_placeholders.ps1.
It uses a placeholder configuration file to replace the connection needed information in the generated artifacts.
If you want more information, please review Replace the placeholders in the Generated Artifacts.
Placeholder configuration
It is composed of a JSON file: replacement_config.json.
The placeholder toolkit uses it to replace the connection needed information in the generated artifacts.
If you want more information, please review Replace the placeholders in the Generated Artifacts.