With the download of the generated Artifacts, a zip file was created in the default download location set for your Web Browser as:
Let's see what this zip file contains now.
Generated Artifacts content
The Generated Artifacts zip file contains the following:
- A Helpers folder
- A Jupyter folder
- A LoadControl folder
- A ParseError.log file (optional)
- The placeholder toolkit is in the replace_placeholders.ps1 file
- The placeholder configuration is in the replacement_config.json file
Helpers folder
The Helpers folder contains a Jupyter Notebook files helpful to check the loaded data:
![](https://knowledge.bigenius-x.com/hs-fs/hubfs/image-png-Jun-24-2024-01-57-12-3614-PM.png?width=401&height=157&name=image-png-Jun-24-2024-01-57-12-3614-PM.png)
Jupyter folder
The Jupyter folder contains 1 Jupyter Notebook file by Target Model Object to deploy and 1 Jupyter Notebook to execute the deployment based on the other files (XXXXX_Deployment.ipynb):
Each file contains all the needed code to create the Target Parquet file in the Target Lake House.
Example of a part of a Jupyter Notebook file:
...
"source" : [
"# HubLoader: Customer_Hub_Hub Loader_1\n",
"\n",
"operation_metrics_collection = {}\n",
"\n",
"result_df = spark.sql(f\"\"\"\n",
"INSERT\n",
"INTO `{fabricXXXXXdatavaultandmart#rawvault#database_name}`.`{fabricXXXXXdatavaultandmart#rawvault#schema_name}`.`rdv_hub_customer_hub` (\n",
" `bg_loadtimestamp`\n",
" ,`hub_hk`\n",
" ,`bg_sourcesystem`\n",
" ,`accountnumber`\n",
")\n",
"SELECT\n",
" {bg_loadtimestamp} AS `bg_loadtimestamp`\n",
" ,`bg_source`.`hub_hk` AS `hub_hk`\n",
" ,`bg_source`.`bg_sourcesystem` AS `bg_sourcesystem`\n",
" ,`bg_source`.`accountnumber` AS `accountnumber`\n",
"FROM `{fabricXXXXXdatavaultandmart#rawvault#database_name}`.`{fabricXXXXXdatavaultandmart#rawvault#schema_name}`.`rdv_hub_customer_hub_delta` AS `bg_source`\n",
"\"\"\")\n",
"from delta.tables import DeltaTable\n",
"table_file_location = spark.sql('describe detail `{fabricXXXXXdatavaultandmart#rawvault#database_name}`.`{fabricXXXXXdatavaultandmart#rawvault#schema_name}`.`rdv_hub_customer_hub`').select('location').collect()[0][0]\n",
"delta_table = DeltaTable.forPath(spark, table_file_location)\n",
"last_operation_history = delta_table.history(1)\n",
"operation_metrics = last_operation_history.select('operationMetrics').collect()[0][0]\n",
"operation_metrics_collection['inserttarget_{fabricXXXXXdatavaultandmart#rawvault#database_name}_{fabricXXXXXdatavaultandmart#rawvault#schema_name}_rdv_hub_customer_hub'] = operation_metrics\n",
"\n"
]
...
As you can observe, a placeholder is used for the target database and schema names.
Before you deploy the generated code in your target environment, please Replace the placeholders in the Generated Artifacts.
LoadControl folder
The LoadControl folder contains, depending on the LoadControl Options Property values:
- Multi-Threading Notebook:
- A JSON file bg_XXXX_dependencies.json containing the dependencies between the Model Objects
- A Jupyter Notebook XXXX_MultithreadingLoadExecution.ipynb to use a native load control using multi-threading load
- Sequential Notebook:
- A Jupyter Notebook XXXX_SimpleLoadExecution.ipynb to use a native load control using a unique thread
- Airflow:
- A Python file XXX_dag.py for a load with Airflow
Before you deploy the generated code in your target environment, please Replace the placeholders in the Generated Artifacts.
If the Property Configure LoadControl is set to False, the configuration files will not be generated.
If the Property Deploy LoadControl is set to False, the load files will not be generated.
ParseError.log file
If some errors occur during the generation of your Artifacts, the file ParseError.log will contain these errors.
Example of error:
ERROR LinkSourceView: Customer_Order_Link_link source view_1 ### TargetName: RDV_LNK_Customer_Order_Link_Source
Invalid amount of dataflows for ArchitectureModelObject name Customer_Order_Link_link source view_1
System.Exception: Invalid amount of dataflows for ArchitectureModelObject name Customer_Order_Link_link source view_1
at biGENIUS.Implementation.Base.DataflowBuilder.Build(ArchitectureModelObject architectureModelObject) in C:\.sources\bgaas\bigenius.implementation.base\src\biGENIUS.Implementation.Base\StatementBuilder\DataflowBuilder.cs:line 18
at biGENIUS.Implementation.Datavault.Mssql.SourceViewBuilder.Build(ArchitectureModelObject architectureModelObject) in C:\.sources\bgaas\bigenius.implementation.datavault.mssql\src\biGENIUS.Implementation.Datavault.Mssql\StatementBuilder\SourceViewBuilder.cs:line 22
at biGENIUS.Implementation.Datavault.Mssql.LinkSourceView.BuildScriptInternal() in C:\.sources\bgaas\bigenius.implementation.datavault.mssql\src\biGENIUS.Implementation.Datavault.Mssql\Template\LinkSourceView.cs:line 16
at biGENIUS.Implementation.Base.TemplateScriptBase.BuildScript(IGeneratorContext generatorContext, ArchitectureModelObject architectureModelObject, TemplateRenderingContext templateRenderingContext, StatementBuilderContext statementBuilderContext) in C:\.sources\bgaas\bigenius.implementation.base\src\biGENIUS.Implementation.Base\TemplateBase\TemplateScriptBase.cs:line 45
at biGENIUS.Implementation.Base.GeneratorHookBase.RenderTemplate(IGeneratorContext generatorContext, Assembly assembly, ArchitectureModelObject amo) in C:\.sources\bgaas\bigenius.implementation.base\src\biGENIUS.Implementation.Base\Hook\GeneratorHookBase.cs:line 169
In this case, no Dataflow was configured for the Link named Customer_Order_Link.
So, please define the Dataflow and generate a new time.
If no errors occur during generation, the file ParseError.log will not exist in the Generated Artifact zip file.
Placeholder toolkit
It is composed when you deploy PowerShell script: replace_placeholders.ps1.
It uses a placeholder configuration file to replace the information needed for the connection in the generated artifacts.
For more information, please review Replace the placeholders in the Generated Artifacts.
Placeholder configuration
It is composed of a JSON file: replacement_config.json.
The placeholder toolkit uses it to replace the connection needed information in the generated artifacts.
For more information, please review Replace the placeholders in the Generated Artifacts.