Create Bonze-Silver-Gold-table in minutes in Databricks following mediation architecture

 AI is doing a good job in Databricks platform. 

Earlier, following typical mediation architecture, we used to write bunch of code to read the raw data, create a bronze table and then silver table, and finally the gold table. 

Now, I open databricks and upload the raw CSV data file. Next, I prompt properly and ask AI to create the code for me.  



Code generate within seconds. Ready to execute.


However, AI is not always right as I'm. This is acceptable. Code unable to understad the final level aggregration columns there. Good news is AI is there to diagnose the error and recomend the right code.



AI, understand what code change is required while doing analysis generated SCHEMA in the dataframe. AI recomending you the right python code to fix this.


Let's divide the phase in two sefment. Creating Bronze and Silver table in one segment and Gold one is another code segment.

Once the Silver DataFrame generates, print the dataframe schema and review the column before write the right column in the Gold phase code segment.

Finally, when placing the right column in the column, and re-run the code segment, all done.


The entire transformation from raw data to gold table data took hardly less than five minutes to achive.

Enjoy AI :)

Comments

Post a Comment

Popular posts from this blog

SharePoint Framework Reference Guide

SPFx stands for SharePoint Framework -  https://learn.microsoft.com/en-us/javascript/api/overview/sharepoint?view=sp-typescript-latest   Below are the packages of SPFx and we are using them while using SPFx framework for development in our day-to-day activities. .'@microsoft/decorators' - https://learn.microsoft.com/en-us/javascript/api/decorators?view=sp-typescript-latest '@microsoft/sp-core-library' - https://learn.microsoft.com/en-us/javascript/api/sp-core-library?view=sp-typescript-latest '@microsoft/sp-component-base package' - https://learn.microsoft.com/en-us/javascript/api/sp-component-base?view=sp-typescript-latest '@microsoft/sp-application-base package' - https://learn.microsoft.com/en-us/javascript/api/sp-application-base?view=sp-typescript-latest '@microsoft/sp-dynamic-data package' - https://learn.microsoft.com/en-us/javascript/api/sp-dynamic-data?view=sp-typescript-latest '@microsoft/sp-extension-base package' - http...
Read more

SharePoint Admin Center

Image
 SharePoint Admin Center allows you to view your organization's SharePoint site including communication sites, channel sites, and sites that belong to Microsoft 365 group. This is same like Site Admin in earlier days of SharePoint versions. You can search organizational site, sort them or filter sites the way you want. Easy way to find and apply/change site settings. Under Policies, click on Sharing - This will allow you to create rules/ set control sharing at the organization level in SharePoint and OneDrive Here, in this page, you can apply settings to restrict how users are allowed to access content in SharePoint and OneDrive. Term store, the word getting populated from SharePoint 2010. You can feed metadata of metadata information of your organization level, program level, project level in Term store. helps your organization's knowledge better.  AI soon will play a vital role here and using AI, you can acyually find collect and process and finally yoour organization's d...
Read more

How to fix Azure DevOps error MSB4126

Image
Setting up build and release pipeline always tedious and we face many build related issues. Here is one common error I face couple of times while setting app Web or windows apps in Azure DevOps After placing the code in repos, and creating the build pipeline and run it, you can face the following error ##[error]D:\a\8\s\<your solution file>.sln.metaproj(0,0): Error MSB4126: The specified solution configuration "release|$(BuildPlatform)" is invalid. Please specify a valid solution configuration using the Configuration and Platform properties (e.g. MSBuild.exe Solution.sln /p:Configuration=Debug /p:Platform="Any CPU") or leave those properties blank to use the default solution configuration. To understand what is target platform defined in your project you need to visit or open the solution(.sln) file and review the following sections. In my case, this has been defined as x86 as target platform.         ...
Read more