Create Bonze-Silver-Gold-table in minutes in Databricks following mediation architecture
AI is doing a good job in Databricks platform. Earlier, following typical mediation architecture, we used to write bunch of code to read the raw data, create a bronze table and then silver table, and finally the gold table. Now, I open databricks and upload the raw CSV data file. Next, I prompt properly and ask AI to create the code for me. Code generate within seconds. Ready to execute. However, AI is not always right as I'm. This is acceptable. Code unable to understad the final level aggregration columns there. Good news is AI is there to diagnose the error and recomend the right code. AI, understand what code change is required while doing analysis generated SCHEMA in the dataframe. AI recomending you the right python code to fix this. Let's divide the phase in two sefment. Creating Bronze and Silver table in one segment and Gold one is another code segment. Once the Silver DataFrame generates, print the dataframe schema and review the column before wri...