Azure, AI, SP, .NET, Microsoft - Knowledge Base

 AI is doing a good job in Databricks platform.  Earlier, following typical mediation architecture, we used to write bunch of code to read the raw data, create a bronze table and then silver table, and finally the gold table.  Now, I open databricks and upload the raw CSV data file. Next, I prompt properly and ask AI to create the code for me.   Code generate within seconds. Ready to execute. However, AI is not always right as I'm. This is acceptable. Code unable to understad the final level aggregration columns there. Good news is AI is there to diagnose the error and recomend the right code. AI, understand what code change is required while doing analysis generated SCHEMA in the dataframe. AI recomending you the right python code to fix this. Let's divide the phase in two sefment. Creating Bronze and Silver table in one segment and Gold one is another code segment. Once the Silver DataFrame generates, print the dataframe schema and review the column before wri...

Databricks project originally initiated by UC Berkeley and then latter known as Apache Spark. It provides all analytics and Big Data processing service. Currently it is part of Microsoft and named as Microsoft Azure Databricks .  We are using Databricks in Azure platform as  a single platform for Big Data processing and Machine Learning. It provides data science and data engineering teams with a fast, easy and collaborative Spark-based platform on Azure.  In this post, I'll walk through how you can connect with Azure storage, read your files and write them in your Azure SQL. Basic Data Engineering steps. Implementing machine learning inelegancy or drill down level data aggregation is out of scope. Let's start by creating Databricks first. To create Databricks, search Azure Databricks in your Azure portal and click on it to create. Once created, you can see this at your Azure portal. Click on it to go to this resource home page. At the home page, click on Launch Wo...