Azure, AI, SP, .NET, Microsoft - Knowledge Base

On-premise data gateway - one of the important feature/service Azure Fabric platform is using to bring your on-premise data into Azure cloud. The data movement is secure and encrypted. You can review my my earlier post  https://splaha.blogspot.com/2025/03/use-on-premise-gateway-in-fabric-and.html   where I explain how to configure On-premise data gateway using Step 1 and Step 2. In this exaple, I'm going to use the same data conenction to pull my on-premise SQL tables using custom join SQL into Lakehouse. The entire post is self explanatory with step-by-step snapshot. I hope this will help you to understand each and individual steps as well as to execute the same at your end. Let's start with my on-premise SQL server details. As you can see the below picture, it talks about the database Employee where there are two tables EMP and DEPT. I'm going to use the same SQL into Fabric to pull the data. I assume you already created the on-premise data gateway using my earlier post...

On-Premise data gateway is an application which is required to be installed in your laptop/computer/server which connect to cloud and uses cloud services to send data back-and-forth from on-prem to cloud. This is secure data connection and data encryption is in place. Here is more about on-premise data gateway - https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-onprem?toc=%2Ffabric%2Fdata-factory%2Ftoc.json https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-onprem?toc=%2Ffabric%2Fdata-factory%2Ftoc.json Architecture of it - https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-onprem-indepth?toc=%2Ffabric%2Fdata-factory%2Ftoc.json https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-onprem-indepth?toc=%2Ffabric%2Fdata-factory%2Ftoc.json Step 1 First thing first - You need to install on-premise data gateway in your lcoal. Second thing is to open the on-premise data gateway and ensure th...

How to load CSV or parquet  file in Microsoft Fabric and create the table to view and work the data. This post is self explanatory :) First login into Micrsoft Fabric. Go to your workspace => Create new item => upload from file => Load CSV file or file => Load to table (Either create new table or update your existing table) Once table created, click on the table to view the data. You can create your custom SQL to fine tune the data. Let's try with parquet format data. Same process: Go to your workspace => Create new item => upload from file => Load parquet file or file.   Once the file is uploaded, click on three dots (...) and click on  Load to table   Simple click on the newly create table and view the data. Again, you have option to create your own sql or notebook to work on this data.

Micrsoft Fabric brings many teams in one platform - It joining Data Engineering, Data Science, and Reporting landscape in one platform.  Lakehouse is new concept in Faric and here it is. Question is why we are going to use Lakehouse while Micrsoft do have existing data storage platform. They do have multiple options in Data Engineering area. I believe Microsoft is now looking to operate on a fully managed compute platform that can support Data Engineering and Data Science experiences - Selecting Apache spark features and services, Microsoft Fabric started it's journey. Fabric do using starter pools. With starter pools, we can expect rapid Spark session initialization, typically within 5 to 10 seconds, with no need for manual setup. Starter pools have Spark clusters that are always on and ready for your requests.  Starter pools are a fast and easy way to use Spark on the Microsoft Fabric platform within seconds. You can use Spark sessions right away, instead of waiting for Spar...