Microsoft Fabric on Hat Full of Data

Using Variable Library in a Dataflow

Wed, 11 Feb 2026 17:21:24 +0000

One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.

I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!

Variable Library Series

Variable libraries should be part of every project. This post is part of my series to help get you started creating the library and then using the variables and finally seeing your hard work pay back when it comes to deployment pipelines.

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Scenario

For this post we are going to use the scenario of a dataflow query that lists projects. Whilst we are developing the data handling we want to limit the number of rows loaded and in production we will have no limit.

I create the dataflow and then using Keep Rows, I select Keep top rows. In the next dialog I enter 5 and click OK. This results in a table with only 5 rows of data. The number of rows though needs to come from the variable library.

Getting the Value

The next step is to get the value from the variable library into the dataflow. We are going to use a new Power Query function for this called Variable.ValueOrDefault. For the first parameter of this function, you need the Variable Library name and a Variable name. Then you combine together in a string.

1

"$(/**/<Library Name>/<Variable Name>)"

So for my example the string will be “$(/**/Finance Variables/Limit)”.

📝 Instructions

On the Home ribbon, expand Get data and select Blank query.

Rename the query to DataRows.

Then in the formula bar enter in the code below.

This will return the current value of the variable in the library.

1

Variable.ValueOrDefault("$(/**/Finance Variables/Limit)",2)

💡 Tip

The documentation as of publishing this post states it won’t work hence we use the OrDefault function and in this example the default is 2. It does work though, and we can see the answer 8 comes through.

Using the Variable Value

We’ve got the variable value from the variable library in a dataflow, now we need to use it. In the last line of my project query that limited the query to 5 rows I can replace the 5 with DataRows. I now get 8 rows of data.

This works as long as we always want to limit the rows, the chances are in production we don’t want to limit, so I add the extra of only limit if DataRows is greater than 0. Here is my new statement, the previous step is call Sorted Rows hence the #”Sorted rows”

1

if DataRows > 0 then Table.FirstN(#"Sorted rows", DataRows) else #"Sorted rows"

References for Variable Library in a Dataflow

Microsoft – Use Fabric variable libraries in Dataflow Gen2

Power Query Reference for Variable.ValueOrDefault

Conclusion on using variable library in a dataflow

Its great we can bring in the values easily. Its a shame that we can’t use them to control the destination, but that is on the road map for 2026 Q1. I’ll blog about it as soon as it arrives! I highly recommend using my pattern of fetching the value as a query and then referring to that. It will help in debugging etc.

Using a Variable Library in Lakehouse Shortcuts

Mon, 26 Jan 2026 18:43:51 +0000

Lakehouse shortcuts are a popular addition to the Fabric set of tools to access data easily without copying it. Using a variable library in lakehouse shortcuts means its easy to point shortcuts to an alternative location. This great for ALM using development, test and production workspaces.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Setting Up the Shortcuts

Using the standard GUI you have no option to use a variable library values in the setup of the shortcuts. So set up your shortcuts as you normally and make sure they work. For this post I’m using a shortcut to a table in warehouse. (I had a whole barny with lakehouses and schemas and it not working – another post for another time!)

According to the documentation using the rest API also does not allow for the use of variables in the creation of shortcuts.

Viewing the Options

If we select the short cut and click on the 3 dots to expand the menu we can then click on Manage Shortcut. This opens a pan that shows the values behind the shortcut. The values that be populated from a variable library have a Variable drop down, so we can see Target workspace, Target item and Target path.

I only want to populate the workspace and item guids, so I set up a those variables in my variable library. The variables must be set up as strings, they might look like guids but the shortcut demands they are strings. I set the default values to match the current values. When we apply each variables it will check the current values work so it only works if you use the current values.

Applying the Variables

Back in the Manage Shortcut pane click on the Variable below the current value to expand that section. Then click on Select variable to open the Select variable dialog. Click on the correct variable and then click Select Variable. This will update the value to come from the variable.

Once you have pointed it to the variable there does not appear to be a method to change back to a hard coded value.

Updating Variable Values

Variable values will get updated, the most common being a different active set being selected. For other artifacts the change will get picked up when they execute or refresh. For shortcuts this is slightly different. You need to force the updating of variables.

Open the lakehouse and stay in the lakehouse view. On the ribbon the last button is Update all variables. A message will show the updating is taking and finally there will be a notification to say they have been updated.

If one of the shortcut fails in the update please be aware the successful ones will have updated. You do get a notification and an error message that tries to help.

Conclusion on Variable Library in Lakehouse Shortcuts

I really like that the variable libraries can be accessed directly in the shortcut details, there is no pre-loading. I would prefer that I could change the values and then press apply so that each value isn’t checked individually. My experience in deployment pipelines means I fell foul of not realising I needed to refresh the variable values.

Resources

Creating Shortcuts – https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-shortcuts

Microsoft Learn – https://learn.microsoft.com/en-us/fabric/onelake/assign-variables-to-shortcuts

Using a Variable Library in a Data Pipeline

Mon, 19 Jan 2026 10:36:26 +0000

You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Adding the Variables to the Data Pipeline

Before you can use the variable library in a data pipeline you need to add the variables. In your data pipeline you need to click on the grey background to bring up the section on the bottom of the screen with Parameters, Variables etc. If you are in a brand new pipeline you need to click on the settings cog first, then it will appear.

Select the Library Variables section. Then click on +New and the select variable dialog box appears. If you have multiple variable libraries you can select the right one, you can also search for a variable in the top right. Once you select a variable click on Select Variable button to add that variable to the pipeline.

Yes you have to add each variable in turn, no multi-select. You can rename the variables but please don’t, name your variables in your library better.

Using Variable Values

You can use the variable values anywhere you can use Dynamic Content. Click on the Add Dynamic content link to open the Pipeline expression builder. Click on the three dots to select Library variables. Click on the variable value you want to use to enter in the formula into the expression. Click OK to put the formula back into the box.

You can use the variable value inside any functions if required.

Using Variables for Data Connections

When a data pipeline move from a dev workspace to test workspace and then onto production workspace we need to change the details of the connections to the new values. For this post we will use a connection to a Fabric warehouse in a workspace. I want to pull the values from a table in the warehouse using a lookup action.

I need to create the following variables in the library:

Project_Warehouse_Connection_ID
Project_Workspace_ID
Project_Warehouse_ID
Project_Warehouse_Connection_String

Connection IDs

For the connection guid you need to click on the cog on the top bar and select Manage connections and gateways. From the list of connections, find the one you are using. From the three dots menu select settings. The Connection ID will be listed and can be highlighted and copied.

Workspace, Warehouse and Lakehouse IDs

IDs for most artifacts can be found when it is opened. Be aware groups=workspaces in the url. Lakehouses have 2 IDs, one for the lakehouse and one for the SQL analytics endpoint or lakewarehouse as the url names it. No comment on names to confuse us!

Connection String

The final part needed is the connection string for the warehouse or lakehouse. The easiest way to copy this is to open the lakehouse SQL endpoint or the warehouse and in the bottom left hand corner is a Copy SQL connection string link.

It can also be found in the lakehouse or warehouse settings under SQL analytics endpoint.

Variable Library

All these values need saving into the Variable Library. I’ve used Guid for the IDs and String for the connection string.

Using the Variables in the Activities

We now need to use the variable values from the variable library in a data pipeline activity connection. We start by adding all 4 variables to the library. Note the guids become strings.

In this post for simplicity I’m using a Lookup activity to just get all the rows from the projects table. So I add a lookup activity and fill in the general details. On the Settings tab I click the drop down for Connection and select Use dynamic content. In the Add dynamic content pane I find Library Variables under the three dots and select FinanceVariables_Warehouse_Connection_ID so it appears in the expression box. Then I click OK for the Connection to now be an expression.

Once the connection has been populated you can select the connection type, in my case Warehouse. Then the text boxes appear to take the Workspace ID, Warehouse ID and SQL Connection string. They all have a link to Add dynamic content.

The table drop down can be populated by entering manually or by clicking Refresh. This will open a pane showing the values that are going to be used in the connection. Click OK to start it fetching the list of tables. Once a table is selected if you click Preview data the same pane appears confirming values. Its a great way to test the connection works.

You now have a variable values from a variable library in a data pipeline activity. Well done!

Conclusion on Using a Variable Library in a Data Pipeline

I like how simple it is to use the variables and that you get to select the variables needed in this data pipeline. I’d prefer if you could add multiple variables at a time. I like the consitent use of the expression builder so its something we get familiar with.

I’d also really like it if in lineage view you could see which artifacts are using of the variable library and the impact was also included. Maybe oneday!

Resources

https://learn.microsoft.com/en-us/fabric/data-factory/variable-library-integration-with-data-pipelines

Accessing a Variable Library in a Notebook

Mon, 12 Jan 2026 09:34:51 +0000

This post walks through how to access a variable library in a notebook in Microsoft Fabric. I recommend a Microsoft Fabric project starts by creating a variable library to store the common values different artifacts need and could be changed if a deployment pipeline gets involved. So when we create a notebook we need to be able to use these variables. This means we need load the variable library in a notebook and then get the variable values.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Connect to Library

The first task is to create a variable for the variable library. The notebookutils package includes the functions we need to do this easily. In the code below the variable vl will refer to the variable library that is called “Finance Variables”. This code assumes the notebook is in the same workspace as the library.

1
2


# Get the Variable library
vl = notebookutils.variableLibrary.getLibrary("Finance Variables")

Get Variable Value

Once we have the variable library loaded, we can access the variable values in one of two syntaxes.

1
2
3


# 2 syntaxes to get variable value
SP_URL = vl.getVariable("SharePoint_URL")
Limit = vl.Limit

Combining both of the two and adding some print statements we can demo the above using this code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Get the Variable library
vl = notebookutils.variableLibrary.getLibrary("Finance Variables")

# 2 syntaxes to get variable value
SP_URL = vl.getVariable("SharePoint_URL")
Limit = vl.Limit

# Check results
print(f"{SP_URL=}")
print(f"{Limit=}")

And here is the run

Get a variable value by reference

There is a third way to get a variable value. The get method from notebookutils variable library uses a reference string that follows the following syntax:

1

"$(/**/<LIBRARY NAME>/<VARIABLE NAME>)"

Before you get excited that the ** implies you could refer to another workspace, sorry that is not supported. This string is used in the variableLibrary.get action in notebookutils. For example to get the two values from before could be done like this

1
2
3


# Get the variable values directly
SP_URL = notebookutils.variableLibrary.get("$(/**/Finance Variables/SharePoint_URL)")
Limit = notebookutils.variableLibrary.get("$(/**/Finance Variables/Limit)")

And here is the run

💡 Tip

The print(f"{Limit=}") prints Limit=4 for us newbies to Python!

Naming Conventions

When you start and there are only 4 variables in the library its easy to remember the names to type in by hand when you are using a variable library in a notebook. When it gets to 20 names, its harder. So create standards, whats the case pattern, you using _ between words or not? Pick a set of rules and stick with it, BronzeWorkspaceID works just as well as Bronze_Workspace_ID, WorkspaceID1 and workspace_id2 are a technical debt you don’t need.

Conclusion on using Variable Library in a Notebook

I’m really impressed on how easy it is to use the variable library in a notebook. The pattern is simple and resusable. If I had a magic wand it it would be great is the intellisense knew the variable names from loading the variable library object.

The method by reference is great if you want to dynamically select which variable to load as the string could be dynamically built.

Using a combination of parameters and accessing a variable library means reusable notebooks are easier to write. This needs to become part of the best practice pattern being used.

References

Microsoft learn covers Variable library utilities as part of the notebook utilities page.

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#variable-library-utilities

Microsoft Fabric – Variable Library

Tue, 06 Jan 2026 13:27:41 +0000

This post to help you get started creating a variable library. When multiple dataflows, notebooks and pipelines are using the same details to perform tasks it helps if those values are stored in one place. When you move to use deployment pipelines and those values change from your development workspace to your test workspace to your prod, it helps if that is easy. The solution in Microsoft Fabric is a Variable Library to store those common values.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Create a Variable Library

This post assumes the workspace is backed with a Fabric capacity. A trial capacity works just fine.

Either from a Develop task add item or from the Add item button select Variable Library. Enter in a meaningful name for the variable library, letters, numbers and spaces are allowed. Then click the Create button and your new library will be opened. I recommend you create a naming convention that all your Fabric developers like and agree to use.

Add a Variable

Your library starts empty, so the next task is to add variables. You can create a new variable row by clicking on + New Variable on the Home ribbon. When there are no variables there will also be a green button in the center of the page to create a new variable.

Enter in a meaningful name and select a data type. A name can only include letters, numbers and underscores. An incorrect name will prevent the library from being saved. Please note some types are not usable in some Fabric artifacts, no I haven’t worked out the logic yet but I’m working on it. Enter in a value for the Default value set. Value sets are used when you are using deployment pipelines for the different values you use development,test and production workspaces. Creating and using variable sets will be covered in another post.

Adding Notes

So that “future you” doesn’t hate the “current you”, it is worth adding notes to variables to explain the purpose of the variable. This should be part of the best practice rules for Fabric development.

Click on the icon in the note column. When there is no note it has a plus symbol and when there is a note it has a pencil symbol.

Saving

Variable libraries do not auto-save. You need to click the save button on home ribbon. Indicators of black dots will show unsaved items you have open and the save button will only be active when there are changes to save.

Conclusion

I highly recommend starting any Fabric project with creating a variable library. Then everytime a value is used in an artifact such as a database name, key vault name etc a descision is made on if it should be in the variable library or if the value is already there. Some values to be more useful will need to be split across multiple variables.

It should always be assumed that a workspace with Fabric items will eventually be in a deployment pipeline. This means that it should be coded to make deployment easier. The aim of these articles is for quick reference on how variable libraries are used across all the artifacts.

References

Microsoft have some getting started articles on setting up a variable library and using it in a deployment pipeline.

Microsoft Learn Variable Libraries

Microsoft Fabric – Using Copy Job with Dataverse

Thu, 27 Nov 2025 15:35:41 +0000

Dataverse is the data store behind parts of Dynamics and lots of Power Platform projects. So Dataverse can contain vital business data that will be needed for reporting. In this post we are going to look at one method which is using copy job with Dataverse to copy across data in Microsoft Fabric.

Scenario
First Copy Job – Dataverse Connector
Copy Job Run
Copy Job using SQL Server connection
Incremental and Merge Running
Trouble Shooting
Alternative Options to using Copy Job with Dataverse
Conclusion

Scenario

For this post we will look at 2 tables, Accounts and Contacts so all the code will be common to any Dataverse environment. I have 5 Account records and 10 Contact records. I also have the environment path org29d4c6d3.crm11.dynamics.com.

In Microsoft Fabric I have a workspace, Finance Reporting, with a Lakehouse, Finance_Lakehouse.

First Copy Job – Dataverse Connector

Copy Job is a great artifact to move data easily into Fabric. Simplest version just copies data, we can then add incremental and merging. More details can be found here https://learn.microsoft.com/en-us/fabric/data-factory/what-is-copy-job

For the first example we will copy from Dataverse using the Dataverse connector. The wizard that walks you through creating a Copy Job has 6 steps. For this first one we will walk through each step.

Create Copy Job

Either using a task flow Get Data’s add item or using Add item in the workspace, select Copy job. Type in a name and click Create.

Choose Data Source

Next step is connecting to Dataverse. For this example we are using the Dataverse connector. Click on the Dataverse option under New sources. When the next screen appears enter in the Environment domain. If it is your first time connecting there you will be given the option to name your connection and select a privacy level. It will guess the authentication and login – this is not the blog to discuss options here.

Once these are done click Next

Choose Data

Choosing the data is very simple, the next screen lists the tables from dataverse using their display names and logical names in brackets. Yes they call them Entities…. No comment. If you expand a table the logical names of fields are listed. There is a search box that will allow you to search for a table using either of the names. If you have selected a table the search will expand to the field names as well. Interestingly for choice columns you only get the code column, not the name column. That is going to be another post!

If you select a table all the columns will be imported. You select the required columns manually by selecting individual columns, its a touch manual and requires searching and scrolling lots in my experience. Next version is better in my opinion – feel free to scroll down!

When you’ve selected your tables and columns click Next.

Choose Data Destination

Data destination is another easy step. Select your lakehouse from the OneLake catalog.

Choose copy job mode

With the dataverse connector you only get one option, Full copy. In the Map to destination you get to choose if that is an append or overwrite.

Map to destination

The next screen shows update method and the tables being imported. Clicking on Edit update method will give you the option of Append or Overwrite. Append is the default but this will just duplicate you data every time you run this copy job. I have changed it to Overwrite. Merge is not available in this version.

You can enter names for your destination tables and click Edit column mappings to tweak column names or remove some mapping. The columns selected will match the columns you selected back in Choose data.

Click Next once you are happy.

Review + save

The final screen will give you an overview of what the copy job is going to do. You can select if you want it to run now, run once or scheduled. I’m choosing to let it run so I’ll just click Save + run

Copy Job Run

When a copy job runs such as after saving in the previous steps you can see the execution happening. Each table is listed and when it succeeds you will see how many rows were read and how many were written. In our example read and written are the same.

Once the run has completed we can go check the Lakehouse and we can see tables. It might need a refresh to show.

Copy Job using SQL Server connection

Although the dataverse connector works, the selection of columns is very manual. You also get no option for incremental updates or using merge. So the next example will use SQL Server connector. In a previous post https://hatfullofdata.blog/using-sql-on-dataverse-for-power-bi/ I show how you can use SQL on Dataverse with some great hints from Scott Sewell.

From that post and with the addition of the modifiedon field here is the SQL for the accounts table. We will use this later.

1
2
3
4
5
6
7
8
9


Select
	accountid as AccountID,
	name as Account,
	industrycodename as Industry,
 modifiedon,
	statecode,
	statecodename
from
	dbo.account

New Copy Job using SQL Server

Create a copy job as before giving it a name. For the connector select SQL Server. The Server is the environment path as used before, e.g. org29d4c6d3.crm11.dynamics.com. Remember to edit the connection name and privacy level. There is no point adding a Database, it makes no difference. After you press next you need to wait for the database list to load and then select the only database offered.

Choose Data

The next screen for choosing the data starts by showing you an error. Ignore that, it means you can’t use the interface to select your tables and columns but you can add queries. Expand Queries in the bottom left and click on New query. Type in a name, eg Accounts and then in the query box on the right put in your query. You can check it returns the right data by clicking on preview data.

Repeat the above for each of your tables and then click Next.

Incremental and Merge setup

When you choose your lakehouse in the data destination and move onto the copy job mode screen you now get 2 options, Full copy and Incremental copy. The incremental copy will initially copy everything and then in the future will only copy across changes. In order to recognise which records have changed you need a data time column of when it last changed. Hence we have included the modifiedon column.

So incremental is only going to process updated or new rows, but what do you want done with those new rows? Append will add the newly updated rows giving you the complete history of each row. Overwrite you will loose rows that have not been modified. Merge is currently in preview but would allow you to update the row that matches on a unique field, e.g. Account Id.

Click Incremental and then Next. In the Map to destination screen, click on Edit update method and select Merge and then select the unique id column for each table. When you click next you will be prompted to select the incremental date time column for each table. Click next to the final stage.

Save, Review and Schedule

On the final screen not only do you get to save and review but you also get to schedule due to us picking incremental. You can pick repeat by the minute or hourly and pick dates etc. Be aware incremental requires more compute so will consume compute faster than a straight copy but on huge tables the reduced copy will be worth it.

Select your options and Save and Run

Incremental and Merge Running

The first time we run just as before the rows read and written include all the data. If we go and edit one row in the accounts table and re-run the copy job, only one row is read and written for the accounts table and nothing for the contacts table.

Trouble Shooting

In preparation for this post I’ve hit a number of issues.

Although there is a back button in the wizard, going back to the data source and tweaking the queries just breaks the copy job.
You name a column in the SQL with a space, that update will fail with a weird error. No spaces in column names!

Alternative Options to using Copy Job with Dataverse

Link to Microsoft Fabric

Using the Link to Microsoft Fabric in Dataverse sounds like a great option. You will be able to add shortcuts in a Lakehouse to the data and it will all be awesome.

Just after you select the workspace that the data will go to there are 2 short sentences that you should read carefully.

“We will link all tables so that you don’t have to choose. You may incur additional charges.”

I’ll not comment on the “so you don’t have to choose” as I’m not sure I have anything polite to say. That additional storage is in Dataverse which could cost $40 per 1GB per month and with all the tables that could easily become a huge expense, even empty tables take up space. All this so we can have a shortcut in Fabric.

Platform Guardian blog has a great post to walk you through doing thishttps://platformguardian.blog/2023/11/05/creating-the-dataverse-connector-in-fabric/

Conclusion

Copy Job is one of my favourite Fabric artifacts because they are simple to set up. Editing them is foul so I end up re-building them. Its a brilliant low-code solution to copying across data from dataverse and into a lakehouse. Be aware incremental is going to eat capacity so don’t bother with small tables just do a full copy over writing every time.

I don’t see any benefit using the dataverse connector except you don’t have to write SQL. A simple select statement is not complex I’m sure an AI prompt somewhere will help you out.

Microsoft Fabric – Direct Lake and User Permissions

Fri, 07 Nov 2025 19:40:06 +0000

Direct Lake is the best way to build a semantic model on data in a Lakehouse or a Warehouse. We can then use Power BI to build reports and possibly even a make a Workspace App to deliver those reports to the business. So what permissions do the business users need to view those reports? This post will explore using Direct Lake and User Permissions to make sure everyone can see data in the reports.

IMPORTANT

This is the simplest method of fixing permissions so someone can view a report using direct lake. If your lakehouse or warehouse contains data they should not have access to this is not the method to use.

Scenario

For this post I have built the following :

Workspace that has a workspace app built and up to date.
Lakehouse_Report connected to a semantic model, Lakehouse_Model, that has a direct lake connection to a lakehouse, Permissions_Demo_Lakehouse
Warehouse_Report connected to a semantic model, Warehouse_Model, that has a direct lake connection to a lakehouse, Permissions_Demo_Warehouse
Both the Warehouse and Lakehouse contain one table, Projects with 10 rows of data. Both reports show this in a chart.

I have given one user, Dan, Vistor access to the workspace and I’ve given the Project team security group access to the workspace app.

Both reports work fine for me as the workspace owner. The report is direct lake and user permissions are admin.

Viewing Reports as Dan

Viewing the report as Dan is an example of a report that is direct lake and user permission is viewer. Dan can see the warehouse data but he cannot see the lakehouse data.

Viewing Workspace App as Sam

Viewing the report as Sam, who is a member of the Project team security is an example of a reports that are direct lake and user permission is app only. Sam cannot see either the lakehouse or the warehouse report charts.

Permissions

Whenever data is not coming through for some but not others it is usually an issue with permissions somewhere. Everyone can see the report, they get the headings. The connection is direct lake into the semantic model we need to look at user permissions where the data lives, the lakehouse and the warehouse.

Each item in the workspace has permissions. By default permissions are inherited from the workspace. There are two fixes to do. The first is Dan’s access to the lakehouse and the second is project team access to both lakehouse and the warehouse.

Fix Dan’s Lakehouse Access

In the workspace, click on the three dots on the lakehouse. Then from the menu select Manage permissions. This shows a list of the people with access and the Role which gave them that access. We can see Dan has Read and ViewOutput

Read only gives the user permission to connect to the lakehouse, it does not let them see the data. Neither does ViewOutput. I’d love to find documentation as to what ViewOutput allows but I’ve not found it yet. I will update this post when I do.

Click on the three dots at the end of permissions and select Add ReadAll. Now when Dan looks at the reports the Lakehouse one will work. Sometimes it takes 10-15 minutes for the permissions to work..

Fix Sam’s Access via the Workspace App

Sam gains access to the workspace app through being a member of the Project Team security group. The security group is not mentioned in the permissions of the lakehouse or warehouse. If you go look it is listed in the permissions of the semantic model as App.

In the lakehouse permissions, click on +Add user. In the Grant people access dialog add the user group or people into the box. Decide if you are emailing them and add a message, I’m not emailing anyone an auto generated email! I have not ticked any of the additional permissions. Then click Grant.

This will give them Read access. Then as we did for Dan, click on the three dots and click Add ReadAll and this.

Warehouse permissions are slightly different. Dan has Read and ReadData permissions which gives enough access for him to view the report. So we can use the same process as on the lakehouse and duplicate those permissions. Note that in the Grant people access dialog no additional permissions were ticked.

And after a few minutes when Sam refreshes the Workspace App they can see the reports both working.

Note!

The permissions given do mean that if Sam now has access to run SQL against the Warehouse but not the Lakehouse which is behind the semantic model so any row level security in the model will not apply. They do allow Sam to explore the data in both the semantic models.

Resources

The reason I wrote this blog post was I couldn’t find a resource to show the simple fix of manage permissions. Lakehouse sharing and permission management – Microsoft Fabric | Microsoft Learn is the closest I got.

Conclusion

Giving someone access to the whole lakehouse or warehouse could be dangerous. This post gives the simplest of fixes assuming the report viewer is allowed to see all the data with a report that is direct lake and user permissions are read only in the workspace or app only.

With new security features now available we can make the permissions fix more complex requirements. Handling Direct and User permissions needs careful consideration and should be discussed as part of your data governance.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

Lewis and Laura vs. Notebook and Microsoft Graph: Part 2

Tue, 26 Aug 2025 13:07:20 +0000

A recent project made me reach out to Lewis Baybutt to scream “Help!, I need help getting with authentication for an Excel file in SharePoint in a Microsoft Fabric Notebook”. Yes all the delights of Excel and SharePoint in one project. So started the battle with understanding authentication, permissions and apparently a Notebook and Microsoft Graph is the way.

The reason for the project was the original solution was using Gen2 Dataflows and we ran out of capacity. Datameerkat has done a nice summary of comparing Dataflow to Copy Activity to a Notebook, https://datameerkat.com/copy-activity-dataflows-gen2-and-notebooks-vs-sharepoint-lists. Its for lists but the same logic works for files in SharePoint. So for this project we selected the Notebook route.

Final Goal of Notebook and Microsoft Graph

On a SharePoint site there is an Excel file that contains some data I want to load into a table in a Microsoft Fabric Lakehouse. The Pandas library includes a method called read_excel that can pull the data into a dataframe if it has the file content. And we know how to write a dataframe into table, so the real goal is to get the file content of a file on a SharePoint site.

We save the details of file name etc into variables to use later.

1
2
3
4
5


sharepoint_domain = "YOURDOMAIN.sharepoint.com"
site_name = "FabricDemo"
library_name = "Documents"
file_name = "PandaDemo.xlsx"
sheet_name = "Demo 1"

Service Principal, Permissions Etc

The first part of this series is definitely not my skill set, it involved Power Shell! So we agreed he would write the first part and I would write part 2. So Lewis has covered that topic in Part 1 which can be found here https://www.lewisdoes.dev/blog/lewis-and-laura-vs-fabric-notebooks-and-microsoft-graph-part-1/

The credentials created in the post should be stored in Azure Key Vault. I wrote a blog post how to do that and another one how to retrieve it into notebook

I created a code block to create three variables, tenant_id, client_id and client_secret. Swap in your TENANT ID etc to make it work for you. The notebookutils library is always loaded in a Microsoft Fabric Notebook.

1
2
3
4
5
6
7
8


# Authentication details
tenant_id = "TENANT ID"
client_id = "CLIENT ID"
# Get secret from Key Vault
azure_key_vault_name = "VAULT NAME"
azure_key_vault_secret_name = "SECRET NAME"
azure_key_vault_url = f"https://{azure_key_vault_name}.vault.azure.net/" 
client_secret = notebookutils.credentials.getSecret(azure_key_vault_url,azure_key_vault_secret_name)

Requests Library

This whole post is lots of HTTP get requests made way easier using the Requests library. Documentation for this library can is here Requests: HTTP for Humans™ — Requests 2.32.5 documentation

We use two patterns very similar, obviously there is a import requests statement in a previous code block.

1
2
3


reponse = requests.post(url , data=token_data)
# Raise error if request fails
response.raise_for_status()

1
2
3


reponse = requests.get(url , headers=headers)
# Raise error if request fails
response.raise_for_status()

Get Access Token

Before we can use Notebook and Microsoft Graph together we need an access token. For all the requests.get we need headers which includes an access token. So the next task is to get that access token which uses the first pattern from above. If you’ve used the tenant_id, client_id and client_secret variables you can just copy and paste this code into a block

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


token_url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token"
token_data = {
 "grant_type": "client_credentials",
 "client_id": client_id,
 "client_secret": client_secret,
 "scope": "https://graph.microsoft.com/.default"
}
response = requests.post(token_url, data=token_data)
response.raise_for_status() # Raise error if request fails
access_token = response.json().get("access_token")

# Print the result
print(" Access Token Received:", access_token[:50], "...")

headers = {"Authorization": f"Bearer {access_token}"}

The last step is to put the access token into a JSON object called headers which will be used for the next stages.

Getting the Site ID and Drive ID

We need the url to download the file content. The request to get that url is

https://graph.microsoft.com/v1.0/sites/{site_id}/drives/{drive_id}/root:/{file_name}:/content

So we need the site_id and the drive_id. Btw drives are document libraries… yeah naming confusion, don’t blame me! We already have the file_name, if its in a folder the name needs to include that.

Site ID

Using https://graph.microsoft.com/v1.0/sites/{sharepoint_domain}:/sites/{site_name} with the headers we built earlier in a requests.get we get a json response that includes an id value

1
2
3
4
5


site_id_url = f"https://graph.microsoft.com/v1.0/sites/{sharepoint_domain}:/sites/{site_name}"
print("Site ID URL:",site_id_url)
response = requests.get(site_id_url, headers=headers)
response.raise_for_status() # Raise error if request fails
display(response.json())

Produces

So we then use the following code to get the site_id and print it for debugging.

1
2


site_id=response.json()['id']
print("Site ID:",site_id[:50], "...")

Drive ID

Using https://graph.microsoft.com/v1.0/sites/{site_id}/drives?$select=name,id will give us a list of the document libraries on the site. A site can have more than one library so we need to handle multiple rows being returned. My choice was to save it into dataframe and then filter that to get the result.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


drive_id_url = f"https://graph.microsoft.com/v1.0/sites/{site_id}/drives?$select=name,id"
response = requests.get(drive_id_url, headers=headers)
response.raise_for_status() # Raise error if request fails

# Convert response json into a dataframe
df_drives = spark.createDataFrame(response.json()['value'])
display(df_drives)

# Filter the dataframe to the specified library and get the id
drive_id = df_drives.filter(col("name")== library_name).collect()[0]["id"]
print("Drive ID:",drive_id[:25], "...")

That gives this output

So now we have the site id and the drive id so now we can get the file content using our Notebook and Microsoft Graph.

Get the File Content

This uses the same pattern as the previous 2 requests, the difference is the path includes /content on the end to indicate we want the content. There is very little we can print out to debug the response as the file contant is binary

1
2
3
4
5


# Retrieve the File Content from SharePoint using Graph API
file_url = f"https://graph.microsoft.com/v1.0/sites/{site_id}/drives/{drive_id}/root:/{file_name}:/content"
print("File URL:",file_url[:75], "...")
response = requests.get(file_url, headers=headers)
response.raise_for_status() # Raise error if request fails

Pandas and File Content

Pandas read_excel method wants a file type to work with. Using BytesIO from io library we can create that. Documentation for BytesIO can be found here https://docs.python.org/3/library/io.html. We can then use read_excel from Pandas library to fetch the data from the sheet. I’ve kept it simple, the tricks on handling more complex data in Excel is for another post.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Import methods
from io import BytesIO
from pandas import read_excel

# Convert response
xls = BytesIO(response.content)

# Get data from sheet
df = read_excel(xls, sheet_name=sheet_name) 
display(df)

From the above code we get the following in a panda dataframe ready to save into a table.

Final Notebook

The complete notebook that could be imported into a Fabric Workspace can be found here https://github.com/HatFullOfData/CodeStuff/blob/main/Sample%20Files/Access%20SharePoint%20using%20Microsoft%20Graph.ipynb

Conclusion

Congratulations on getting this far with Notebook and Microsoft Graph. The above looks like hard work when connecting from a Gen2 Dataflow is so much easier. But the saving on capacity usage and the flexibility of a notebook I think gives us more options in the future.

Get Secret from Azure Key Vault in a Microsoft Fabric Notebook

Thu, 21 Aug 2025 16:33:55 +0000

I’ve recently been working with Notebooks in Microsoft Fabric and have needed to get a credentials information that should be kept secure. The obvious place to store such credentials is Azure Key Vault so now we need to get our secret from there.

This is a really short post to help me out when I need a fast explanation on projects.

Azure Key Vault Part

I’ve previously blogged how to setup a key vault and add a secret – https://hatfullofdata.blog/create-azure-key-vault-to-store-id-and-secret/.

For this post I have a vault called LGB-Secrets and it has a secret called SPConnectorSecret.

The vault URI is https://lgb-secrets.vault.azure.net/ which can be calculated from the vault name.

Python to Get Secret from Azure Key Vault

Notebookutils is a built in package for Microsoft Fabric notebooks. It was previously known as MSSparkUtils. It contains plenty of really useful methods to perform tasks with files, environments and credentials. Microsoft’s documentation can be found at :

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#credentials-utilities

The method notebookutils.credentials.getSecret requires 2 parameters, vault URI and secret name. This code block looks like this.

1
2
3
4


azure_key_vault_name = "lgb-secrets"
azure_key_vault_secret_name = "SPConnectorSecret"
azure_key_vault_url = f"https://{azure_key_vault_name}.vault.azure.net/" 
client_secret = notebookutils.credentials.getSecret(azure_key_vault_url,azure_key_vault_secret_name)

The account being used to run the notebook will need access to the key vault.

Reacted!

The notebook understands that the value you fetched from the key vault should be kept secure. So if you try to print the client_secret it will not show you the result and will show REDACTED instead.

Conclusion

I know this is a very short post, which is a good thing as that means its a simple concept. It is here because I also wanted a short post that contains the code block I want to be able to copy and use easily.

Microsoft Fabric – Create a Data Pipeline with Parameters

Thu, 05 Jun 2025 10:26:00 +0000

One of the skills of creating good solutions on any platform is building reusable modules. For a part to be reusable we reduce the hard coding and pass in parameters. This is true for data pipelines. So this post post covers creating a data pipeline with parameters.

Initial Scenario

In order to demonstrate how to create a data pipeline with parameters we need a simple data pipeline as a demo. For this post we are going to use the Star Wars api called SWAPI. The details of the api can be found at https://swapi.info/. Its a very simple api that allows anonymous connections and returns data in a JSON format.

In our pipeline we add a copy data activity. For the Source, the connection is a http connection to https://swapi.info/api/ and the relative URL is the table name, in this case films. And the file format is JSON.

For the destination the connection is to a Lakehouse. The Root folder is files with the file path being the table name + .json so in this case films.json. To match the source the file format is JSON.

The issue with the above is the “films” is hard coded. If this pipeline is going to be reusable for other tables from this api we need to move the “films” into a parameter.

Adding Parameters to the Data Pipeline

In the data pipeline click on the grey background around the activities to make sure no activity is selected. In the bottom panel you should now see options including Parameters. Click on the + New to add a parameter. Then add a name, select a type and give a default value.

Multiple parameters can be added. When the data pipeline is run the parameters can be updated. You now have a data pipeline with parameters you just need to use them in activities.

Using Parameters in Activities

We now need to edit the copy data activity to use the TableName parameter rather than the hard coded films. Edit the copy data action and on the source tab clear the Relative URL value. Click on the Add dynamic content or Alt+Shift+D. Then in the panel, under Parameters click the parameter, in this case TableName. This puts the formula in the expression builder. Then click OK to put the expression back into the Relative URL.

The above example was using the parameter straight. For the file name in the destination we need to concatenate “.json” onto the table name. In the expression builder Functions such as concat can be found in the Functions tab and in the right category group.

If you know the function name that you want to use there is intellisense and typing in “@conc” will give you a list of matching functions to use.

Clicking OK will add the expression to the filename box.

Testing the Data Pipeline with Parameters

When the pipeline is run manually a panel appears prompting for the parameter values. Enter in the values and click OK. When the pipeline completes the run successfully we can see in the lakehouse the new file has been created.

Conclusion

I first learnt best programming techniques over 30 years ago. It included concepts I still use and teach today. This is one of them. Of course the pipeline could be more complex, include more parameters and more activities but the concept is still the same, make things reusable by creating data pipeline with parameters.

First refresh it broke!

Mon, 12 May 2025 15:57:47 +0000

I set up my dataflow and it was working great. Then it broke on the refresh. Here is how I fixed it.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

Previously

In the previous post we ingested a SharePoint folder into a table in a Lakehouse. That is a well known process and nothing new except for one thing. I split the process into two stages, firstly the FullFileList which contained the list of files and the binary content of the files and then a separate query to combine data from those files. Power Query created Helper queries to perform the combining. Then it broke.

It Broke

It all appears to work fine. Then I try a refresh and it fails. On going back to look at the dataflow I see that there are 2 extra steps in FullFileList that have been added by Microsoft Fabric.

1
2
3
4
5
6
7


let
 Source = SharePoint.Files("https://lgb123.sharepoint.com/sites/FabricDemo/", [ApiVersion = 15]),
 #"Filtered rows" = Table.SelectRows(Source, each [Folder Path] = "https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/"),
 #"From Value" = Table.FromValue(#"Filtered rows"),
 #"Remove Columns" = Table.RemoveColumns(#"From Value", Table.ColumnsOfType(#"From Value", {type table, type record, type list, type nullable binary, type binary, type function}))
in
 #"Remove Columns"

The last step Remove Columns, removes all columns of type table, record, list, binary or function. So this removes the Content column that is binary and therefore the queries doing the combine of the Content column all fail. Hence it broke.

It Broke in the Staging

To improve performance and reliability, Dataflow Gen2 uses staging items to store intermediate data during data transformation

The above quote is from https://learn.microsoft.com/en-us/fabric/data-factory/data-in-staging-artifacts It stores the intermediate queries, so FullFileList would have been stored as a table in the staginglakehouse or stagingwarehouse. So Fabric is adding the step to remove the binary column so staging can work.

Turn Off Staging

When I right click on the FullFileList query in the query list the menu includes Enable Staging, by default it is turned on, indicated with a tick. So I click on Enable staging to remove the tick and turn it off. I publish the dataflow and wait. Yes! the refresh works, I test it multiple times.

Conclusion

I am curious as to how turning off staging will impact the dataflow. So I will be keeping an eye on this one. But this appears to be the fix we need to get it working again for now. I will do my enquiries and retests to see if this is still needed.

SharePoint Folder into Microsoft Fabric

Fri, 09 May 2025 17:08:15 +0000

SharePoint folder has been a connector into Power Query for Excel, Power BI Desktop and dataflows for quite a while. This post is to cover the basics before I write another post to explain a more complex scenario. This post will use GEN2 Dataflows in Microsoft Fabric. Power Query can take files on a SharePoint site and from a selected sheet, table or named range append the contents to one big table. This post explains what you should be aware of when doing this.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

SharePoint Library

I have a SharePoint library on a site called Budgets. It contains 3 files which are all the same format, 2 tabs, 1 table and 2 named regions.

The URL to the library is https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/Forms/AllItems.aspx . From this I can work out the URL to the SharePoint site is https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/ You can always test it by making sure the site url takes you to the front page.

Connecting to SharePoint Folder

In a GEN2 Dataflow, from the Home ribbon select Get Data and then more. If SharePoint folder isn’t showing in the dialog that appears, search for SharePoint. Then select SharePoint folder. Then enter in your site URL. Please note the full library url will not work.

If this is the first time you’ve connected to this SharePoint site you will need to set up the connection. All you need to do is change the Authentication kind to Organizational account and make sure you are signed in. Click Next to move onto the Preview folder data. If you have more than one library or folders it will show too many files, don’t worry we will fix this.

Click Create, rather than Combine so we can control which files get combined.

Filtering to the right SharePoint Folder

In my example I have multiple SharePoint libraries in the same site, so the initial list returned contains too many files. There is a column called Folder Path. I find a row that contains the value I want, right click on the value. Then select Text filters – Equals. This will add a filter a step to the query to filter to only files in that path.

Even if your current site only has one document library so currently this step is not required, I would add it anyway. Then when anyone adds files a new library it will not break your query.

Combining the Files

In order to make future additions to these queries easier and to reduce repeated work happening I would rename the query so far to FullFileList or similar and the right click on the query and select Reference. Rename the new query to something suitable, in my case it was Budget. The diagram will now show the two queries connected.

The next stage is to do the combining. Make sure the Content column has a data type of Binary. Then click the button in the top corner of the Content column to start the file combination. After a few seconds the Combine Files will appear requesting you to select which sheet, table or named range you are combining.

Once you click Combine it will create the Helper queries and related queries and will combine the files. You then will have a query with the first column Source.Name and the rest the data from the selected item.

Adding a Destination

There might be other transformation steps you need to add before your data is ready. I’ve added an unpivot to my data, but that would a complete post on its own. Finally we get to add a destination to save the data to. If you are going to a Lakehouse, Warehouse or Database you should be aware of option available to you.

The first option is if you are saving to an existing table or creating a new table.

When you click next the second option is available. Turning off Use automatic settings will give you the options. Is the update an append or a replace? Is the schema dynamic or fixed, ie if your query adds a new column what happens? I’ve picked replace, the default option but as the SharePoint folder grows I might look to more clever file filters so that only new data is selected and therefore can be appended.

Conclusion

Uploading multiple Excel files will always be a requirement. So its worth knowing what Power Query can do for us in a dataflow to do that. There are probably more capacity efficient methods which need exploring.

Trigger Microsoft Fabric Data Pipeline using Power Automate

Sun, 06 Oct 2024 14:18:24 +0000

Microsoft Fabric Data Pipelines can be scheduled to happen at regular times. What happens when you want trigger Microsoft Fabric data pipelines to run as a reaction to an event? For example someone pressing a button in an app. For this we can use Power Automate to start a pipeline using a http request.

Fixed

This post has incorrect details for the connector. These have now been fixed.

YouTube Version

Workspace and Pipeline GUIDs

The flow is going to use a http action that uses a URL that includes GUIDs for both the workspace and the data pipeline. Open the data pipeline that will be triggered. Look at the URL in the browser. The Workspace GUID is the string after /groups/ and the data pipeline GUID is the string after /pipelines/.

Start Flow and Add Variables

Now we have the required GUIDS we can start writing the flow. For this example we are going to use a manually triggered flow. Then we add an actions to initialise variables WorkspaceID and PipelineID and select Type of string and put in the values from the previous step.

URL of the Trigger Microsoft Fabric Data Pipeline request

Next I recommend you use a compose step to calculate the URL for the HTTP call. Interestingly the api uses workspaces rather than groups. Make sure you have the corrects /s and no spaces between. If you have called your variables WorkspaceID and PipelineID you can just copy the code below into a Compose action.

1

https://api.fabric.microsoft.com/v1/workspaces/@{variables('WorkspaceID')}/items/@{variables('PipelineID')}/jobs/instances?jobType=Pipeline

The above url including all the flags is documented by Microsoft here – https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#run-on-demand-item-job

HTTP Request

For this action we are going to use HTTP with Microsoft Entra ID (preauthorized) connection’s Invoke an HTTP request action. Search for HTTP and scroll down till you find it.

If this is the first time you’ve used this action you will need to setup the connection to connect to this api you will need to set it up. The two urls required are below, sometimes they appear the other way around.

Base Resource URL

1

https://api.fabric.microsoft.com/

Microsoft Entra ID Resource URI

1

https://analysis.windows.net/powerbi/api

Once you have the connection sorted you can enter in the details of the request. The Method is POST and the Url of the request is the output from the compose step.

Testing Trigger Microsoft Fabric Data Pipeline Flow

The flow requires no inputs so can just be run. When the flow runs it appears to fail, but if you check the data pipeline run history you will see it was triggered just fine.

If you look in the documentation found at https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#run-on-demand-item-job. There is a note that says “There is no body returned currently”. Eventually it will return the job id of the pipeline run, so I am guessing because that is not returned Power Automate assumes it failed.

Conclusion

Data pipelines are the orchestration tool of Microsoft Fabric so being able to connect it to the Power Platform orchestration tool makes perfect sense. I look forward to the day I can mark this post as redundant and there is a Power Automate connector with an obvious action for this.

Power Automate – Save a File to OneLake Lakehouse

Thu, 03 Oct 2024 14:48:09 +0000

Currently there is no Microsoft Fabric connector in Power Automate. When we want Power Automate to save a file to OneLake Lakehouse, we need to use a workaround. This post walks through the demo I did for the Microsoft Fabric European Conference 2024. You need to be aware that this uses a premium connector!

YouTube Version

Getting the File Content

Before we can save a file anywhere we need the file content we are going to save. For this post I am going to work with the flow triggering from a new file in a SharePoint library and getting the file content action.

From the SharePoint connector triggers, select the When a file is created (properties only) and populate the site address and library name. Next add a SharePoint action, Get file content. Populate the Site Address same as the trigger. For the File Identifier, select Identifier from the dynamic content from the trigger.

Lakehouse Folder URL

Next we need to know the url of the Lakehouse folder where we want the file to go. This can be found in the Lakehouse by clicking on the three dots next to the required folder and from the menu selecting properties. This will open the properties pane on the right hand side. Click the copy button next to the URL.

To make life easy for ourselves we put this into a compose step followed by name of the file that triggered the flow. This is in the dynamic content from the trigger and is called File name with extension. Please note there is a / before the file name.

Create the file in the LakeHouse

Set up the connection

We are going to use the HTTP with Microsoft Entra ID (preauthorised) action to interact with the Lakehouse to create the file. When you add the new action search for http and then scroll down to find the right action.

If this is the first time you’ve used this action you will need to set-up the connection. Also if this is the first time you’ve connected to the Lakehouse in this way you need to setup a new connection. It asks for two URLS. Once connected you are ready to save a File to OneLake Lakehouse

1

https://onelake.dfs.fabric.microsoft.com/

1

https://storage.azure.com/

Create Blank File

The first stage is to save a file to OneLake Lakehouse that is empty. This is a simple PUT action with a parameter of ?resource=file after the path created in the compose. So select PUT for the method. In the URL of the request but the Output from the compose and add on ?resource=file. I also renamed the action.

Append Content

Finally the last action is to append the file content from SharePoint onto the file. Change the Method to PATCH. The Url of the request is the output from the compose followed by three parameters (copy from below). Expand the Advanced parameters and into the body of the request select File Content from the dynamic content.

1

?action=append&position=0&flush=true

Test Save a File to OneLake Lakehouse

Save your flow and add a file to your SharePoint library and wait a minute or 2 for the flow to trigger. A file should appear in your Lake House.

Conclusion

There are plenty of business cases of use cases to be able to store a file into OneLake. Various legacy systems email daily exports which will need to be saved and processed in the LakeHouse. When / If we get a Fabric connector I look forward to making this post as redundant!

Microsoft Fabric – Notebook and Data Wrangler

Mon, 13 Nov 2023 16:22:16 +0000

Before I started with Microsoft Fabric I had never created a spark notebook. This doesn’t stop me though. By using a combination of Notebook and Data Wrangler means I can transform data and write to table writing very little code by hand.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Create and rename a Notebook

Inside the Lakehouse, on the ribbon drop down Open notebook and select New notebook. This will create an empty notebook with a default name. We can rename the notebook by clicking on the down arrow next to the name and entering a new name.

Adding a Markdown Title cell

Notebooks use 2 different types of cells, Code and Markdown. The Markdown cells allow for documentation to be included in the notebook. Any cell in the notebook can be converted into a Markdown cell. Below are screen grabs of converting the default cell into a Markdown cell, by clicking on the M button so giving a title cell to the notebook.

Markdown includes syntax for lots of different text formatting. A good reference can be found at https://www.markdownguide.org/basic-syntax/

Load the file into a data frame

Notebooks use data frames to handle blocks of data. The next task is to load a csv file. On the left hand side select Files to list the right file. Then on the Products.csv, click on the three dots menu. Next select Load data and then Spark. This will add a new code cell to the notebook.

Run a code cell

Each code cell has a blue arrow on the left, clicking this runs the code in the cell. The function display lists the data frame contents below the code cell. Being able to run each cell in turn really helps with development. There is a run all option on the toolbar.

We can limit the display function to only list a few lines by using a method on the data frame called take. df.take(4) just returns 4 rows of data. Another method that is useful to examine a dataframe is dtypes. This returns all the datatypes of the data frame. So I edit the code and re-run the cell.

Data Wrangler

Before I can write the data frame to a table there are transformations that need to be done. Firstly there are spaces in some of the column names. Next the ID and Price columns need to be numbers rather than strings. Finally I do not want to include the Notes column.

I could do the searches and find the Spark code to do those transforms. There is an alternative introduced in Microsoft Fabric called Data Wrangler. This will write the code for the three changes we need. We launch Data Wrangler from the Data ribbon – Transform Data Frame in Data Wrangler. This will list your data frames – which means you need to have run the cell that populates the dataframe first. Then we get to see the Data Wrangler screen.

Rename Columns

Product ID column needs to be renamed. Clicking on the three dots on the Product ID column heading. Then select rename column and the Operations pane changes to show rename options. Enter the new name Product_ID and the data pane will update to show the old in red and the new in green. Finally click Apply to save the changes.

The same needs to be done on the Base Price column to change it to Base_Price.

Changing Column Types

The next requirement is to change the column data types. This is very similar to renaming and can be found on the column heading menu. I change the Product_ID column to Int64 and the Base_Price column to Float64.

Dropping a column

Finally we need to drop the Notes column. Again this is found on the column header menu.

Add Data Wrangler code to Notebook

Once we have finished all the transformations we are ready to add the code into the notebook. Click on the Add code to notebook. When the dialog appears, check the code and click Add.

Run the Data Wrangler code

When the new code cell appears, we can see it ends by creating a new data frame df_clean and displaying it. So if we run the code we can see the result of the transformations.

Write data frame to a table

After the transformation code has run, the data frame is ready to be saved into a table. This is the single line of code we have write by hand. A data frame has a write method which accepts format and mode options. Using Ctrl + Spacebar will get the autocomplete to help. Here is the required code.

1
2


# Save df_clean to a table called Products
df_clean.write.format("delta").mode("overwrite").saveAsTable("Products")

Once the code has run successfully, we can look in the Lakehouse and see the new table products.

Conclusion

For the newcomer, notebooks are daunting. With the addition of Data Wrangler we have a tool to help write the transformation code. One quote I heard was Data Wrangler is the Power Query tool for notebooks. I have yet to work out when I would use a notebook over a dataflow but I’m assuming there will be pros and cons that vary depending on the data involved.

Microsoft Fabric – Create tables with Dataflows

Fri, 10 Nov 2023 09:09:14 +0000

As part of Microsoft Fabric, we have new Gen2 Dataflows and they have a new ability. A Gen2 Dataflow entity can have a destination. This means using a dataflow we can create a table in Fabric from another source, such as a folder of files.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Creating Gen2 Dataflows

Open the Lakehouse and then from the Get data drop down on the Home ribbon select New Dataflow Gen2. This will create an empty dataflow with a default name. Clicking on the name in the top left, then a pop-up appears allowing you to change the name.

For this blog post we are going to load the csv files in Sales folder, loaded in an earlier post and video. The files have been loaded into the Lakehouse so the data source is OneLake. In the Power Query window, on the Home ribbon, click Get data. From the menu that appears, we select More.. When the Choose data source appears, the bottom half lists OneLake sources, select the Lakehouse.

When the Choose data appears navigate to find the Sales folder. Then click Create. After a few moments a query will appear showing a row for every file in the folder.

Power Query includes a feature to combine files, it requires the Content column to be binary. On the Content column, click on the datatype ABC/123 in the top left of the column and select binary. This will give you a column of binary

The Content column is now the correct data type to combine the files. When you click on the top right icon it will start creating the helper queries to do the file combining. It will open a combine files dialog, showing you the content of the first file. When you click OK it will build the helper queries and the Sales query will now show the data from the files appended.

Fixing the date column

The CSV files contains dates in American format, MM/DD/YYYY. The dataflow is looking for dates in my format DD/MM/YYYY so makes the date column string format. This needs fixing before saving to a table.

Click on the ABC in the top left of the Order Date column. Then from the data type options select Using locale. When the dialog appears, I select Date for the data type and the English (United States) for the locale. When I click okay it changes the strings into dates. This feature is awesome and probably deserves a post of its own!

Adding a Destination

Now the data is ready we can set up the destination. In the bottom right of the screen it shows Data Destination and should show No data destination.

Click on the plus next to Data destination and from the options select Lakehouse. You will get a connections window appear and you can just click Next. Then the next step asks you to select which workspace and then which container in that workspace. Once you have selected the right place you can update the Table Name. Now you are ready to click Next.

The last step update method and column mapping. The update method is a choice between adding data to the bottom of the table, Append, or overwriting the data, Replace. We are going to use Replace in this instance. The column mapping takes the columns from the query and tries to use those for column name. Column names can only contain letters, numbers and underscores. So if you have column names that don’t fit that it offers to fix that. Once you are happy click Save Settings

Publishing and Refreshing

Now we have a query that shapes our data and we have a destination set up we can Publish the dataflow. This will save the dataflow and will refresh it. If you wanted to just save the dataflow and not refresh you can click the arrow next to Publish and select Publish later.

Once the refresh has completed, we can open the Lakehouse and see the new Sales table has been created.

Gen2 Dataflows Conclusion

Dataflows having a destination to write to is an exciting update. Power Platform has dataflows that can write to Dataverse but they have more issues than I’m willing to list here. Probably 90% of the transformations you need in Dataflows you can do by clicking rather than writing M. So this is mostly a low-code solution for getting data into Fabric. A later post will talk about how to handle not just overwriting the table.

Microsoft Fabric – Create table from CSV file

Thu, 09 Nov 2023 08:41:51 +0000

In a previous post we uploaded a csv file into the Lakehouse. In this post we take the next step to create table from csv file loaded into the Lakehouse. This is the simplest method to create a table and requires no coding. The simplicity means that there are limitations.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Create Table from CSV file

From within the Lakehouse, navigate to view the filename of the file you want to load into a table. Click on the 3 dots next to the filename. Then click on the arrow next to Load to Tables and select New table. When the next dialog appears, by default the table name will match the csv file name in lowercase. The name can be updated. Tick or untick the box regarding the headers, and click Load.

After a short while the new table will appear in the Explorer pane. When you click on the table name, a preview will appear.

Limitations

This method of creating a table leaves the determination of column type to the process. So we have no control of that. For some files this will cause an issue. To avoid any issue on the date format I made sure the date format was YYYY-MM-DD.

You can see what the column types are in the table by clicking on the arrow next to the table name to expand it.

You cannot change the column types from this view. The table is stored in delta format and clicking on the 3 dots menu will give you options to explore the properties and view the actual files that contain the data.

Updating a Table

When selecting to load into a table and selecting Existing table instead of New table will give a slightly different dialog box. You need to select the table and then select if the file data is to be appended or will overwrite the existing data. Be aware there is no undo.

Conclusion

For a clean, well structured CSV file, this is a great place to start. With a slow moving dimension table this could be a good solution as it just requires maintaining a csv file. For more complex files that require transformation there are better methods such as a dataflow or a notebook.

Microsoft Fabric – Upload a CSV file and folder

Tue, 07 Nov 2023 16:50:43 +0000

The simplest way to add data to a Lakehouse is upload a csv file. In this blog post we will upload a csv file and upload a folder of csv files. You can upload other file types, but they take more work than csv files to quickly load into tables.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Upload a CSV

We open the Lakehouse, see previous post on how to create one. Then in the Explorer pane on the left next to Files and click on the three dots. On the popup menu click on Upload and then Upload files. When the pane appears on the right, click on the folder icon in the input box. Once you’ve selected the file, click the Upload button.

Once the file is uploaded it can be found by clicking on Files. If you click on the file it the content will be displayed.

Multiple files could have been selected to be uploaded at the same time. All the files would have been uploaded into the same location.

Upload a Folder of Files

Uploading a folder is very similar to uploading a single file. In the explorer pane on the left, click on the three dots next to File, or a subfolder and select Upload. Then select Upload folder and the Upload folder pane will appear on the right. Clicking on the folder icon will allow you to select a folder. Finally you can click Upload and you will a progress line for every file in the folder.

The list of files that have been uploaded can be seen by clicking on the folder name in the explorer pane on the left

Updating a File or Folder

Data is rarely the same forever so you will want to upload an updated file or folder. If you select a file or folder that already exists you will not be able to click the upload button unless you tick the Overwrite if files already exist.

Conclusion

Manually uploading files and folders is not a great solution for a large number of files but it is a good place to start. The OneLake app will be a great alternative when its fully available and using pipelines to orchestrate file management is also a good alternative and future posts will cover both.

Microsoft Fabric – Create a Lakehouse

Thu, 02 Nov 2023 22:43:20 +0000

Lakehouses are one of the main building blocks in Microsoft Fabric. A lakehouse allows you to store structured and unstructured data in a single location. In this post we are going to create an empty one. Future posts will populate it.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Empty Workspace with a Fabric Capacity

We start in an empty workspace that has a capacity assigned to it. A jewel logo next to the workspace name or checking the workspace settings are indicators that it has a capacity. A premium per user capacity does not include all Fabric features.

Create a Lakehouse

There are two ways to start creating a lakehouse. First method is to click New in the top left of the workspace area, by default it shows Power BI options so click on More options. When the screen updates, on the second row is Lakehouse.

The second option is switch the experience in the bottom left of the screen to show all the Experiences. When the menu pops up, select Data Engineering. Then the screen will update and show a slightly different Lakehouse button.

The next step in the create a lakehouse process is to name the lakehouse. The name can only include letters, numbers, and underscore. It will display an error message to correct you. Enter the name and click Create. Then wait a few moments and you will get your empty lakehouse.

What is created?

Every lakehouse created in Fabric also creates an SQL Endpoint and a Power BI Dataset. Even though we have not added data the containers are already there to handle the data.

References

Microsoft’s Lakehouse Overview – https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-overview

Microsoft Fabric on Hat Full of Data

Using Variable Library in a Dataflow

Variable Library Series

Scenario

Getting the Value

Using the Variable Value

References for Variable Library in a Dataflow

Conclusion on using variable library in a dataflow

Using a Variable Library in Lakehouse Shortcuts

Variable Library Series

Setting Up the Shortcuts

Viewing the Options

Applying the Variables

Updating Variable Values

Conclusion on Variable Library in Lakehouse Shortcuts

Resources

Using a Variable Library in a Data Pipeline

Variable Library Series

Adding the Variables to the Data Pipeline

Using Variable Values

Using Variables for Data Connections

Connection IDs

Workspace, Warehouse and Lakehouse IDs

Connection String

Variable Library

Using the Variables in the Activities

Conclusion on Using a Variable Library in a Data Pipeline

Resources

Accessing a Variable Library in a Notebook

Variable Library Series

Connect to Library

Get Variable Value

Get a variable value by reference

Naming Conventions

Conclusion on using Variable Library in a Notebook

References

Microsoft Fabric – Variable Library

Variable Library Series

Create a Variable Library

Add a Variable

Adding Notes

Saving

Conclusion

References

Microsoft Fabric – Using Copy Job with Dataverse

Table of Contents

Scenario

First Copy Job – Dataverse Connector

Create Copy Job

Choose Data Source

Choose Data

Choose Data Destination

Choose copy job mode

Map to destination

Review + save

Copy Job Run

Copy Job using SQL Server connection

New Copy Job using SQL Server

Choose Data

Incremental and Merge setup

Save, Review and Schedule

Incremental and Merge Running

Trouble Shooting

Alternative Options to using Copy Job with Dataverse

Link to Microsoft Fabric

Conclusion

Microsoft Fabric – Direct Lake and User Permissions

IMPORTANT

Scenario

Viewing Reports as Dan

Viewing Workspace App as Sam

Permissions

Fix Dan’s Lakehouse Access

Fix Sam’s Access via the Workspace App

Note!

Resources

Conclusion

Other Posts that might Interest

Microsoft Fabric Quick Guides

Lewis and Laura vs. Notebook and Microsoft Graph: Part 2