2025-2026 on Hat Full of Data

Using Variable Library in a Dataflow

Wed, 11 Feb 2026 17:21:24 +0000

One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.

I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!

Variable Library Series

Variable libraries should be part of every project. This post is part of my series to help get you started creating the library and then using the variables and finally seeing your hard work pay back when it comes to deployment pipelines.

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines
Connection variables in a variable library

Scenario

For this post we are going to use the scenario of a dataflow query that lists projects. Whilst we are developing the data handling we want to limit the number of rows loaded and in production we will have no limit.

I create the dataflow and then using Keep Rows, I select Keep top rows. In the next dialog I enter 5 and click OK. This results in a table with only 5 rows of data. The number of rows though needs to come from the variable library.

Getting the Value

The next step is to get the value from the variable library into the dataflow. We are going to use a new Power Query function for this called Variable.ValueOrDefault. For the first parameter of this function, you need the Variable Library name and a Variable name. Then you combine together in a string.

1

"$(/**/<Library Name>/<Variable Name>)"

So for my example the string will be “$(/**/Finance Variables/Limit)”.

📝 Instructions

On the Home ribbon, expand Get data and select Blank query.

Rename the query to DataRows.

Then in the formula bar enter in the code below.

This will return the current value of the variable in the library.

1

Variable.ValueOrDefault("$(/**/Finance Variables/Limit)",2)

💡 Tip

The documentation as of publishing this post states it won’t work hence we use the OrDefault function and in this example the default is 2. It does work though, and we can see the answer 8 comes through.

Using the Variable Value

We’ve got the variable value from the variable library in a dataflow, now we need to use it. In the last line of my project query that limited the query to 5 rows I can replace the 5 with DataRows. I now get 8 rows of data.

This works as long as we always want to limit the rows, the chances are in production we don’t want to limit, so I add the extra of only limit if DataRows is greater than 0. Here is my new statement, the previous step is call Sorted Rows hence the #”Sorted rows”

1

if DataRows > 0 then Table.FirstN(#"Sorted rows", DataRows) else #"Sorted rows"

References for Variable Library in a Dataflow

Microsoft – Use Fabric variable libraries in Dataflow Gen2

Power Query Reference for Variable.ValueOrDefault

Conclusion on using variable library in a dataflow

Its great we can bring in the values easily. Its a shame that we can’t use them to control the destination, but that is on the road map for 2026 Q1. I’ll blog about it as soon as it arrives! I highly recommend using my pattern of fetching the value as a query and then referring to that. It will help in debugging etc.

Using a Variable Library in Lakehouse Shortcuts

Mon, 26 Jan 2026 18:43:51 +0000

Lakehouse shortcuts are a popular addition to the Fabric set of tools to access data easily without copying it. Using a variable library in lakehouse shortcuts means its easy to point shortcuts to an alternative location. This great for ALM using development, test and production workspaces.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines
Connection variables in a variable library

Setting Up the Shortcuts

Using the standard GUI you have no option to use a variable library values in the setup of the shortcuts. So set up your shortcuts as you normally and make sure they work. For this post I’m using a shortcut to a table in warehouse. (I had a whole barny with lakehouses and schemas and it not working – another post for another time!)

According to the documentation using the rest API also does not allow for the use of variables in the creation of shortcuts.

Viewing the Options

If we select the short cut and click on the 3 dots to expand the menu we can then click on Manage Shortcut. This opens a pan that shows the values behind the shortcut. The values that be populated from a variable library have a Variable drop down, so we can see Target workspace, Target item and Target path.

I only want to populate the workspace and item guids, so I set up a those variables in my variable library. The variables must be set up as strings, they might look like guids but the shortcut demands they are strings. I set the default values to match the current values. When we apply each variables it will check the current values work so it only works if you use the current values.

Applying the Variables

Back in the Manage Shortcut pane click on the Variable below the current value to expand that section. Then click on Select variable to open the Select variable dialog. Click on the correct variable and then click Select Variable. This will update the value to come from the variable.

Once you have pointed it to the variable there does not appear to be a method to change back to a hard coded value.

Updating Variable Values

Variable values will get updated, the most common being a different active set being selected. For other artifacts the change will get picked up when they execute or refresh. For shortcuts this is slightly different. You need to force the updating of variables.

Open the lakehouse and stay in the lakehouse view. On the ribbon the last button is Update all variables. A message will show the updating is taking and finally there will be a notification to say they have been updated.

If one of the shortcut fails in the update please be aware the successful ones will have updated. You do get a notification and an error message that tries to help.

Conclusion on Variable Library in Lakehouse Shortcuts

I really like that the variable libraries can be accessed directly in the shortcut details, there is no pre-loading. I would prefer that I could change the values and then press apply so that each value isn’t checked individually. My experience in deployment pipelines means I fell foul of not realising I needed to refresh the variable values.

Resources

Creating Shortcuts – https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-shortcuts

Microsoft Learn – https://learn.microsoft.com/en-us/fabric/onelake/assign-variables-to-shortcuts

Using a Variable Library in a Data Pipeline

Mon, 19 Jan 2026 10:36:26 +0000

You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines
Connection variables in a variable library

Adding the Variables to the Data Pipeline

Before you can use the variable library in a data pipeline you need to add the variables. In your data pipeline you need to click on the grey background to bring up the section on the bottom of the screen with Parameters, Variables etc. If you are in a brand new pipeline you need to click on the settings cog first, then it will appear.

Select the Library Variables section. Then click on +New and the select variable dialog box appears. If you have multiple variable libraries you can select the right one, you can also search for a variable in the top right. Once you select a variable click on Select Variable button to add that variable to the pipeline.

Yes you have to add each variable in turn, no multi-select. You can rename the variables but please don’t, name your variables in your library better.

Using Variable Values

You can use the variable values anywhere you can use Dynamic Content. Click on the Add Dynamic content link to open the Pipeline expression builder. Click on the three dots to select Library variables. Click on the variable value you want to use to enter in the formula into the expression. Click OK to put the formula back into the box.

You can use the variable value inside any functions if required.

Using Variables for Data Connections

When a data pipeline move from a dev workspace to test workspace and then onto production workspace we need to change the details of the connections to the new values. For this post we will use a connection to a Fabric warehouse in a workspace. I want to pull the values from a table in the warehouse using a lookup action.

I need to create the following variables in the library:

Project_Warehouse_Connection_ID
Project_Workspace_ID
Project_Warehouse_ID
Project_Warehouse_Connection_String

Connection IDs

For the connection guid you need to click on the cog on the top bar and select Manage connections and gateways. From the list of connections, find the one you are using. From the three dots menu select settings. The Connection ID will be listed and can be highlighted and copied.

Workspace, Warehouse and Lakehouse IDs

IDs for most artifacts can be found when it is opened. Be aware groups=workspaces in the url. Lakehouses have 2 IDs, one for the lakehouse and one for the SQL analytics endpoint or lakewarehouse as the url names it. No comment on names to confuse us!

Connection String

The final part needed is the connection string for the warehouse or lakehouse. The easiest way to copy this is to open the lakehouse SQL endpoint or the warehouse and in the bottom left hand corner is a Copy SQL connection string link.

It can also be found in the lakehouse or warehouse settings under SQL analytics endpoint.

Variable Library

All these values need saving into the Variable Library. I’ve used Guid for the IDs and String for the connection string.

Using the Variables in the Activities

We now need to use the variable values from the variable library in a data pipeline activity connection. We start by adding all 4 variables to the library. Note the guids become strings.

In this post for simplicity I’m using a Lookup activity to just get all the rows from the projects table. So I add a lookup activity and fill in the general details. On the Settings tab I click the drop down for Connection and select Use dynamic content. In the Add dynamic content pane I find Library Variables under the three dots and select FinanceVariables_Warehouse_Connection_ID so it appears in the expression box. Then I click OK for the Connection to now be an expression.

Once the connection has been populated you can select the connection type, in my case Warehouse. Then the text boxes appear to take the Workspace ID, Warehouse ID and SQL Connection string. They all have a link to Add dynamic content.

The table drop down can be populated by entering manually or by clicking Refresh. This will open a pane showing the values that are going to be used in the connection. Click OK to start it fetching the list of tables. Once a table is selected if you click Preview data the same pane appears confirming values. Its a great way to test the connection works.

You now have a variable values from a variable library in a data pipeline activity. Well done!

Conclusion on Using a Variable Library in a Data Pipeline

I like how simple it is to use the variables and that you get to select the variables needed in this data pipeline. I’d prefer if you could add multiple variables at a time. I like the consitent use of the expression builder so its something we get familiar with.

I’d also really like it if in lineage view you could see which artifacts are using of the variable library and the impact was also included. Maybe oneday!

Resources

https://learn.microsoft.com/en-us/fabric/data-factory/variable-library-integration-with-data-pipelines