Variable Libraries on Hat Full of Data

Using Variable Library in a Dataflow

Wed, 11 Feb 2026 17:21:24 +0000

One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.

I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!

Variable Library Series

Variable libraries should be part of every project. This post is part of my series to help get you started creating the library and then using the variables and finally seeing your hard work pay back when it comes to deployment pipelines.

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Scenario

For this post we are going to use the scenario of a dataflow query that lists projects. Whilst we are developing the data handling we want to limit the number of rows loaded and in production we will have no limit.

I create the dataflow and then using Keep Rows, I select Keep top rows. In the next dialog I enter 5 and click OK. This results in a table with only 5 rows of data. The number of rows though needs to come from the variable library.

Getting the Value

The next step is to get the value from the variable library into the dataflow. We are going to use a new Power Query function for this called Variable.ValueOrDefault. For the first parameter of this function, you need the Variable Library name and a Variable name. Then you combine together in a string.

1

"$(/**/<Library Name>/<Variable Name>)"

So for my example the string will be “$(/**/Finance Variables/Limit)”.

📝 Instructions

On the Home ribbon, expand Get data and select Blank query.

Rename the query to DataRows.

Then in the formula bar enter in the code below.

This will return the current value of the variable in the library.

1

Variable.ValueOrDefault("$(/**/Finance Variables/Limit)",2)

💡 Tip

The documentation as of publishing this post states it won’t work hence we use the OrDefault function and in this example the default is 2. It does work though, and we can see the answer 8 comes through.

Using the Variable Value

We’ve got the variable value from the variable library in a dataflow, now we need to use it. In the last line of my project query that limited the query to 5 rows I can replace the 5 with DataRows. I now get 8 rows of data.

This works as long as we always want to limit the rows, the chances are in production we don’t want to limit, so I add the extra of only limit if DataRows is greater than 0. Here is my new statement, the previous step is call Sorted Rows hence the #”Sorted rows”

1

if DataRows > 0 then Table.FirstN(#"Sorted rows", DataRows) else #"Sorted rows"

References for Variable Library in a Dataflow

Microsoft – Use Fabric variable libraries in Dataflow Gen2

Power Query Reference for Variable.ValueOrDefault

Conclusion on using variable library in a dataflow

Its great we can bring in the values easily. Its a shame that we can’t use them to control the destination, but that is on the road map for 2026 Q1. I’ll blog about it as soon as it arrives! I highly recommend using my pattern of fetching the value as a query and then referring to that. It will help in debugging etc.

Using a Variable Library in Lakehouse Shortcuts

Mon, 26 Jan 2026 18:43:51 +0000

Lakehouse shortcuts are a popular addition to the Fabric set of tools to access data easily without copying it. Using a variable library in lakehouse shortcuts means its easy to point shortcuts to an alternative location. This great for ALM using development, test and production workspaces.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Setting Up the Shortcuts

Using the standard GUI you have no option to use a variable library values in the setup of the shortcuts. So set up your shortcuts as you normally and make sure they work. For this post I’m using a shortcut to a table in warehouse. (I had a whole barny with lakehouses and schemas and it not working – another post for another time!)

According to the documentation using the rest API also does not allow for the use of variables in the creation of shortcuts.

Viewing the Options

If we select the short cut and click on the 3 dots to expand the menu we can then click on Manage Shortcut. This opens a pan that shows the values behind the shortcut. The values that be populated from a variable library have a Variable drop down, so we can see Target workspace, Target item and Target path.

I only want to populate the workspace and item guids, so I set up a those variables in my variable library. The variables must be set up as strings, they might look like guids but the shortcut demands they are strings. I set the default values to match the current values. When we apply each variables it will check the current values work so it only works if you use the current values.

Applying the Variables

Back in the Manage Shortcut pane click on the Variable below the current value to expand that section. Then click on Select variable to open the Select variable dialog. Click on the correct variable and then click Select Variable. This will update the value to come from the variable.

Once you have pointed it to the variable there does not appear to be a method to change back to a hard coded value.

Updating Variable Values

Variable values will get updated, the most common being a different active set being selected. For other artifacts the change will get picked up when they execute or refresh. For shortcuts this is slightly different. You need to force the updating of variables.

Open the lakehouse and stay in the lakehouse view. On the ribbon the last button is Update all variables. A message will show the updating is taking and finally there will be a notification to say they have been updated.

If one of the shortcut fails in the update please be aware the successful ones will have updated. You do get a notification and an error message that tries to help.

Conclusion on Variable Library in Lakehouse Shortcuts

I really like that the variable libraries can be accessed directly in the shortcut details, there is no pre-loading. I would prefer that I could change the values and then press apply so that each value isn’t checked individually. My experience in deployment pipelines means I fell foul of not realising I needed to refresh the variable values.

Resources

Creating Shortcuts – https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-shortcuts

Microsoft Learn – https://learn.microsoft.com/en-us/fabric/onelake/assign-variables-to-shortcuts

Using a Variable Library in a Data Pipeline

Mon, 19 Jan 2026 10:36:26 +0000

You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Adding the Variables to the Data Pipeline

Before you can use the variable library in a data pipeline you need to add the variables. In your data pipeline you need to click on the grey background to bring up the section on the bottom of the screen with Parameters, Variables etc. If you are in a brand new pipeline you need to click on the settings cog first, then it will appear.

Select the Library Variables section. Then click on +New and the select variable dialog box appears. If you have multiple variable libraries you can select the right one, you can also search for a variable in the top right. Once you select a variable click on Select Variable button to add that variable to the pipeline.

Yes you have to add each variable in turn, no multi-select. You can rename the variables but please don’t, name your variables in your library better.

Using Variable Values

You can use the variable values anywhere you can use Dynamic Content. Click on the Add Dynamic content link to open the Pipeline expression builder. Click on the three dots to select Library variables. Click on the variable value you want to use to enter in the formula into the expression. Click OK to put the formula back into the box.

You can use the variable value inside any functions if required.

Using Variables for Data Connections

When a data pipeline move from a dev workspace to test workspace and then onto production workspace we need to change the details of the connections to the new values. For this post we will use a connection to a Fabric warehouse in a workspace. I want to pull the values from a table in the warehouse using a lookup action.

I need to create the following variables in the library:

Project_Warehouse_Connection_ID
Project_Workspace_ID
Project_Warehouse_ID
Project_Warehouse_Connection_String

Connection IDs

For the connection guid you need to click on the cog on the top bar and select Manage connections and gateways. From the list of connections, find the one you are using. From the three dots menu select settings. The Connection ID will be listed and can be highlighted and copied.

Workspace, Warehouse and Lakehouse IDs

IDs for most artifacts can be found when it is opened. Be aware groups=workspaces in the url. Lakehouses have 2 IDs, one for the lakehouse and one for the SQL analytics endpoint or lakewarehouse as the url names it. No comment on names to confuse us!

Connection String

The final part needed is the connection string for the warehouse or lakehouse. The easiest way to copy this is to open the lakehouse SQL endpoint or the warehouse and in the bottom left hand corner is a Copy SQL connection string link.

It can also be found in the lakehouse or warehouse settings under SQL analytics endpoint.

Variable Library

All these values need saving into the Variable Library. I’ve used Guid for the IDs and String for the connection string.

Using the Variables in the Activities

We now need to use the variable values from the variable library in a data pipeline activity connection. We start by adding all 4 variables to the library. Note the guids become strings.

In this post for simplicity I’m using a Lookup activity to just get all the rows from the projects table. So I add a lookup activity and fill in the general details. On the Settings tab I click the drop down for Connection and select Use dynamic content. In the Add dynamic content pane I find Library Variables under the three dots and select FinanceVariables_Warehouse_Connection_ID so it appears in the expression box. Then I click OK for the Connection to now be an expression.

Once the connection has been populated you can select the connection type, in my case Warehouse. Then the text boxes appear to take the Workspace ID, Warehouse ID and SQL Connection string. They all have a link to Add dynamic content.

The table drop down can be populated by entering manually or by clicking Refresh. This will open a pane showing the values that are going to be used in the connection. Click OK to start it fetching the list of tables. Once a table is selected if you click Preview data the same pane appears confirming values. Its a great way to test the connection works.

You now have a variable values from a variable library in a data pipeline activity. Well done!

Conclusion on Using a Variable Library in a Data Pipeline

I like how simple it is to use the variables and that you get to select the variables needed in this data pipeline. I’d prefer if you could add multiple variables at a time. I like the consitent use of the expression builder so its something we get familiar with.

I’d also really like it if in lineage view you could see which artifacts are using of the variable library and the impact was also included. Maybe oneday!

Resources

https://learn.microsoft.com/en-us/fabric/data-factory/variable-library-integration-with-data-pipelines

Accessing a Variable Library in a Notebook

Mon, 12 Jan 2026 09:34:51 +0000

This post walks through how to access a variable library in a notebook in Microsoft Fabric. I recommend a Microsoft Fabric project starts by creating a variable library to store the common values different artifacts need and could be changed if a deployment pipeline gets involved. So when we create a notebook we need to be able to use these variables. This means we need load the variable library in a notebook and then get the variable values.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Connect to Library

The first task is to create a variable for the variable library. The notebookutils package includes the functions we need to do this easily. In the code below the variable vl will refer to the variable library that is called “Finance Variables”. This code assumes the notebook is in the same workspace as the library.

1
2


# Get the Variable library
vl = notebookutils.variableLibrary.getLibrary("Finance Variables")

Get Variable Value

Once we have the variable library loaded, we can access the variable values in one of two syntaxes.

1
2
3


# 2 syntaxes to get variable value
SP_URL = vl.getVariable("SharePoint_URL")
Limit = vl.Limit

Combining both of the two and adding some print statements we can demo the above using this code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Get the Variable library
vl = notebookutils.variableLibrary.getLibrary("Finance Variables")

# 2 syntaxes to get variable value
SP_URL = vl.getVariable("SharePoint_URL")
Limit = vl.Limit

# Check results
print(f"{SP_URL=}")
print(f"{Limit=}")

And here is the run

Get a variable value by reference

There is a third way to get a variable value. The get method from notebookutils variable library uses a reference string that follows the following syntax:

1

"$(/**/<LIBRARY NAME>/<VARIABLE NAME>)"

Before you get excited that the ** implies you could refer to another workspace, sorry that is not supported. This string is used in the variableLibrary.get action in notebookutils. For example to get the two values from before could be done like this

1
2
3


# Get the variable values directly
SP_URL = notebookutils.variableLibrary.get("$(/**/Finance Variables/SharePoint_URL)")
Limit = notebookutils.variableLibrary.get("$(/**/Finance Variables/Limit)")

And here is the run

💡 Tip

The print(f"{Limit=}") prints Limit=4 for us newbies to Python!

Naming Conventions

When you start and there are only 4 variables in the library its easy to remember the names to type in by hand when you are using a variable library in a notebook. When it gets to 20 names, its harder. So create standards, whats the case pattern, you using _ between words or not? Pick a set of rules and stick with it, BronzeWorkspaceID works just as well as Bronze_Workspace_ID, WorkspaceID1 and workspace_id2 are a technical debt you don’t need.

Conclusion on using Variable Library in a Notebook

I’m really impressed on how easy it is to use the variable library in a notebook. The pattern is simple and resusable. If I had a magic wand it it would be great is the intellisense knew the variable names from loading the variable library object.

The method by reference is great if you want to dynamically select which variable to load as the string could be dynamically built.

Using a combination of parameters and accessing a variable library means reusable notebooks are easier to write. This needs to become part of the best practice pattern being used.

References

Microsoft learn covers Variable library utilities as part of the notebook utilities page.

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#variable-library-utilities

Microsoft Fabric – Variable Library

Tue, 06 Jan 2026 13:27:41 +0000

This post to help you get started creating a variable library. When multiple dataflows, notebooks and pipelines are using the same details to perform tasks it helps if those values are stored in one place. When you move to use deployment pipelines and those values change from your development workspace to your test workspace to your prod, it helps if that is easy. The solution in Microsoft Fabric is a Variable Library to store those common values.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Create a Variable Library

This post assumes the workspace is backed with a Fabric capacity. A trial capacity works just fine.

Either from a Develop task add item or from the Add item button select Variable Library. Enter in a meaningful name for the variable library, letters, numbers and spaces are allowed. Then click the Create button and your new library will be opened. I recommend you create a naming convention that all your Fabric developers like and agree to use.

Add a Variable

Your library starts empty, so the next task is to add variables. You can create a new variable row by clicking on + New Variable on the Home ribbon. When there are no variables there will also be a green button in the center of the page to create a new variable.

Enter in a meaningful name and select a data type. A name can only include letters, numbers and underscores. An incorrect name will prevent the library from being saved. Please note some types are not usable in some Fabric artifacts, no I haven’t worked out the logic yet but I’m working on it. Enter in a value for the Default value set. Value sets are used when you are using deployment pipelines for the different values you use development,test and production workspaces. Creating and using variable sets will be covered in another post.

Adding Notes

So that “future you” doesn’t hate the “current you”, it is worth adding notes to variables to explain the purpose of the variable. This should be part of the best practice rules for Fabric development.

Click on the icon in the note column. When there is no note it has a plus symbol and when there is a note it has a pencil symbol.

Saving

Variable libraries do not auto-save. You need to click the save button on home ribbon. Indicators of black dots will show unsaved items you have open and the save button will only be active when there are changes to save.

Conclusion

I highly recommend starting any Fabric project with creating a variable library. Then everytime a value is used in an artifact such as a database name, key vault name etc a descision is made on if it should be in the variable library or if the value is already there. Some values to be more useful will need to be split across multiple variables.

It should always be assumed that a workspace with Fabric items will eventually be in a deployment pipeline. This means that it should be coded to make deployment easier. The aim of these articles is for quick reference on how variable libraries are used across all the artifacts.

References

Microsoft have some getting started articles on setting up a variable library and using it in a deployment pipeline.

Microsoft Learn Variable Libraries