Dataflow on Hat Full of Data

Using Variable Library in a Dataflow

Wed, 11 Feb 2026 17:21:24 +0000

One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.

I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!

Variable Library Series

Variable libraries should be part of every project. This post is part of my series to help get you started creating the library and then using the variables and finally seeing your hard work pay back when it comes to deployment pipelines.

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Scenario

For this post we are going to use the scenario of a dataflow query that lists projects. Whilst we are developing the data handling we want to limit the number of rows loaded and in production we will have no limit.

I create the dataflow and then using Keep Rows, I select Keep top rows. In the next dialog I enter 5 and click OK. This results in a table with only 5 rows of data. The number of rows though needs to come from the variable library.

Getting the Value

The next step is to get the value from the variable library into the dataflow. We are going to use a new Power Query function for this called Variable.ValueOrDefault. For the first parameter of this function, you need the Variable Library name and a Variable name. Then you combine together in a string.

1

"$(/**/<Library Name>/<Variable Name>)"

So for my example the string will be “$(/**/Finance Variables/Limit)”.

📝 Instructions

On the Home ribbon, expand Get data and select Blank query.

Rename the query to DataRows.

Then in the formula bar enter in the code below.

This will return the current value of the variable in the library.

1

Variable.ValueOrDefault("$(/**/Finance Variables/Limit)",2)

💡 Tip

The documentation as of publishing this post states it won’t work hence we use the OrDefault function and in this example the default is 2. It does work though, and we can see the answer 8 comes through.

Using the Variable Value

We’ve got the variable value from the variable library in a dataflow, now we need to use it. In the last line of my project query that limited the query to 5 rows I can replace the 5 with DataRows. I now get 8 rows of data.

This works as long as we always want to limit the rows, the chances are in production we don’t want to limit, so I add the extra of only limit if DataRows is greater than 0. Here is my new statement, the previous step is call Sorted Rows hence the #”Sorted rows”

1

if DataRows > 0 then Table.FirstN(#"Sorted rows", DataRows) else #"Sorted rows"

References for Variable Library in a Dataflow

Microsoft – Use Fabric variable libraries in Dataflow Gen2

Power Query Reference for Variable.ValueOrDefault

Conclusion on using variable library in a dataflow

Its great we can bring in the values easily. Its a shame that we can’t use them to control the destination, but that is on the road map for 2026 Q1. I’ll blog about it as soon as it arrives! I highly recommend using my pattern of fetching the value as a query and then referring to that. It will help in debugging etc.

First refresh it broke!

Mon, 12 May 2025 15:57:47 +0000

I set up my dataflow and it was working great. Then it broke on the refresh. Here is how I fixed it.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

Previously

In the previous post we ingested a SharePoint folder into a table in a Lakehouse. That is a well known process and nothing new except for one thing. I split the process into two stages, firstly the FullFileList which contained the list of files and the binary content of the files and then a separate query to combine data from those files. Power Query created Helper queries to perform the combining. Then it broke.

It Broke

It all appears to work fine. Then I try a refresh and it fails. On going back to look at the dataflow I see that there are 2 extra steps in FullFileList that have been added by Microsoft Fabric.

1
2
3
4
5
6
7


let
 Source = SharePoint.Files("https://lgb123.sharepoint.com/sites/FabricDemo/", [ApiVersion = 15]),
 #"Filtered rows" = Table.SelectRows(Source, each [Folder Path] = "https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/"),
 #"From Value" = Table.FromValue(#"Filtered rows"),
 #"Remove Columns" = Table.RemoveColumns(#"From Value", Table.ColumnsOfType(#"From Value", {type table, type record, type list, type nullable binary, type binary, type function}))
in
 #"Remove Columns"

The last step Remove Columns, removes all columns of type table, record, list, binary or function. So this removes the Content column that is binary and therefore the queries doing the combine of the Content column all fail. Hence it broke.

It Broke in the Staging

To improve performance and reliability, Dataflow Gen2 uses staging items to store intermediate data during data transformation

The above quote is from https://learn.microsoft.com/en-us/fabric/data-factory/data-in-staging-artifacts It stores the intermediate queries, so FullFileList would have been stored as a table in the staginglakehouse or stagingwarehouse. So Fabric is adding the step to remove the binary column so staging can work.

Turn Off Staging

When I right click on the FullFileList query in the query list the menu includes Enable Staging, by default it is turned on, indicated with a tick. So I click on Enable staging to remove the tick and turn it off. I publish the dataflow and wait. Yes! the refresh works, I test it multiple times.

Conclusion

I am curious as to how turning off staging will impact the dataflow. So I will be keeping an eye on this one. But this appears to be the fix we need to get it working again for now. I will do my enquiries and retests to see if this is still needed.

Refreshing Datasets Automatically with Power BI Dataflows

Sun, 29 Aug 2021 18:58:12 +0000

I have a report, Open Issues, that has a Power BI dataflow as a data source. If I just use scheduled refreshes, my options are limited. I need to be refreshing datasets automatically after the dataflow has finished so my reports are up to date as soon as possible to avoid a situation where I schedule the dataflow to refresh at 10am and the next slot available might be 10:30am, for example. This can be done using Power Automate.

Create a flow in Power Automate

Head to Power Automate at https://flow.microsoft.com/and click on Create on the left hand side menu. The flow will be triggered by the dataflow refresh finishing. This means it is automated so click on Automated cloud flow.

A dialog appears asking for details of the flow. You need to enter in a name for the flow. Then in the search box enter in power to find relevant triggers. Select When a dataflow refresh completes and click Create.

After you click Create the flow editor appears with the trigger already added. For Group Type select Workspace. Then you can select the workspace name for Group. Then you can select the Dataflow.

We then need to refresh the connected report dataset. So we click New step. In the Choose an operation dialog, we type dataset into the search box. This will find you the action Refresh a dataset. We click on the action to add it to the flow.

Then you need to select the Workspace and Dataset in the action. The flow is complete, so we can click Save.

Testing the Flow

The flow triggers when the dataflow finishes refreshing. If we refresh the dataflow that will test the flow. I clicked refresh on the dataflow at 3pm and it triggered the flow which then refreshed the dataset.

In the workspace we can see the date and time of the refreshes and if the flow has worked correctly the dataset should be refreshed just after the dataflow. If you look at the history of the flow runs there should be a matching flow run just after the dataflow finished refreshing.

When refreshing datasets automatically fails

It is worth pointing out that if the dataset refresh fails the flow will not reflect that and will show succeeded in the flow runs. So it is worth checking that the dataset can refresh successfully.

Conclusion

This idea can be extended to refresh multiple datasets over multiple workspaces that are connected to the dataflow. This simplifies the process of scheduling refreshes. It also could allow for notifications to interested report owners to be told of the refreshes etc.

Power BI Series

Power BI Dataflow – Create Dataflow from Export

Sat, 28 Nov 2020 23:15:04 +0000

There are times when you need to copy a dataflow from one workspace to another workspace. Power BI service provides a simple way to export the definition as a json file and then import.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Export dataflow as json file

In the workspace of the original workspace, click on the three dots next to the dataflow and select Export json. When the file is ready a message will appear in the top right.

Create dataflow from json file

In the destination workspace click on the New dropdown and select dataflow. From the options then displayed select Import Model. Select a your json file. Once the json file has imported a message will appear in the top right.

Although your dataflow has been created you are left on the dataflow creation page. You need to return to the workspace to see your new dataflow.

Check Credentials

The dataflow will not have refreshes setup as that is not part of the model. It is also recommended that you confirm that the credentials are correctly loaded. If you the dataflow is from another user or another tenancy you will need to re-login each credential.

To check the credentials you need to click on the three dots next to your new dataflow and select Settings. In settings, expand Data source credentials. If any of the credentials have a cross next to them click on Edit credentials to re-login.

Conclusion

Part of my work is to build dataflows to assist people fathom the depths of Microsoft Project data. The ability to export a dataflow means I have been able to transfer a solution between different teams across the company.

Power BI Series

Power BI Dataflow – Refresh History

Mon, 23 Nov 2020 16:20:50 +0000

You can find out which entity took the longest to refresh or which entity caused the error in a failed refresh by looking at the refresh history of your dataflow in the Power BI workspace.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Refresh History List

In the workspace, click on the three dots next to the dataflow and from the menu select Refresh History. This will open a dialog that has a row for every refresh.

Every refresh will show when it happened, how long it took and if it completed or failed. In the above image we can see the last 2 refreshes failed but the 4 previous refreshes completed.

Refresh History Files

The details of the refresh for each entity that loads from the dataflow can be found by downloading the csv file. You can download the csv file for a refresh by clicking on the down arrow on the right of each refresh row.

Calculating the Refresh Time

Excel will happily open the CSV file. The first column might say Invalid Date this is due to the Power BI service trying to treat a dd/mm/yyyy formatted date as mm/dd/yyyy. I have reported the issue to Microsoft.

A simple calculation of (End Time – Start Time) will give you the duration in days so in this example I then multiplied it 246060 to give seconds. Adjust the calculation to minutes or hours to fit your durations.

Showing Errors

If you download the file of a failed refresh you will either get a single line why the whole refresh failed or details of which entity failed. there are occasions when the error is not clear, for example the second one shown.

In the first example the sales entity has failed due to a problem with the “Qty” column. This causes the other entities’ refreshes to be cancelled.

Conclusion

The Refresh History is an important part of understanding how a dataflow is performing. Hopefully at some point there will be an API to fetch the history directly, last time I looked there wasn’t.

Power BI Series

Power BI Dataflow – New Diagram View

Sat, 21 Nov 2020 22:32:08 +0000

In November 2020 a new diagram view was introduced to the online Power Query editor. The new feature produces a brilliant diagram of how the queries and parameters relate within the dataflow.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Displaying Diagram View

When editing your dataflow, click on the View ribbon tab and select Diagram View. The diagram view will appear between the ribbon and the formula bar.

Object Types Shown in Diagram View

The entities with a blue border are loaded entities, i.e. listed when connecting and the entities with a grey border are not loaded, so they refresh but are not exposed to the report builder. Parameters and functions have a relevant symbols in the top left and a grey border.

Query Steps Preview

If you hover your mouse pointer over the step counter, it will display a list of the query steps.

Expanding Display View boxes

In the top right corner of each query and function box there is a double-headed arrow button. Click this button to expand the query and show all the steps. The button is a toggle to switch between collapsed and expanded views.

If multiple queries are expanded, you can collapse all query boxes by using the toolbar in the bottom right of the canvas. It has collapse all and expand all buttons.

Displaying Step Names

By default an expanded query displays step labels, i.e. the name of the action being applied. You can display the names of the steps by clicking on arrow below Diagram view button and selecting Show step names.

Related queries can be highlighted using the model diagram button in the top right of the query box. It will highlight all related objects blue or grey based on their borders. In a complex dataflow this can be useful to understand the complexities.

Adding Steps

When a query is selected and expanded it displays a plus sign. Clicking on the plus button displays a list of possible steps to add, it will be based on the column you have selected. There is a search box to assist in finding the right step.

Clicking on the three dots in the top right hand corner offer multiple query steps to add to the query plus some other options.

Conclusion

Diagram view is a great addition to the online dataflow editor. Diagrams are a great way to explain a structure and this should help explain transforms within a dataflow. Personally I doubt I will be using the plus button to add steps, but then I am not the target audience.

Power BI Series

Power BI Dataflows – Endorsement as Promoted and Certified

Sat, 21 Nov 2020 20:55:18 +0000

Inform your users as to which dataflow to use by using Endorsement. This allows you to label a dataflow as promoted or certified.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Before Endorsement

When a report builder selects dataflow as the data source, the list of dataflows only shows the title of the dataflows. It gives no indication of the last refresh date or owner so if there are similar names your report builders could easily pick the incorrect dataflow.

In the above example there is Biscuits and Biscuit Data and no information to help a report builder decide which dataflow is the correct one.

Adding Endorsement

Endorsement has to happen in workspace. Click on the three dots next to the dataflow and select Settings. In settings, expand the Endorsement section to see three options.

None – default setting with no endorsement.
Promoted – this level can be added by the dataflow owner.
Certified – this level can only be added by owners who have been given permission by the tenancy admins. Details of how an admin can add be found in Endorse a dataset post.

Once you have selected the level, click Apply to save it.

After Endorsement

Now when a user selects dataflow as their connector they will see the same list of dataflows but they will be tagged as Promoted or Certified.

Conclusion

When someone has done the hard work to create a dataflow to assist the users in building reports it is important to let your users know which dataflow is the one to use.

Power BI Series

Power BI – Scheduled Refresh for your Dataflow

Mon, 09 Nov 2020 17:45:23 +0000

Most dataflows needs to have scheduled refresh setup so that the data will be up to date ready for the reports connecting to it. This post walks through the basic steps to set up the refresh.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Setup Scheduled Refresh

In the workspace containing the dataflow, there is no Next Refresh on the dataflow. Therefore the dataflow will not refresh automatically.

Click on the three dots on the dataflow row, they appear when your mouse hovers over the row. Select Settings from the options.

Expand Scheduled refresh.
Click on the toggle to turn on scheduled refresh.
Select refresh frequency, daily means every day, weekly means you can select which day of the week.
Select your time zone.
If you have selected weekly, select days.
Click on Add another time and enter the hour, minutes and am or pm for every time you want the refresh to happen.
Leave the tick so you you will be notified if the dataflow refresh fails.
Click Apply.

Once the above has been completed, we can now return to the workspace. The dataflow now has a Next Refresh date and time for next Wednesday at 8:30am.

Conclusion

It always fascinates me how people are not confident in setting up the refreshes of datasets and dataflows. Hopefully this quick simple guide will give enough confidence to setup the refreshes to happen without people logging into Power BI to refresh the dataset or dataflow manually.

Dataflow on Hat Full of Data

Using Variable Library in a Dataflow

Variable Library Series

Scenario

Getting the Value

Using the Variable Value

References for Variable Library in a Dataflow

Conclusion on using variable library in a dataflow

First refresh it broke!

SharePoint and Microsoft Fabric

Previously

It Broke

It Broke in the Staging

Turn Off Staging

Conclusion

Refreshing Datasets Automatically with Power BI Dataflows

Create a flow in Power Automate

Testing the Flow

When refreshing datasets automatically fails

Conclusion

More Power Automate Posts

More Power BI Posts

Power BI Series

Power BI Dataflow – Create Dataflow from Export

Dataflow Series

YouTube Version

Export dataflow as json file

Create dataflow from json file

Check Credentials

Conclusion

More Power BI Posts

Power BI Series

Power BI Dataflow – Refresh History

Dataflow Series

YouTube Version

Refresh History List

Refresh History Files

Calculating the Refresh Time

Showing Errors

Conclusion

More Power BI Posts

Power BI Series

Power BI Dataflow – New Diagram View

Dataflow Series

YouTube Version

Displaying Diagram View

Object Types Shown in Diagram View

Query Steps Preview

Expanding Display View boxes

Displaying Step Names

Related Queries

Adding Steps

Conclusion

More Power BI Posts

Power BI Series

Power BI Dataflows – Endorsement as Promoted and Certified

Dataflow Series

YouTube Version

Before Endorsement

Adding Endorsement

After Endorsement

Conclusion

More Power BI Posts

Power BI Series

Power BI – Scheduled Refresh for your Dataflow

Dataflow Series

YouTube Version

Setup Scheduled Refresh

Conclusion

More Power BI Posts

Power BI Series