Hat Full of Data

Humbled and Honoured

Wed, 15 Jul 2026 00:00:00 +0000

Today I got the news I get to be a Data Platform MVP for another year!

As tradition dictates I am of course Humbled and Honoured. In reality though, I’m hugely grateful to lots of people who let me be me in this community.

Hosts of user groups and conferences who create the place for us to meet and present
Helpers who make conferences work, tell me when to end a session and help me find my room
Humans who came to my sessions, read my blog, watched my youtube channel, asked questions and made it feel worth while. If any non-humans were involved I am of course grateful to them as well.
Hecklers¹
Handful of friends who have shared the highs, the hurdles, headaches and the hilarity of another year in tech.
Hubby, Harry, who has done the airport drop offs and pickups, cooked while I build slides and demos, never complains when head off to another conference or user group.

Turns out H is a great letter.

The award comes with a badge, an updated one this year, but in reality the award opened doors for me to an amazing community. We use the ever changing, name changing, icon changing² tech as an excuse to gather, but we stay because of the community of wonderfully diverse, usually neurodivergent and amazing humans we are.

A huge thank you to my co-speakers this year Claire Edgson, Joe Griffin, Lewis Baybutt and Will Johnson. We rocked!

Also to Mark Pryce-Mayer who makes me laugh and fixes the lakehouses I break, Sue Bayes who makes me smile every week, Alex Powers who encouraged me to keep going, Ian Tweedie who helped me transfer my blog from wordpress to Hugo, so blogging will again be enjoyable and Ash Graham-Brown who did the accessibility work to make sure a screen reader makes some sense on my blog.

And last, but never least, Rie Merritt who looks after us Data Platform MVPs, keeps us in line and always supports us non-males in a male dominated industry.

My advice to anyone looking in on the community, unsure how to join, please reach out. Come talk to a speaker after a session, ask the questions, be brave and say hello. Many of us are incredibly shy, anxious and imposter syndrome is our best skill, so we understand your shyness. You are one of us, I promise. Take a look at my Upcoming Events page and if you are going to be at one of them reach out to me beforehand and we can have a coffee together and I’ll introduce to people you want to meet.

¹ You know who you are
² Microsoft stop it!

Using Variable Library in a Dataflow

Wed, 11 Feb 2026 17:21:24 +0000

One of the popular low-code tools within Microsoft Fabric is the Gen2 Dataflow. Power BI report builders already know some Power Query. So armed with this knowledge is a popular starting point to load data into Microsoft Fabric. Adding values from the Variable Library in a Dataflow is an obvious plan to make it more future proof and to work better with Deployment pipelines.

I will confess the first time I tried these I could not get them to work till I read the instructions correctly. So they do work just understand the limitations!

Variable Library Series

Variable libraries should be part of every project. This post is part of my series to help get you started creating the library and then using the variables and finally seeing your hard work pay back when it comes to deployment pipelines.

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Scenario

For this post we are going to use the scenario of a dataflow query that lists projects. Whilst we are developing the data handling we want to limit the number of rows loaded and in production we will have no limit.

I create the dataflow and then using Keep Rows, I select Keep top rows. In the next dialog I enter 5 and click OK. This results in a table with only 5 rows of data. The number of rows though needs to come from the variable library.

Getting the Value

The next step is to get the value from the variable library into the dataflow. We are going to use a new Power Query function for this called Variable.ValueOrDefault. For the first parameter of this function, you need the Variable Library name and a Variable name. Then you combine together in a string.

1

"$(/**/<Library Name>/<Variable Name>)"

So for my example the string will be “$(/**/Finance Variables/Limit)”.

📝 Instructions

On the Home ribbon, expand Get data and select Blank query.

Rename the query to DataRows.

Then in the formula bar enter in the code below.

This will return the current value of the variable in the library.

1

Variable.ValueOrDefault("$(/**/Finance Variables/Limit)",2)

💡 Tip

The documentation as of publishing this post states it won’t work hence we use the OrDefault function and in this example the default is 2. It does work though, and we can see the answer 8 comes through.

Using the Variable Value

We’ve got the variable value from the variable library in a dataflow, now we need to use it. In the last line of my project query that limited the query to 5 rows I can replace the 5 with DataRows. I now get 8 rows of data.

This works as long as we always want to limit the rows, the chances are in production we don’t want to limit, so I add the extra of only limit if DataRows is greater than 0. Here is my new statement, the previous step is call Sorted Rows hence the #”Sorted rows”

1

if DataRows > 0 then Table.FirstN(#"Sorted rows", DataRows) else #"Sorted rows"

References for Variable Library in a Dataflow

Microsoft – Use Fabric variable libraries in Dataflow Gen2

Power Query Reference for Variable.ValueOrDefault

Conclusion on using variable library in a dataflow

Its great we can bring in the values easily. Its a shame that we can’t use them to control the destination, but that is on the road map for 2026 Q1. I’ll blog about it as soon as it arrives! I highly recommend using my pattern of fetching the value as a query and then referring to that. It will help in debugging etc.

Using a Variable Library in Lakehouse Shortcuts

Mon, 26 Jan 2026 18:43:51 +0000

Lakehouse shortcuts are a popular addition to the Fabric set of tools to access data easily without copying it. Using a variable library in lakehouse shortcuts means its easy to point shortcuts to an alternative location. This great for ALM using development, test and production workspaces.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Setting Up the Shortcuts

Using the standard GUI you have no option to use a variable library values in the setup of the shortcuts. So set up your shortcuts as you normally and make sure they work. For this post I’m using a shortcut to a table in warehouse. (I had a whole barny with lakehouses and schemas and it not working – another post for another time!)

According to the documentation using the rest API also does not allow for the use of variables in the creation of shortcuts.

Viewing the Options

If we select the short cut and click on the 3 dots to expand the menu we can then click on Manage Shortcut. This opens a pan that shows the values behind the shortcut. The values that be populated from a variable library have a Variable drop down, so we can see Target workspace, Target item and Target path.

I only want to populate the workspace and item guids, so I set up a those variables in my variable library. The variables must be set up as strings, they might look like guids but the shortcut demands they are strings. I set the default values to match the current values. When we apply each variables it will check the current values work so it only works if you use the current values.

Applying the Variables

Back in the Manage Shortcut pane click on the Variable below the current value to expand that section. Then click on Select variable to open the Select variable dialog. Click on the correct variable and then click Select Variable. This will update the value to come from the variable.

Once you have pointed it to the variable there does not appear to be a method to change back to a hard coded value.

Updating Variable Values

Variable values will get updated, the most common being a different active set being selected. For other artifacts the change will get picked up when they execute or refresh. For shortcuts this is slightly different. You need to force the updating of variables.

Open the lakehouse and stay in the lakehouse view. On the ribbon the last button is Update all variables. A message will show the updating is taking and finally there will be a notification to say they have been updated.

If one of the shortcut fails in the update please be aware the successful ones will have updated. You do get a notification and an error message that tries to help.

Conclusion on Variable Library in Lakehouse Shortcuts

I really like that the variable libraries can be accessed directly in the shortcut details, there is no pre-loading. I would prefer that I could change the values and then press apply so that each value isn’t checked individually. My experience in deployment pipelines means I fell foul of not realising I needed to refresh the variable values.

Resources

Creating Shortcuts – https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-shortcuts

Microsoft Learn – https://learn.microsoft.com/en-us/fabric/onelake/assign-variables-to-shortcuts

Using a Variable Library in a Data Pipeline

Mon, 19 Jan 2026 10:36:26 +0000

You’ve set up your variable pipeline with WorkspaceIDs and LakehouseIDs etc etc Now you want to use that variable library in a data pipeline so we can use the common values and they will behave in a deployment pipeline.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Adding the Variables to the Data Pipeline

Before you can use the variable library in a data pipeline you need to add the variables. In your data pipeline you need to click on the grey background to bring up the section on the bottom of the screen with Parameters, Variables etc. If you are in a brand new pipeline you need to click on the settings cog first, then it will appear.

Select the Library Variables section. Then click on +New and the select variable dialog box appears. If you have multiple variable libraries you can select the right one, you can also search for a variable in the top right. Once you select a variable click on Select Variable button to add that variable to the pipeline.

Yes you have to add each variable in turn, no multi-select. You can rename the variables but please don’t, name your variables in your library better.

Using Variable Values

You can use the variable values anywhere you can use Dynamic Content. Click on the Add Dynamic content link to open the Pipeline expression builder. Click on the three dots to select Library variables. Click on the variable value you want to use to enter in the formula into the expression. Click OK to put the formula back into the box.

You can use the variable value inside any functions if required.

Using Variables for Data Connections

When a data pipeline move from a dev workspace to test workspace and then onto production workspace we need to change the details of the connections to the new values. For this post we will use a connection to a Fabric warehouse in a workspace. I want to pull the values from a table in the warehouse using a lookup action.

I need to create the following variables in the library:

Project_Warehouse_Connection_ID
Project_Workspace_ID
Project_Warehouse_ID
Project_Warehouse_Connection_String

Connection IDs

For the connection guid you need to click on the cog on the top bar and select Manage connections and gateways. From the list of connections, find the one you are using. From the three dots menu select settings. The Connection ID will be listed and can be highlighted and copied.

Workspace, Warehouse and Lakehouse IDs

IDs for most artifacts can be found when it is opened. Be aware groups=workspaces in the url. Lakehouses have 2 IDs, one for the lakehouse and one for the SQL analytics endpoint or lakewarehouse as the url names it. No comment on names to confuse us!

Connection String

The final part needed is the connection string for the warehouse or lakehouse. The easiest way to copy this is to open the lakehouse SQL endpoint or the warehouse and in the bottom left hand corner is a Copy SQL connection string link.

It can also be found in the lakehouse or warehouse settings under SQL analytics endpoint.

Variable Library

All these values need saving into the Variable Library. I’ve used Guid for the IDs and String for the connection string.

Using the Variables in the Activities

We now need to use the variable values from the variable library in a data pipeline activity connection. We start by adding all 4 variables to the library. Note the guids become strings.

In this post for simplicity I’m using a Lookup activity to just get all the rows from the projects table. So I add a lookup activity and fill in the general details. On the Settings tab I click the drop down for Connection and select Use dynamic content. In the Add dynamic content pane I find Library Variables under the three dots and select FinanceVariables_Warehouse_Connection_ID so it appears in the expression box. Then I click OK for the Connection to now be an expression.

Once the connection has been populated you can select the connection type, in my case Warehouse. Then the text boxes appear to take the Workspace ID, Warehouse ID and SQL Connection string. They all have a link to Add dynamic content.

The table drop down can be populated by entering manually or by clicking Refresh. This will open a pane showing the values that are going to be used in the connection. Click OK to start it fetching the list of tables. Once a table is selected if you click Preview data the same pane appears confirming values. Its a great way to test the connection works.

You now have a variable values from a variable library in a data pipeline activity. Well done!

Conclusion on Using a Variable Library in a Data Pipeline

I like how simple it is to use the variables and that you get to select the variables needed in this data pipeline. I’d prefer if you could add multiple variables at a time. I like the consitent use of the expression builder so its something we get familiar with.

I’d also really like it if in lineage view you could see which artifacts are using of the variable library and the impact was also included. Maybe oneday!

Resources

https://learn.microsoft.com/en-us/fabric/data-factory/variable-library-integration-with-data-pipelines

Accessing a Variable Library in a Notebook

Mon, 12 Jan 2026 09:34:51 +0000

This post walks through how to access a variable library in a notebook in Microsoft Fabric. I recommend a Microsoft Fabric project starts by creating a variable library to store the common values different artifacts need and could be changed if a deployment pipeline gets involved. So when we create a notebook we need to be able to use these variables. This means we need load the variable library in a notebook and then get the variable values.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Connect to Library

The first task is to create a variable for the variable library. The notebookutils package includes the functions we need to do this easily. In the code below the variable vl will refer to the variable library that is called “Finance Variables”. This code assumes the notebook is in the same workspace as the library.

1
2


# Get the Variable library
vl = notebookutils.variableLibrary.getLibrary("Finance Variables")

Get Variable Value

Once we have the variable library loaded, we can access the variable values in one of two syntaxes.

1
2
3


# 2 syntaxes to get variable value
SP_URL = vl.getVariable("SharePoint_URL")
Limit = vl.Limit

Combining both of the two and adding some print statements we can demo the above using this code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Get the Variable library
vl = notebookutils.variableLibrary.getLibrary("Finance Variables")

# 2 syntaxes to get variable value
SP_URL = vl.getVariable("SharePoint_URL")
Limit = vl.Limit

# Check results
print(f"{SP_URL=}")
print(f"{Limit=}")

And here is the run

Get a variable value by reference

There is a third way to get a variable value. The get method from notebookutils variable library uses a reference string that follows the following syntax:

1

"$(/**/<LIBRARY NAME>/<VARIABLE NAME>)"

Before you get excited that the ** implies you could refer to another workspace, sorry that is not supported. This string is used in the variableLibrary.get action in notebookutils. For example to get the two values from before could be done like this

1
2
3


# Get the variable values directly
SP_URL = notebookutils.variableLibrary.get("$(/**/Finance Variables/SharePoint_URL)")
Limit = notebookutils.variableLibrary.get("$(/**/Finance Variables/Limit)")

And here is the run

💡 Tip

The print(f"{Limit=}") prints Limit=4 for us newbies to Python!

Naming Conventions

When you start and there are only 4 variables in the library its easy to remember the names to type in by hand when you are using a variable library in a notebook. When it gets to 20 names, its harder. So create standards, whats the case pattern, you using _ between words or not? Pick a set of rules and stick with it, BronzeWorkspaceID works just as well as Bronze_Workspace_ID, WorkspaceID1 and workspace_id2 are a technical debt you don’t need.

Conclusion on using Variable Library in a Notebook

I’m really impressed on how easy it is to use the variable library in a notebook. The pattern is simple and resusable. If I had a magic wand it it would be great is the intellisense knew the variable names from loading the variable library object.

The method by reference is great if you want to dynamically select which variable to load as the string could be dynamically built.

Using a combination of parameters and accessing a variable library means reusable notebooks are easier to write. This needs to become part of the best practice pattern being used.

References

Microsoft learn covers Variable library utilities as part of the notebook utilities page.

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#variable-library-utilities

Happy New Year and 2025 Review

Wed, 07 Jan 2026 10:52:18 +0000

Welcome to 2026!

In a world full of AI generated content I am very proud that none of the content here on this blog will be AI generated. Mostly because here is my personal notebook of how to do stuff. I still refer back to my first posts from April 2019 on writing SVG. That copy button in code blocks I sorted for my own sanity.

In the 5+ years of writing this blog I’ve posted 128 posts, had 600k+ views, way too many spam comments and had a subscribe button which over 200 people clicked but never sent an email. I’m hoping this is my first post that will email out to subscribers. Apologies for not sending before and for those who had forgotten they subscribed 3 years ago.

Blog Activity in 2025

17 new posts
140k views

I was not as active as I’d like to have been in 2025. No good reasons but it gives me some new goals for 2026. I’m aiming for more regular posts, using tools so I’m not overwhelmed by spam comments.

The most popular post on this site is how to pull data from a Power BI dataset using Power Automate, it got 10% of last years views. The second place post was Power Automate to trigger a Fabric data pipeline. Just goes to show I should still do the Power Automate working with data stuff posts and add videos.

YouTube Activity in 2025

2025 was also not a good year for video creation for me. I only made 3 videos! Amazingly I got 170k views and gained 800+ subscribers. Another goal for me to improve on video creation and maybe get over that 10k subscribers. Lets see what we can do. For those who haven’t subscribed here is the link

https://www.youtube.com/c/hatfullofdata123

Do let me know if there are videos you think I should do.

Speaking

I had a fun year of going to conferences and user groups. It is my second family of speakers and regular attendees and I am incredibly grateful to those who volunteer to make these events happen. From a user group with only 3 attendees to a conference hall with 100 people in it, I love the buzz of sharing knowledge and being a community.

In May last year I started putting my PowerPoint decks on github with other resources to support sessions. If I’m honest its so I know where I’ve stored my decks when I need to refer to it for a future session or remember what I did last year! Here’s the link

https://github.com/HatFullOfData/Sessions

Looking Forward

There will be more blog posts, there will be more videos. I enjoy doing them, its helps my mental health. I already have conferences planned that I will be at, speaking or attending. (Love a conference that I just attend occasionally)

Here are a few of them, if you are going to be there, please come say hello.

Microsoft Fabric – Variable Library

Tue, 06 Jan 2026 13:27:41 +0000

This post to help you get started creating a variable library. When multiple dataflows, notebooks and pipelines are using the same details to perform tasks it helps if those values are stored in one place. When you move to use deployment pipelines and those values change from your development workspace to your test workspace to your prod, it helps if that is easy. The solution in Microsoft Fabric is a Variable Library to store those common values.

Variable Library Series

Getting started with variable libraries
Variable Values in a Fabric Notebook
Variable Values in a Data Pipeline
Variable Values in Lakehouse Shortcuts
Variable Values in Dataflows
Variable Libraries in Deployment Pipelines

Create a Variable Library

This post assumes the workspace is backed with a Fabric capacity. A trial capacity works just fine.

Either from a Develop task add item or from the Add item button select Variable Library. Enter in a meaningful name for the variable library, letters, numbers and spaces are allowed. Then click the Create button and your new library will be opened. I recommend you create a naming convention that all your Fabric developers like and agree to use.

Add a Variable

Your library starts empty, so the next task is to add variables. You can create a new variable row by clicking on + New Variable on the Home ribbon. When there are no variables there will also be a green button in the center of the page to create a new variable.

Enter in a meaningful name and select a data type. A name can only include letters, numbers and underscores. An incorrect name will prevent the library from being saved. Please note some types are not usable in some Fabric artifacts, no I haven’t worked out the logic yet but I’m working on it. Enter in a value for the Default value set. Value sets are used when you are using deployment pipelines for the different values you use development,test and production workspaces. Creating and using variable sets will be covered in another post.

Adding Notes

So that “future you” doesn’t hate the “current you”, it is worth adding notes to variables to explain the purpose of the variable. This should be part of the best practice rules for Fabric development.

Click on the icon in the note column. When there is no note it has a plus symbol and when there is a note it has a pencil symbol.

Saving

Variable libraries do not auto-save. You need to click the save button on home ribbon. Indicators of black dots will show unsaved items you have open and the save button will only be active when there are changes to save.

Conclusion

I highly recommend starting any Fabric project with creating a variable library. Then everytime a value is used in an artifact such as a database name, key vault name etc a descision is made on if it should be in the variable library or if the value is already there. Some values to be more useful will need to be split across multiple variables.

It should always be assumed that a workspace with Fabric items will eventually be in a deployment pipeline. This means that it should be coded to make deployment easier. The aim of these articles is for quick reference on how variable libraries are used across all the artifacts.

References

Microsoft have some getting started articles on setting up a variable library and using it in a deployment pipeline.

Microsoft Learn Variable Libraries

Microsoft Fabric – Using Copy Job with Dataverse

Thu, 27 Nov 2025 15:35:41 +0000

Dataverse is the data store behind parts of Dynamics and lots of Power Platform projects. So Dataverse can contain vital business data that will be needed for reporting. In this post we are going to look at one method which is using copy job with Dataverse to copy across data in Microsoft Fabric.

Scenario
First Copy Job – Dataverse Connector
Copy Job Run
Copy Job using SQL Server connection
Incremental and Merge Running
Trouble Shooting
Alternative Options to using Copy Job with Dataverse
Conclusion

Scenario

For this post we will look at 2 tables, Accounts and Contacts so all the code will be common to any Dataverse environment. I have 5 Account records and 10 Contact records. I also have the environment path org29d4c6d3.crm11.dynamics.com.

In Microsoft Fabric I have a workspace, Finance Reporting, with a Lakehouse, Finance_Lakehouse.

First Copy Job – Dataverse Connector

Copy Job is a great artifact to move data easily into Fabric. Simplest version just copies data, we can then add incremental and merging. More details can be found here https://learn.microsoft.com/en-us/fabric/data-factory/what-is-copy-job

For the first example we will copy from Dataverse using the Dataverse connector. The wizard that walks you through creating a Copy Job has 6 steps. For this first one we will walk through each step.

Create Copy Job

Either using a task flow Get Data’s add item or using Add item in the workspace, select Copy job. Type in a name and click Create.

Choose Data Source

Next step is connecting to Dataverse. For this example we are using the Dataverse connector. Click on the Dataverse option under New sources. When the next screen appears enter in the Environment domain. If it is your first time connecting there you will be given the option to name your connection and select a privacy level. It will guess the authentication and login – this is not the blog to discuss options here.

Once these are done click Next

Choose Data

Choosing the data is very simple, the next screen lists the tables from dataverse using their display names and logical names in brackets. Yes they call them Entities…. No comment. If you expand a table the logical names of fields are listed. There is a search box that will allow you to search for a table using either of the names. If you have selected a table the search will expand to the field names as well. Interestingly for choice columns you only get the code column, not the name column. That is going to be another post!

If you select a table all the columns will be imported. You select the required columns manually by selecting individual columns, its a touch manual and requires searching and scrolling lots in my experience. Next version is better in my opinion – feel free to scroll down!

When you’ve selected your tables and columns click Next.

Choose Data Destination

Data destination is another easy step. Select your lakehouse from the OneLake catalog.

Choose copy job mode

With the dataverse connector you only get one option, Full copy. In the Map to destination you get to choose if that is an append or overwrite.

Map to destination

The next screen shows update method and the tables being imported. Clicking on Edit update method will give you the option of Append or Overwrite. Append is the default but this will just duplicate you data every time you run this copy job. I have changed it to Overwrite. Merge is not available in this version.

You can enter names for your destination tables and click Edit column mappings to tweak column names or remove some mapping. The columns selected will match the columns you selected back in Choose data.

Click Next once you are happy.

Review + save

The final screen will give you an overview of what the copy job is going to do. You can select if you want it to run now, run once or scheduled. I’m choosing to let it run so I’ll just click Save + run

Copy Job Run

When a copy job runs such as after saving in the previous steps you can see the execution happening. Each table is listed and when it succeeds you will see how many rows were read and how many were written. In our example read and written are the same.

Once the run has completed we can go check the Lakehouse and we can see tables. It might need a refresh to show.

Copy Job using SQL Server connection

Although the dataverse connector works, the selection of columns is very manual. You also get no option for incremental updates or using merge. So the next example will use SQL Server connector. In a previous post https://hatfullofdata.blog/using-sql-on-dataverse-for-power-bi/ I show how you can use SQL on Dataverse with some great hints from Scott Sewell.

From that post and with the addition of the modifiedon field here is the SQL for the accounts table. We will use this later.

1
2
3
4
5
6
7
8
9


Select
	accountid as AccountID,
	name as Account,
	industrycodename as Industry,
 modifiedon,
	statecode,
	statecodename
from
	dbo.account

New Copy Job using SQL Server

Create a copy job as before giving it a name. For the connector select SQL Server. The Server is the environment path as used before, e.g. org29d4c6d3.crm11.dynamics.com. Remember to edit the connection name and privacy level. There is no point adding a Database, it makes no difference. After you press next you need to wait for the database list to load and then select the only database offered.

Choose Data

The next screen for choosing the data starts by showing you an error. Ignore that, it means you can’t use the interface to select your tables and columns but you can add queries. Expand Queries in the bottom left and click on New query. Type in a name, eg Accounts and then in the query box on the right put in your query. You can check it returns the right data by clicking on preview data.

Repeat the above for each of your tables and then click Next.

Incremental and Merge setup

When you choose your lakehouse in the data destination and move onto the copy job mode screen you now get 2 options, Full copy and Incremental copy. The incremental copy will initially copy everything and then in the future will only copy across changes. In order to recognise which records have changed you need a data time column of when it last changed. Hence we have included the modifiedon column.

So incremental is only going to process updated or new rows, but what do you want done with those new rows? Append will add the newly updated rows giving you the complete history of each row. Overwrite you will loose rows that have not been modified. Merge is currently in preview but would allow you to update the row that matches on a unique field, e.g. Account Id.

Click Incremental and then Next. In the Map to destination screen, click on Edit update method and select Merge and then select the unique id column for each table. When you click next you will be prompted to select the incremental date time column for each table. Click next to the final stage.

Save, Review and Schedule

On the final screen not only do you get to save and review but you also get to schedule due to us picking incremental. You can pick repeat by the minute or hourly and pick dates etc. Be aware incremental requires more compute so will consume compute faster than a straight copy but on huge tables the reduced copy will be worth it.

Select your options and Save and Run

Incremental and Merge Running

The first time we run just as before the rows read and written include all the data. If we go and edit one row in the accounts table and re-run the copy job, only one row is read and written for the accounts table and nothing for the contacts table.

Trouble Shooting

In preparation for this post I’ve hit a number of issues.

Although there is a back button in the wizard, going back to the data source and tweaking the queries just breaks the copy job.
You name a column in the SQL with a space, that update will fail with a weird error. No spaces in column names!

Alternative Options to using Copy Job with Dataverse

Link to Microsoft Fabric

Using the Link to Microsoft Fabric in Dataverse sounds like a great option. You will be able to add shortcuts in a Lakehouse to the data and it will all be awesome.

Just after you select the workspace that the data will go to there are 2 short sentences that you should read carefully.

“We will link all tables so that you don’t have to choose. You may incur additional charges.”

I’ll not comment on the “so you don’t have to choose” as I’m not sure I have anything polite to say. That additional storage is in Dataverse which could cost $40 per 1GB per month and with all the tables that could easily become a huge expense, even empty tables take up space. All this so we can have a shortcut in Fabric.

Platform Guardian blog has a great post to walk you through doing thishttps://platformguardian.blog/2023/11/05/creating-the-dataverse-connector-in-fabric/

Conclusion

Copy Job is one of my favourite Fabric artifacts because they are simple to set up. Editing them is foul so I end up re-building them. Its a brilliant low-code solution to copying across data from dataverse and into a lakehouse. Be aware incremental is going to eat capacity so don’t bother with small tables just do a full copy over writing every time.

I don’t see any benefit using the dataverse connector except you don’t have to write SQL. A simple select statement is not complex I’m sure an AI prompt somewhere will help you out.

Microsoft Fabric – Direct Lake and User Permissions

Fri, 07 Nov 2025 19:40:06 +0000

Direct Lake is the best way to build a semantic model on data in a Lakehouse or a Warehouse. We can then use Power BI to build reports and possibly even a make a Workspace App to deliver those reports to the business. So what permissions do the business users need to view those reports? This post will explore using Direct Lake and User Permissions to make sure everyone can see data in the reports.

IMPORTANT

This is the simplest method of fixing permissions so someone can view a report using direct lake. If your lakehouse or warehouse contains data they should not have access to this is not the method to use.

Scenario

For this post I have built the following :

Workspace that has a workspace app built and up to date.
Lakehouse_Report connected to a semantic model, Lakehouse_Model, that has a direct lake connection to a lakehouse, Permissions_Demo_Lakehouse
Warehouse_Report connected to a semantic model, Warehouse_Model, that has a direct lake connection to a lakehouse, Permissions_Demo_Warehouse
Both the Warehouse and Lakehouse contain one table, Projects with 10 rows of data. Both reports show this in a chart.

I have given one user, Dan, Vistor access to the workspace and I’ve given the Project team security group access to the workspace app.

Both reports work fine for me as the workspace owner. The report is direct lake and user permissions are admin.

Viewing Reports as Dan

Viewing the report as Dan is an example of a report that is direct lake and user permission is viewer. Dan can see the warehouse data but he cannot see the lakehouse data.

Viewing Workspace App as Sam

Viewing the report as Sam, who is a member of the Project team security is an example of a reports that are direct lake and user permission is app only. Sam cannot see either the lakehouse or the warehouse report charts.

Permissions

Whenever data is not coming through for some but not others it is usually an issue with permissions somewhere. Everyone can see the report, they get the headings. The connection is direct lake into the semantic model we need to look at user permissions where the data lives, the lakehouse and the warehouse.

Each item in the workspace has permissions. By default permissions are inherited from the workspace. There are two fixes to do. The first is Dan’s access to the lakehouse and the second is project team access to both lakehouse and the warehouse.

Fix Dan’s Lakehouse Access

In the workspace, click on the three dots on the lakehouse. Then from the menu select Manage permissions. This shows a list of the people with access and the Role which gave them that access. We can see Dan has Read and ViewOutput

Read only gives the user permission to connect to the lakehouse, it does not let them see the data. Neither does ViewOutput. I’d love to find documentation as to what ViewOutput allows but I’ve not found it yet. I will update this post when I do.

Click on the three dots at the end of permissions and select Add ReadAll. Now when Dan looks at the reports the Lakehouse one will work. Sometimes it takes 10-15 minutes for the permissions to work..

Fix Sam’s Access via the Workspace App

Sam gains access to the workspace app through being a member of the Project Team security group. The security group is not mentioned in the permissions of the lakehouse or warehouse. If you go look it is listed in the permissions of the semantic model as App.

In the lakehouse permissions, click on +Add user. In the Grant people access dialog add the user group or people into the box. Decide if you are emailing them and add a message, I’m not emailing anyone an auto generated email! I have not ticked any of the additional permissions. Then click Grant.

This will give them Read access. Then as we did for Dan, click on the three dots and click Add ReadAll and this.

Warehouse permissions are slightly different. Dan has Read and ReadData permissions which gives enough access for him to view the report. So we can use the same process as on the lakehouse and duplicate those permissions. Note that in the Grant people access dialog no additional permissions were ticked.

And after a few minutes when Sam refreshes the Workspace App they can see the reports both working.

Note!

The permissions given do mean that if Sam now has access to run SQL against the Warehouse but not the Lakehouse which is behind the semantic model so any row level security in the model will not apply. They do allow Sam to explore the data in both the semantic models.

Resources

The reason I wrote this blog post was I couldn’t find a resource to show the simple fix of manage permissions. Lakehouse sharing and permission management – Microsoft Fabric | Microsoft Learn is the closest I got.

Conclusion

Giving someone access to the whole lakehouse or warehouse could be dangerous. This post gives the simplest of fixes assuming the report viewer is allowed to see all the data with a report that is direct lake and user permissions are read only in the workspace or app only.

With new security features now available we can make the permissions fix more complex requirements. Handling Direct and User permissions needs careful consideration and should be discussed as part of your data governance.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

Power BI – Use ISINSCOPE to improve a Matrix

Mon, 22 Sep 2025 09:07:40 +0000

I recently had a request for totals in a Matrix to be shown for some columns and not for other columns. My solution was to use ISINSCOPE in a Matrix to work out which level of the matrix to choose what to return.

Scenario

At an event you sell tickets. So for the Cheese and Wine evening we can see 62 tickets are sold so we expect 62 people. BUT some tickets are for multiple people such as Family Ticket is 2 adults and 3 children so the 55 tickets sold for the Christmas Fayre means very little as a Family ticket is 5 people and a Group ticket is 10. So total tickets for more than one ticket type does not make sense and is misleading.

So I’d like the Tickets Sold measure to not show a value on rows combining the ticket types, ie the total rows on this matrix.

Determining the Ticket Type Rows

The first step is to determine which rows are the ticket type rows and which rows aren’t. For this we can use the DAX function ISINSCOPE.

For demo purposes I created a measure that returns a true or false using the function ISINSCOPE. We can see it returns true on the ticket type lines but not the event or total lines. If we want to show all the layers we could look at all the fields in the matrix in a measure. Make sure you start with the lowest level first, so in my case Ticket Type.

1
2
3
4
5
6
7


Scope =
SWITCH (
 TRUE (),
 ISINSCOPE ( Tickets[TicketType] ), "Ticket Type",
 ISINSCOPE ( Event[Event Name] ), "Event",
 "Total"
)

As you can see, using the above measure we can tell which level of the matrix a value is on. So we only need to return total tickets if the ticket type is in scope and we can return nothing on other levels.

Final Measure

We can use a simple if statement based on the ISINSCOPE for Ticket Type. That works really well and the customer were partly happy.

1
2
3
4
5


Tickets Sold =
IF (
 ISINSCOPE ( Tickets[TicketType] ),
 SUM ( Bookings[Qty] )
)

Next Request

Once we showed that could be done they wanted to combine the 2 columns of numbers. The request was to not have a separate Total Sales column but to show the Total Sales value as in the total row for each event and the grand total at the bottom. For this we wrote a new measure called Event Sales.

1
2
3
4
5
6


Event Sales =
IF (
 ISINSCOPE ( Tickets[TicketType] ),
 SUM ( Bookings[Qty] ),
 SUMX ( Bookings, Bookings[Qty] * RELATED ( Tickets[Price] ) )
)

We can see in the matrix above the money value appears in the total rows and the number of tickets in the ticket type rows. But they are all formatted the same so it is not clear what the numbers mean.

Adding Dynamic Formatting

This is not going to be a full description of dynamic formatting, that needs its own post. For the Event sales measure we select Dynamic formatting. In the formula we use the same pattern as the measure for the two different formats.

Resources

Microsoft Learn ISINSCOPE – https://learn.microsoft.com/en-us/dax/isinscope-function-dax
DAX Guide ISINSCOPE – https://dax.guide/isinscope/
Hat Full of Data Dynamic Formatting – Its coming!
Microsoft Learn Dynamic Formatting – https://learn.microsoft.com/en-us/power-bi/create-reports/desktop-dynamic-format-strings
SQLBI – https://www.sqlbi.com/articles/introducing-dynamic-format-strings-for-dax-measures/

Conclusion

Adding ISINSCOPE into your repertoire of functions opens up lots of possibilities. For the start of this post it removed a confusing total, in the development of that Matrix it added an extra dimension to a measure. Be careful to not make it too confusing, I know some will dislike the one column having two values in it.

Power BI – Counting Active records easily

Mon, 15 Sep 2025 16:58:48 +0000

Recently in a few Power BI projects and training courses we’ve needed to create a measure for counting active records based on start and end dates for a time period. This has varied from active projects, active employees to active contracts. All of them have had the common feature of start and end dates.

Scenario

Task	Start	End
A	01-May	20-May
B	15-May	15-Jun
C	5-Jun	25-Jun
D	20-Jun	20-Jul
E	10-Jul	25-Jul
F	25-Jun	15-Jul

We have 6 tasks A-F. We want to know how many tasks are active in June. In the list in table above and shown in the image the 2 that don’t count are task A, that finishes before June starts and task E that starts after June finishes. I have a separate calendar table called Calendar.

Understanding the logic of the filters needed is important when writing a measure. For a task to be active in a period, the task must finish after the start of the period and the task must start before the period ends.

DAX Code for counting Active Records

In DAX we are going to use CALCULATE function to apply the above filters. _MinDate and _MaxDate give us the start and end of the period.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Active Tasks = 
VAR MinDate = MIN('Calendar'[Date])
VAR MaxDate = MAX('Calendar'[Date])
VAR Result = 
 CALCULATE(
 COUNTROWS(Tasks),
 Tasks[Finish] >= MinDate,
 Tasks[Start] <= MaxDate
 )
RETURN Result

This code does not rely on any relationships between the tasks and calendar tables. If there is a relationship that is altering the calculation a CROSSFILTER function can be used to set the relationship to None.

It can be used in a chart to show 3 active tasks in May, 4 in June and back to 3 in July.

This is a simple example of using CALCULATE and creating a useful measure. One I’ve taught a few times and promised to document.

Resources

Conclusion on Counting Active Records

Yes this is another post on me being lazy to answer a question I end up answering lots. Understanding how you can use CALCULATE and how to work out the right filters for your logic is the crux of understanding quite a bit of DAX. Something I am still learning every day still working on.

Lewis and Laura vs. Notebook and Microsoft Graph: Part 2

Tue, 26 Aug 2025 13:07:20 +0000

A recent project made me reach out to Lewis Baybutt to scream “Help!, I need help getting with authentication for an Excel file in SharePoint in a Microsoft Fabric Notebook”. Yes all the delights of Excel and SharePoint in one project. So started the battle with understanding authentication, permissions and apparently a Notebook and Microsoft Graph is the way.

The reason for the project was the original solution was using Gen2 Dataflows and we ran out of capacity. Datameerkat has done a nice summary of comparing Dataflow to Copy Activity to a Notebook, https://datameerkat.com/copy-activity-dataflows-gen2-and-notebooks-vs-sharepoint-lists. Its for lists but the same logic works for files in SharePoint. So for this project we selected the Notebook route.

Final Goal of Notebook and Microsoft Graph

On a SharePoint site there is an Excel file that contains some data I want to load into a table in a Microsoft Fabric Lakehouse. The Pandas library includes a method called read_excel that can pull the data into a dataframe if it has the file content. And we know how to write a dataframe into table, so the real goal is to get the file content of a file on a SharePoint site.

We save the details of file name etc into variables to use later.

1
2
3
4
5


sharepoint_domain = "YOURDOMAIN.sharepoint.com"
site_name = "FabricDemo"
library_name = "Documents"
file_name = "PandaDemo.xlsx"
sheet_name = "Demo 1"

Service Principal, Permissions Etc

The first part of this series is definitely not my skill set, it involved Power Shell! So we agreed he would write the first part and I would write part 2. So Lewis has covered that topic in Part 1 which can be found here https://www.lewisdoes.dev/blog/lewis-and-laura-vs-fabric-notebooks-and-microsoft-graph-part-1/

The credentials created in the post should be stored in Azure Key Vault. I wrote a blog post how to do that and another one how to retrieve it into notebook

I created a code block to create three variables, tenant_id, client_id and client_secret. Swap in your TENANT ID etc to make it work for you. The notebookutils library is always loaded in a Microsoft Fabric Notebook.

1
2
3
4
5
6
7
8


# Authentication details
tenant_id = "TENANT ID"
client_id = "CLIENT ID"
# Get secret from Key Vault
azure_key_vault_name = "VAULT NAME"
azure_key_vault_secret_name = "SECRET NAME"
azure_key_vault_url = f"https://{azure_key_vault_name}.vault.azure.net/" 
client_secret = notebookutils.credentials.getSecret(azure_key_vault_url,azure_key_vault_secret_name)

Requests Library

This whole post is lots of HTTP get requests made way easier using the Requests library. Documentation for this library can is here Requests: HTTP for Humans™ — Requests 2.32.5 documentation

We use two patterns very similar, obviously there is a import requests statement in a previous code block.

1
2
3


reponse = requests.post(url , data=token_data)
# Raise error if request fails
response.raise_for_status()

1
2
3


reponse = requests.get(url , headers=headers)
# Raise error if request fails
response.raise_for_status()

Get Access Token

Before we can use Notebook and Microsoft Graph together we need an access token. For all the requests.get we need headers which includes an access token. So the next task is to get that access token which uses the first pattern from above. If you’ve used the tenant_id, client_id and client_secret variables you can just copy and paste this code into a block

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


token_url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token"
token_data = {
 "grant_type": "client_credentials",
 "client_id": client_id,
 "client_secret": client_secret,
 "scope": "https://graph.microsoft.com/.default"
}
response = requests.post(token_url, data=token_data)
response.raise_for_status() # Raise error if request fails
access_token = response.json().get("access_token")

# Print the result
print(" Access Token Received:", access_token[:50], "...")

headers = {"Authorization": f"Bearer {access_token}"}

The last step is to put the access token into a JSON object called headers which will be used for the next stages.

Getting the Site ID and Drive ID

We need the url to download the file content. The request to get that url is

https://graph.microsoft.com/v1.0/sites/{site_id}/drives/{drive_id}/root:/{file_name}:/content

So we need the site_id and the drive_id. Btw drives are document libraries… yeah naming confusion, don’t blame me! We already have the file_name, if its in a folder the name needs to include that.

Site ID

Using https://graph.microsoft.com/v1.0/sites/{sharepoint_domain}:/sites/{site_name} with the headers we built earlier in a requests.get we get a json response that includes an id value

1
2
3
4
5


site_id_url = f"https://graph.microsoft.com/v1.0/sites/{sharepoint_domain}:/sites/{site_name}"
print("Site ID URL:",site_id_url)
response = requests.get(site_id_url, headers=headers)
response.raise_for_status() # Raise error if request fails
display(response.json())

Produces

So we then use the following code to get the site_id and print it for debugging.

1
2


site_id=response.json()['id']
print("Site ID:",site_id[:50], "...")

Drive ID

Using https://graph.microsoft.com/v1.0/sites/{site_id}/drives?$select=name,id will give us a list of the document libraries on the site. A site can have more than one library so we need to handle multiple rows being returned. My choice was to save it into dataframe and then filter that to get the result.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


drive_id_url = f"https://graph.microsoft.com/v1.0/sites/{site_id}/drives?$select=name,id"
response = requests.get(drive_id_url, headers=headers)
response.raise_for_status() # Raise error if request fails

# Convert response json into a dataframe
df_drives = spark.createDataFrame(response.json()['value'])
display(df_drives)

# Filter the dataframe to the specified library and get the id
drive_id = df_drives.filter(col("name")== library_name).collect()[0]["id"]
print("Drive ID:",drive_id[:25], "...")

That gives this output

So now we have the site id and the drive id so now we can get the file content using our Notebook and Microsoft Graph.

Get the File Content

This uses the same pattern as the previous 2 requests, the difference is the path includes /content on the end to indicate we want the content. There is very little we can print out to debug the response as the file contant is binary

1
2
3
4
5


# Retrieve the File Content from SharePoint using Graph API
file_url = f"https://graph.microsoft.com/v1.0/sites/{site_id}/drives/{drive_id}/root:/{file_name}:/content"
print("File URL:",file_url[:75], "...")
response = requests.get(file_url, headers=headers)
response.raise_for_status() # Raise error if request fails

Pandas and File Content

Pandas read_excel method wants a file type to work with. Using BytesIO from io library we can create that. Documentation for BytesIO can be found here https://docs.python.org/3/library/io.html. We can then use read_excel from Pandas library to fetch the data from the sheet. I’ve kept it simple, the tricks on handling more complex data in Excel is for another post.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Import methods
from io import BytesIO
from pandas import read_excel

# Convert response
xls = BytesIO(response.content)

# Get data from sheet
df = read_excel(xls, sheet_name=sheet_name) 
display(df)

From the above code we get the following in a panda dataframe ready to save into a table.

Final Notebook

The complete notebook that could be imported into a Fabric Workspace can be found here https://github.com/HatFullOfData/CodeStuff/blob/main/Sample%20Files/Access%20SharePoint%20using%20Microsoft%20Graph.ipynb

Conclusion

Congratulations on getting this far with Notebook and Microsoft Graph. The above looks like hard work when connecting from a Gen2 Dataflow is so much easier. But the saving on capacity usage and the flexibility of a notebook I think gives us more options in the future.

Get Secret from Azure Key Vault in a Microsoft Fabric Notebook

Thu, 21 Aug 2025 16:33:55 +0000

I’ve recently been working with Notebooks in Microsoft Fabric and have needed to get a credentials information that should be kept secure. The obvious place to store such credentials is Azure Key Vault so now we need to get our secret from there.

This is a really short post to help me out when I need a fast explanation on projects.

Azure Key Vault Part

I’ve previously blogged how to setup a key vault and add a secret – https://hatfullofdata.blog/create-azure-key-vault-to-store-id-and-secret/.

For this post I have a vault called LGB-Secrets and it has a secret called SPConnectorSecret.

The vault URI is https://lgb-secrets.vault.azure.net/ which can be calculated from the vault name.

Python to Get Secret from Azure Key Vault

Notebookutils is a built in package for Microsoft Fabric notebooks. It was previously known as MSSparkUtils. It contains plenty of really useful methods to perform tasks with files, environments and credentials. Microsoft’s documentation can be found at :

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#credentials-utilities

The method notebookutils.credentials.getSecret requires 2 parameters, vault URI and secret name. This code block looks like this.

1
2
3
4


azure_key_vault_name = "lgb-secrets"
azure_key_vault_secret_name = "SPConnectorSecret"
azure_key_vault_url = f"https://{azure_key_vault_name}.vault.azure.net/" 
client_secret = notebookutils.credentials.getSecret(azure_key_vault_url,azure_key_vault_secret_name)

The account being used to run the notebook will need access to the key vault.

Reacted!

The notebook understands that the value you fetched from the key vault should be kept secure. So if you try to print the client_secret it will not show you the result and will show REDACTED instead.

Conclusion

I know this is a very short post, which is a good thing as that means its a simple concept. It is here because I also wanted a short post that contains the code block I want to be able to copy and use easily.

Power BI – Improve a card with SVG

Tue, 29 Jul 2025 11:00:32 +0000

The not so new card in Power BI, introduced in 2023, includes some lovely features. The first is the simplest of adding multiple values and having cards laid out evenly in a row in a single visual. The second awesome feature adding an image to a card with SVG.

Starting Point

This report is visualising web site views per year. In the report there are three measures, Views, Views YTD LY and Growth. In the picture above we can see views have gone down by 19% but its not very visual and takes up three cards. This post is to walk through changing that first card to show an arrow indication growth direction and the %. Something similar to the cards shown below for 2 different time periods. I am no graphic designer but these cards tell the story more effectively in my opinion. See the bottom of the post for the even prettier version.

Create the SVG Measure

Before we can create a card with SVG we need to write the measure that contains the SVG. So here is where we cheat and refer to post I wrote years ago, called SVG in Power BI Part 2 – KPI Shapes. This uses a measure called Growth and draws either a Green arrow pointing up or a red arrow pointing down. The final code from this blog is

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


Arrow = 
// Variables to store arrow paths
var uparrow = ""
var downarrow = ""

// Select arrow to use
var arrow = IF([Growth]>=0,uparrow,downarrow)

// Insert arrow into SVG
var svg ="data:image/svg+xml;utf8," &
 ""
 & arrow & 
 ""

RETURN svg

That would create the arrow using the Growth measure, I also want to add text below the arrow. So we increase the viewBox to be 130 tall so line 11 becomes

1

""

To add in the text I make use of another previously written post SVG in Power BI – Part 4 – Adding SVG Text. I create a text tag that is middle anchored to 50,130 and shows the Growth measure formatted as a percentage.

1
2


var growthtext = "" 
 & FORMAT([Growth],"0.00%") & ""

So the final measure looks like this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Arrow = 
// Variables to store arrow paths
var uparrow = ""
var downarrow = ""

// Select arrow to use
var arrow = IF([Growth]>=0,uparrow,downarrow)

// Add text
var growthtext = "" & FORMAT([Growth],"0.00%") & ""
// Insert arrow into SVG
var svg ="data:image/svg+xml;utf8," &
 ""
 & arrow & growthtext &
 ""

RETURN svg

Measure Data Category

As with any SVG measures you need to remember to change the Data category of the measure to Image URL on the measures toolbar or in the Model view properties pane.

Add SVG to the Card

The final part is to add the SVG measure to the card. With the card selected head to the formatting settings. Turn on the Image and then change the Image type to Image URL. The click on the fx button under Image URL to select the SVG measure we created, Arrow.

The image is there, even though you can’t see it due to it being too small. Change the size of the image to fit your card. You can change the position, alignment and spacing to fit your card. Do make sure you remember that numbers in a report will change so allow for that.

Accessibility

I know accessibility in published Power BI reports is not great, but it is made even worse by not making use of the Alt text options. So I created a measure Arrow Alt Text that will describe the image. I then click the fx button under Accessibility and use the measure.

1
2
3
4


Arrow Alt Text = 
var ArrowText = IF( [Growth] >= 0 , "Green arrow shape pointing up ", "Red arrow shape pointing down " )
var Result = ArrowText & "with " & FORMAT([Growth],"0.00%") & " below."
Return Result

Even More Tricks

Guy in a Cube have a video that does a few more tricks to make it even prettier. Its a high speed video so expect to stop, rewind and understand all the tricks Adam uses. His shadow suggestions are great. Adding his tricks and my svg I changed the card to look like this.

References

Microsoft New Card Description

Conclusion

The new cards are very clever and in the right hands will enhance reports. Adding SVG is a great addition and I know clients who love adding their own graphics. The complexity of understanding which formatting features are available depending on if you pick one series or All is frustrating but that perhaps is a separate post all together.

Power BI – Calculation Groups aren’t scary

Tue, 15 Jul 2025 16:09:30 +0000

Let’s start by being honest, calculation groups scared me. I had read the brilliant articles, seen the demos and walked away going “hell no!”. Then in a training session a client asks for a visual that really really should be done with a calculation group, so I had to sit down and get my head around them. This post is not to replace any of the superb articles that cover the details, those are referenced below. This post is my notes to remind future me how they work and that they aren’t scary, honest!

YouTube Version

Scenario

The visual requested was similar to this wating to show multiple measures with time intelligence applied. So the example I’ve used is a UK Weather report based of met office data publicly available. The row labels, e.g. Avg Rainfall, are all measures that have been formatted to match the data, e.g. 0 mm for Avg Rainfall. This could probably be done using a complex pile of measures but the technical debt that creates goes against even the minimalist best practice policies.

The model includes a calendar that includes a decade offset column. So if we were going to calculate the Avg Rainfall for this decade and last decade the DAX would be

1
2
3


This Decade Avg Rainfall = CALCULATE( [Avg Rainfall] , 'Calendar'[Decade Offset] = 0 )

Last Decade Avg Rainfall = CALCULATE( [Avg Rainfall] , 'Calendar'[Decade Offset] = -1 )

Create a Calculation Group

With measures created and the calendar table prepared we are now ready to create the calculation group. In the report go to the Model View. Now on the Home ribbon there will be Calculation group button. Click this to start the process, the first step will probably be a warning that this will turn on the “Discourage Implicit measures” option for this model. If you are following best practices this should not be an issue.

Then on the model diagram and in the model pane on the right hand side you will see a Calculation Group has been added, with one Calculation item. Double clicking on the name of the Calculation group in the model pane will allow you to rename the calculation group, e.g, Time Intelligence. This would be good practice.

Add Calculation Items

Calculation items are the different calculations you want to include in the group. For this example I want This Decade, Last Decade and % Change. Right click on the calculation group and from the menu select New calculation item. Then a formula appears similar to when you create a column or measure. Once you add in your formula and press enter it will appear under Calculation items.

I renamed the original calculation item to All Time by double clicking on it.

1
2
3
4
5


This Decade = 
CALCULATE( 
 SELECTEDMEASURE() , 
 'Calendar'[Decade Offset] = 0 
)

Add Calculation Group to a Matrix

Now we have created a calculation group we can start to create the matrix. Add a matrix to the report and add the measures. The measure names will appear as column headings. You can switch these to row headings in the visual formatting by looking under Values and then Options and turning on Switch values to rows.

After you have added the measures and switched the values to rows you can add the calculation group to the Columns area in the visual. All the calculation items appear as columns.

Formatting the calculations items

The numbers shown in the previous diagram are formatted the same as the measure formatting, e.g. Max Temp is 0.0 ºC format. This is correct for all the measures except the % Change, which should be formatted 0.00%.

Select the calculation item that needs different formatting. In the properties panel turn on Dynamic format string. Then in the Format string box enter in the format as a string, e.g. “0.00%”. You can edit in the formula bar which is slightly easier.

Sorting and Selecting Columns

The Calculation items are sorted alphabetically by default. If you change the order of one calculation item it will fix the order of all the items. So I move the % Change to the end

Also by default all columns are shown. The columns can be selected using the filters panel. In the filters panel, expand Calculation group column to show all the calculation item names. Un-select the columns you don’t want.

Resources

As with anything DAX related my first port of call is SQLBI and of course Microsoft and recent announcements.

Conclusion on Calculation Groups

Calculation groups are a very powerful addition to creating Power BI models that are not bloated with 100s of measures. I really hope they will be expanded to allow for dynamic names and some tricks to do conditional formatting.

Paginated Reports – Power Query as a Source

Tue, 08 Jul 2025 16:01:05 +0000

In March 2024 Microsoft added Power Query as a source for Paginated Reports. This means a paginated report can connect to any data source that Power Query can connect to. This post walks through the process with a few gotchas to be aware of.

Paginated Reports have had some Love

This post is part of the series covering my session titled Paginated Reports have had some Love. The resources for this series can be found at https://github.com/HatFullOfData/Sessions/tree/main/Paginated-Reports-have-had-some-love

Power Query as a Source
Create a report online
Dynamic Subscribing

Get Data

As with most reporting the first task is connect to the data using Power Query as a source. Inside a report in Report Builder, click on the data ribbon. Click on Get data to launch the Choose data source dialog. From here you can search and find lots of different data sources. For this post we are going to use an Excel workbook, so click on that.

When the connection settings dialog opens enter the File path or URL for the Excel file. There is a button to browse OneDrive, bad idea in my opinion but that is a different post! Please note you need to select the correct Authentication type. I’ve used Anonymous as my file is public on GitHub, you will need Organization Account for OneDrive or SharePoint.

Then click Next.

The next dialog, Choose Data, asks you to select the tables or sheets. Once you have ticked at least one you can then click Transform data to move to the Power Query window.

Power Query Editor

When the Power Query window opens you can add steps as per normal such as choosing columns etc. After you add all the transformations you need, click Create to add it to the report.

Power Query as a source in Explorer

When the data loads into the report it comes in 2 parts. Under data sources is Power Query and in datasets are the individual queries. Unlike other data sources such as a database each dataset will be a brand new query.

Editing the Query

When you need to edit the query, right click on the query and select Edit Mashup. It will always request you configure the connection. No Idea why but clicking the Configure Connection fixes the issue.

A few Gotchas

Multiple Queries

Each query can only return a single table. So when you have multiple queries, which one does it return? It returns the last one in the list of queries. It has nothing to do with marking a query for loading or not loading.

Refresh or not?

When you publish the report there is no semantic model. There is just a report. So every time the report is opened the Power Query executes. So lets keep those queries tidy and efficient.

Parameters

Parameters make Paginated Reports reusable and a powerful tool for the analyst. Power Query parameters though do not connect to the parameters in Report Builder. The Dataset properties show the M code, kind of. Its not editable so we can’t embed the Report Builder parameters. The only way to perform a filter is using the Filters in dataset properties which has the disadvantage that it filters after pulling all the data in.

Power Query as a Source Conclusion

I love Power Query, second to SQL, every data handler in the Microsoft stack should know it. But I don’t believe it brings any benefit putting it in report builder when I can do more from a semantic model. The gotchas aren’t a real problem on their own but they add up. So my recommendation would be to build the semantic model in Power BI desktop or in Microsoft Fabric and connect to that. That is a separate blog post, link up above.

Power BI – Create a calendar using TMDL

Wed, 11 Jun 2025 15:18:18 +0000

TMDL is short for Tabular Model Definition Language in Power BI desktop. It allows us to script definitions and changes to a Power BI model. In this post I use TMDL to quickly create a calendar for a report that includes column options such as sort by, formatting and hidden columns.

Calendar Table

Nearly every report I create includes a calendar table. Every Power BI introduction course I run I send delegates to SQLBI site to grab the code for a simple calendar. Using a few example exercises I explain column sorting for the months, hiding the columns used for sorting and formatting the date columns. Tasks we all do every new report.

Just to be clear the code I usually use comes from one of these links, in this post I’ve used the second one.

Turning on TMDL View

At the point of publishing this post (Jun 2025) TMDL view is still in preview so it need be turned on. From the File tab select, Options and settings and then Options to open the dialog. Then under Preview features put a tick next to TMDL view. This will add a new button on the top left views menu.

TMDL View

This view allows you to define tables, columns, relationships and measures. You can drag a Power BI item into the view from the data pane to see the definition. Bu this post is not to teach TMDL but to show one use of it. More information can be found at https://learn.microsoft.com/en-us/power-bi/transform-model/desktop-tmdl-view

Adding the Calendar

Another use is to use code to create a calendar table. I created the original code by creating calendar in a report and making all the adjustments I would usually do and then dragged the calendar table onto the TMDL window. So now the code includes the adjustments mentioned at the start of this post. Then I go to a report without a calendar and paste in the code below in the TMDL view. Then click Apply. The table will appear in the Data pane but possibly with red exclamation mark. There also might be a message stating a calculated table needs refreshing. If that appears click the Refresh now button that appears.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88


createOrReplace

	table Calendar
		lineageTag: a8e10f84-8296-4cf3-bda3-412811e34790

		column Date
			formatString: General Date
			lineageTag: b0b99eff-688f-4526-81d2-8afc25541413
			summarizeBy: none
			isNameInferred
			sourceColumn: [Date]

			annotation SummarizationSetBy = Automatic

		column Year
			formatString: 0
			lineageTag: 4ad002b0-e801-4457-865d-8766808bacb9
			summarizeBy: none
			isNameInferred
			sourceColumn: [Year]

			annotation SummarizationSetBy = User

		column 'Month Number'
			isHidden
			formatString: 0
			lineageTag: e14ae615-2ba5-4e4a-ad44-ff0fcb4fb3ed
			summarizeBy: none
			isNameInferred
			sourceColumn: [Month Number]

			annotation SummarizationSetBy = User

		column Month
			lineageTag: e9c8caaf-f192-4478-86bd-d811116a5ac1
			summarizeBy: none
			isNameInferred
			sourceColumn: [Month]
			sortByColumn: 'Month Number'

			annotation SummarizationSetBy = Automatic

		column 'Year Month Number'
			isHidden
			formatString: 0
			lineageTag: 9c81640a-3399-4b97-87dd-884390419cb4
			summarizeBy: none
			isNameInferred
			sourceColumn: [Year Month Number]

			annotation SummarizationSetBy = User

		column 'Year Month'
			lineageTag: a2d59320-d499-43cf-a365-44c1f33b8ed4
			summarizeBy: none
			isNameInferred
			sourceColumn: [Year Month]
			sortByColumn: 'Year Month Number'

			annotation SummarizationSetBy = Automatic

		partition Calendar = calculated
			mode: import
			source =
					
					VAR MinYear = 2025
					VAR MaxYear = 2025
					VAR BaseCalendar =
					 CALENDAR ( DATE ( MinYear, 1, 1 ), DATE ( MaxYear, 12, 31 ) )
					RETURN
					 GENERATE (
					 BaseCalendar,
					 VAR BaseDate = [Date]
					 VAR YearDate = YEAR ( BaseDate )
					 VAR MonthNumber = MONTH ( BaseDate )
					 VAR MonthName = FORMAT ( BaseDate, "mmmm" )
					 VAR YearMonthName = FORMAT ( BaseDate, "mmm yy" )
					 VAR YearMonthNumber = YearDate * 12 + MonthNumber - 1
					 RETURN ROW (
					 "Year", YearDate,
					 "Month Number", MonthNumber,
					 "Month", MonthName,
					 "Year Month Number", YearMonthNumber,
					 "Year Month", YearMonthName
					 )
					 )

		annotation PBI_Id = 0bd3b6910f2844c8b21afe01ffb36109

The report now has a calendar table matching your original calendar.

Conclusion

Being able to use TMDL will allow me to build a library of code snippets to quickly build the report elements I regularly build. This is an exciting extra tool for the toolbox. I have already started a github library of snippets I think will be useful. Not ready to make that public yet but I will.

Power BI Series

Microsoft Fabric – Create a Data Pipeline with Parameters

Thu, 05 Jun 2025 10:26:00 +0000

One of the skills of creating good solutions on any platform is building reusable modules. For a part to be reusable we reduce the hard coding and pass in parameters. This is true for data pipelines. So this post post covers creating a data pipeline with parameters.

Initial Scenario

In order to demonstrate how to create a data pipeline with parameters we need a simple data pipeline as a demo. For this post we are going to use the Star Wars api called SWAPI. The details of the api can be found at https://swapi.info/. Its a very simple api that allows anonymous connections and returns data in a JSON format.

In our pipeline we add a copy data activity. For the Source, the connection is a http connection to https://swapi.info/api/ and the relative URL is the table name, in this case films. And the file format is JSON.

For the destination the connection is to a Lakehouse. The Root folder is files with the file path being the table name + .json so in this case films.json. To match the source the file format is JSON.

The issue with the above is the “films” is hard coded. If this pipeline is going to be reusable for other tables from this api we need to move the “films” into a parameter.

Adding Parameters to the Data Pipeline

In the data pipeline click on the grey background around the activities to make sure no activity is selected. In the bottom panel you should now see options including Parameters. Click on the + New to add a parameter. Then add a name, select a type and give a default value.

Multiple parameters can be added. When the data pipeline is run the parameters can be updated. You now have a data pipeline with parameters you just need to use them in activities.

Using Parameters in Activities

We now need to edit the copy data activity to use the TableName parameter rather than the hard coded films. Edit the copy data action and on the source tab clear the Relative URL value. Click on the Add dynamic content or Alt+Shift+D. Then in the panel, under Parameters click the parameter, in this case TableName. This puts the formula in the expression builder. Then click OK to put the expression back into the Relative URL.

The above example was using the parameter straight. For the file name in the destination we need to concatenate “.json” onto the table name. In the expression builder Functions such as concat can be found in the Functions tab and in the right category group.

If you know the function name that you want to use there is intellisense and typing in “@conc” will give you a list of matching functions to use.

Clicking OK will add the expression to the filename box.

Testing the Data Pipeline with Parameters

When the pipeline is run manually a panel appears prompting for the parameter values. Enter in the values and click OK. When the pipeline completes the run successfully we can see in the lakehouse the new file has been created.

Conclusion

I first learnt best programming techniques over 30 years ago. It included concepts I still use and teach today. This is one of them. Of course the pipeline could be more complex, include more parameters and more activities but the concept is still the same, make things reusable by creating data pipeline with parameters.

First refresh it broke!

Mon, 12 May 2025 15:57:47 +0000

I set up my dataflow and it was working great. Then it broke on the refresh. Here is how I fixed it.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

Previously

In the previous post we ingested a SharePoint folder into a table in a Lakehouse. That is a well known process and nothing new except for one thing. I split the process into two stages, firstly the FullFileList which contained the list of files and the binary content of the files and then a separate query to combine data from those files. Power Query created Helper queries to perform the combining. Then it broke.

It Broke

It all appears to work fine. Then I try a refresh and it fails. On going back to look at the dataflow I see that there are 2 extra steps in FullFileList that have been added by Microsoft Fabric.

1
2
3
4
5
6
7


let
 Source = SharePoint.Files("https://lgb123.sharepoint.com/sites/FabricDemo/", [ApiVersion = 15]),
 #"Filtered rows" = Table.SelectRows(Source, each [Folder Path] = "https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/"),
 #"From Value" = Table.FromValue(#"Filtered rows"),
 #"Remove Columns" = Table.RemoveColumns(#"From Value", Table.ColumnsOfType(#"From Value", {type table, type record, type list, type nullable binary, type binary, type function}))
in
 #"Remove Columns"

The last step Remove Columns, removes all columns of type table, record, list, binary or function. So this removes the Content column that is binary and therefore the queries doing the combine of the Content column all fail. Hence it broke.

It Broke in the Staging

To improve performance and reliability, Dataflow Gen2 uses staging items to store intermediate data during data transformation

The above quote is from https://learn.microsoft.com/en-us/fabric/data-factory/data-in-staging-artifacts It stores the intermediate queries, so FullFileList would have been stored as a table in the staginglakehouse or stagingwarehouse. So Fabric is adding the step to remove the binary column so staging can work.

Turn Off Staging

When I right click on the FullFileList query in the query list the menu includes Enable Staging, by default it is turned on, indicated with a tick. So I click on Enable staging to remove the tick and turn it off. I publish the dataflow and wait. Yes! the refresh works, I test it multiple times.

Conclusion

I am curious as to how turning off staging will impact the dataflow. So I will be keeping an eye on this one. But this appears to be the fix we need to get it working again for now. I will do my enquiries and retests to see if this is still needed.

SharePoint Folder into Microsoft Fabric

Fri, 09 May 2025 17:08:15 +0000

SharePoint folder has been a connector into Power Query for Excel, Power BI Desktop and dataflows for quite a while. This post is to cover the basics before I write another post to explain a more complex scenario. This post will use GEN2 Dataflows in Microsoft Fabric. Power Query can take files on a SharePoint site and from a selected sheet, table or named range append the contents to one big table. This post explains what you should be aware of when doing this.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

SharePoint Library

I have a SharePoint library on a site called Budgets. It contains 3 files which are all the same format, 2 tabs, 1 table and 2 named regions.

The URL to the library is https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/Forms/AllItems.aspx . From this I can work out the URL to the SharePoint site is https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/ You can always test it by making sure the site url takes you to the front page.

Connecting to SharePoint Folder

In a GEN2 Dataflow, from the Home ribbon select Get Data and then more. If SharePoint folder isn’t showing in the dialog that appears, search for SharePoint. Then select SharePoint folder. Then enter in your site URL. Please note the full library url will not work.

If this is the first time you’ve connected to this SharePoint site you will need to set up the connection. All you need to do is change the Authentication kind to Organizational account and make sure you are signed in. Click Next to move onto the Preview folder data. If you have more than one library or folders it will show too many files, don’t worry we will fix this.

Click Create, rather than Combine so we can control which files get combined.

Filtering to the right SharePoint Folder

In my example I have multiple SharePoint libraries in the same site, so the initial list returned contains too many files. There is a column called Folder Path. I find a row that contains the value I want, right click on the value. Then select Text filters – Equals. This will add a filter a step to the query to filter to only files in that path.

Even if your current site only has one document library so currently this step is not required, I would add it anyway. Then when anyone adds files a new library it will not break your query.

Combining the Files

In order to make future additions to these queries easier and to reduce repeated work happening I would rename the query so far to FullFileList or similar and the right click on the query and select Reference. Rename the new query to something suitable, in my case it was Budget. The diagram will now show the two queries connected.

The next stage is to do the combining. Make sure the Content column has a data type of Binary. Then click the button in the top corner of the Content column to start the file combination. After a few seconds the Combine Files will appear requesting you to select which sheet, table or named range you are combining.

Once you click Combine it will create the Helper queries and related queries and will combine the files. You then will have a query with the first column Source.Name and the rest the data from the selected item.

Adding a Destination

There might be other transformation steps you need to add before your data is ready. I’ve added an unpivot to my data, but that would a complete post on its own. Finally we get to add a destination to save the data to. If you are going to a Lakehouse, Warehouse or Database you should be aware of option available to you.

The first option is if you are saving to an existing table or creating a new table.

When you click next the second option is available. Turning off Use automatic settings will give you the options. Is the update an append or a replace? Is the schema dynamic or fixed, ie if your query adds a new column what happens? I’ve picked replace, the default option but as the SharePoint folder grows I might look to more clever file filters so that only new data is selected and therefore can be appended.

Conclusion

Uploading multiple Excel files will always be a requirement. So its worth knowing what Power Query can do for us in a dataflow to do that. There are probably more capacity efficient methods which need exploring.

VBA – Using Dictionaries to improve searching

Sat, 08 Mar 2025 20:41:16 +0000

In VBA programming, dictionaries are invaluable tools that allow you to store key-value pairs efficiently. Checking a list of values to see if it contains a value is a regular requirement in many projects. The obvious solution in VBA is the values in an array and loop through till you find or don’t find the value. On a recent project, I was reminded that another solution is to use dictionaries, which have the exist method that quickly returns a true or false.

VBA Project References

Dictionaries are part of the Microsoft Scripting Runtime library which is not loaded by default. Within VBA in your Office app, in the Tools menu select References. From the list of available references tick Microsoft Scripting Runtime.

Once the library is loaded you can now declare a variable as a dictionary.

1

Dim dictNames as Scripting.Dictionary

Initialising the Dictionary

Dictionaries need initialising before you can use them. This can be done in the declaration of the variable or as a separate statement. I personally prefer the separate statement, means I get to chose when the variable is created.

1
2
3
4
5
6


' === Part of the declaration
Dim dictNames As New Scripting.Dictionary
 
' === Separate statement to initialise
Dim dictNames As Scripting.Dictionary
Set dictNames = New Scripting.Dictionary

Adding Items to Dictionaries

Dictionary entries have 2 parts, key and item. The key is used when checking to see if a dictionary entry exists and the item value is returned by using the key. So if all you are using the dictionary for is to have a searchable list the item value could be the same as the key or a constant.

1
2


' === Add a dictionary entry
dictNames.Add Key:="Adam", Item:="0"

Checking for an item in a dictionary

Once you have values in the dictionary you can check if a key value exists in the dictionary. The Exists method returns a Boolean based on if it can find the key value.

1
2
3
4
5
6


' === Check for a key in the dictionary
If dictNames.Exists(Key:="Adam") Then
 MsgBox "Adam is in the list"
Else
 MsgBox "Adam is not in the list"
End If

Compare a dictionaries to an arrays

The data analyst in me wanted to check how each method compared. I expected the arrays method of looping through till you found or didn’t find would be slower. Both methods are fast so by doing 10,000 searches on a list 4,000 long with 25% of not found values the dictionary method won, but if the values were all found it was very close. Speed is not the only concern though when coding.

The winner for me was the dictionary code was simpler, which reduces the technical debt of VBA code often written and maintained by business developers. The code below is to check if the value in strName is in an array or in a dictionary.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


' === Array Method
For i = LBound(arrNames) To UBound(arrNames)
 If arrNames(i) = strName Then
 blnFound = True
 Exit For
 End If
Next i

' === Dictionary Method
blnFound = dictNames.Exists(strName)

Conclusion

VBA is not leaving soon, however much every IT department would like it to vanish. Just because most VBA programmers are not professional coders, it doesn’t mean VBA can’t be written well. Adding references to helpful libraries is good practice and will help in the long term. If there is concern for future coders understanding dictionaries add links in the comments of your code to the references below.

References

Microsoft’s documentation is limited but can be found here https://learn.microsoft.com/en-us/office/vba/language/reference/user-interface-help/dictionary-object

Paul Kelly has written a better blog with examples – https://excelmacromastery.com/vba-dictionary/

VBA Posts

VBA to Edit a Power Query Parameter Value

Power Query – Handling Null Values Properly

Mon, 13 Jan 2025 10:53:43 +0000

This post is about handling null values in Power Query calculations. If you do a calculation in Power Query that involves a null value the answer returned is null. In the example below we get nulls for the first three rows because either Credit or Debit values are null. On the fourth row we get the answer 460 because Credit and Debit contain values.

1

= Table.AddColumn(#"Changed Type", "Total", each [Credit] - [Debit] )

This post gives you 2 methods on how handle them. I personally prefer method 2, but each to their own.

Method 1 – Replace Null Values

The most common method recommended is to replace null values with a value that works, so in the above example a 0. So before I add the above calculation I highlight both the Credit and Debit columns. Then I click replace values on the Transform ribbon.

The above works and is simple to explain and document. It does have the side effect that the average Credit and average Debit are now altered and filters to find just credits or debits will have to be different. It also takes 2 steps. The averages issue is the one that most people dislike.

Method 2 – Using the Coalesce operator with Null Values

Coalesce functions, DAX has one, return the first non-null value. In Power Query there is a coalesce operator ??. So Credit ?? 0 will return the credit value unless it is null and then will return 0.

So we can change our calculation to put ?? 0 after the column values and make it not return nulls. The Credit and Debit columns still have their original values for averages etc. It also is done within one step.

1

= Table.AddColumn(#"Changed Type", "Total", each [Credit] ?? 0 - [Debit] ?? 0 )

Conclusion

Power Query M is one of my favourite languages but its weird. So these posts are for purely selfish reasons to remind me how to do things. ?? is a really simple operator that solves so many problems and Microsoft give it 2 lines! https://learn.microsoft.com/en-us/powerquery-m/m-spec-operators. It could of course be chained to Column1 ?? Column2 ?? Column3 will give the the first non-null value from the three columns, and return null if they are all null.

Power Query – Fixing Missing Columns Dynamically

Wed, 08 Jan 2025 11:49:26 +0000

One of the challenges of making sure your query works in Power Query is data sources that have a changing schema. Some APIs miss out fields if the data is null and sometimes we need one report able to handle data files that are almost the same but not quite. In this post we will look at fixing 2 scenarios, a single missing column and multiple missing columns.

You know you have this problem when you get the an error similar to this:

Single Missing Column

This scenario is fixing one column missing. In my example the query was written based on a file that had a Debit column. The query is now pointed to file that is missing that column. We need the query to work if the Debit column is in the original or not.

Start in the broken version and find the first step where there is not an error. The fix is to add the Debit column if the table does not have the column.

M is not friendly to write so lets get the interface to write part of it. Click on Custom Column on the Add Column ribbon. Click Insert when prompted about inserting a step. Then enter in the name of the missing column for the column name and null for the value. When you click OK the column gets inserted and your query goes back to working.

But the query will fail when the file does include the Debit column. So we need to add some logic to the new step we added. We need to test to see if the table already has the column, if it does just return the table, if not add the column. This can be done using Table.HasColumns function and an “if” statement.

The line of M created by the inserting a custom column is

1

= Table.AddColumn(#"Promoted Headers", "Debit", each null)

This needs to become

1
2
3


= if Table.HasColumns(#"Promoted Headers","Debit") 
 then #"Promoted Headers" 
 else Table.AddColumn(#"Promoted Headers", "Debit", each null)

Please note the #”Promoted Headers” comes from the name of the previous step.

Reference for HasColumns can be found at https://learn.microsoft.com/en-us/powerquery-m/table-hascolumns

Multiple Missing Columns

The previous method works great for a single column and maybe even 2 but if you have more columns than that you really need another method. In this example we have a set of data with columns for months Jan – Dec. At the start of the year there is only a Jan column but we need the query to return all the columns.

The error will only complain about the first column it finds missing but if we fixed Feb it would then complain that Mar was missing etc. When you append 2 tables it creates a table with all the columns of both tables even if one table does not contain any data. So if we can create an empty table with all the columns we want and then append the tables we will get all the columns added.

Create a new blank query and enter in the formula to create a list of all the column names you want to make sure exist. Use {} brackets containing the column name strings separated by commas. This will display the list of values. Rename the query to something meaningful, for example Months.

1

= {"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"}

Now we need to add a step that converts this into a table. Right click on the Source step and select Insert Step After. Next we use #table function that takes 2 parameters, list of columns and values. We want an empty table so will use {} for the values. Change the formula to the following

1

= #table( Source , {} )

Reference for #table can be found at https://learn.microsoft.com/en-us/powerquery-m/sharptable

Return to the broken query and find the last step that the query works. On the Home ribbon select Append Queries and select Insert when prompted about inserting a step. In the Append dialog select the query that has the empty table just created. Then click OK. Your query now has all the columns required and should work if the columns are missing columns or not

Advanced Notes on Missing Columns

When appending 2 tables the first table columns will come first and in the order of the first table. So if you wish to order the columns of the query as well make sure your blank table query contains all the columns and the swap the order of the tables named in the Table.Combine step.

The list of column names and creating the blank table could be put as steps within the original query. That requires being comfortable with the Advanced editor and writing M. For those interested here is an example query.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


let
 Source = Excel.Workbook(Web.Contents(ExcelPath), null, true),
 Sheet2_Sheet = Source{[Item="Sheet2",Kind="Sheet"]}[Data],
 #"Promoted Headers" = Table.PromoteHeaders(Sheet2_Sheet, [PromoteAllScalars=true]),
 AllColumns = {"Project","Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"},
 BlankTable = #table( AllColumns , {} ),
 #"Appended Query" = Table.Combine({BlankTable, #"Promoted Headers"}),
 #"Changed Type" = Table.TransformColumnTypes(#"Appended Query",{{"Project", type text}, {"Jan", Int64.Type}, {"Feb", Int64.Type}, {"Mar", Int64.Type}, {"Apr", Int64.Type}, {"May", Int64.Type}, {"Jun", Int64.Type}, {"Jul", Int64.Type}, {"Aug", Int64.Type}, {"Sep", Int64.Type}, {"Oct", Int64.Type}, {"Nov", Int64.Type}, {"Dec", Int64.Type}})
in
 #"Changed Type"

Conclusion

Queries should be dynamic when required and missing columns is a common issue. Writing M is not friendly, writing dynamic M takes practice but it is worth it. Power Query that copes with your unclean data is the reality of handling data. Bast of luck I hope this helped.

Power Query – Creating New Parameters

Mon, 06 Jan 2025 11:52:37 +0000

When fetching data into Power Query you need to use values to point you to the write the data, for example name of a database server or path to a csv file. Good practice is to use parameters for these values so your query is reusable etc. And its always our intention to go and create the new parameter later.

Missing New Parameter Option

So lets take the simplest option of connecting to a csv file on the web. I have my path so I click get data and select Web. For those that want to play along here is the path to the file I’m using.

1

https://raw.githubusercontent.com/HatFullOfData/Demo/refs/heads/main/Products.csv

The dialog that appears has a box for the URL but is missing the drop down on the left that will allow me to select New Parameter. If I had already created a parameter it gives me the option but not if I have no parameters. Bizarre functionality.

Always Allow Parameters

The fix for this is to allow parameterization in data source dialogs. This can be done in the Power BI desktop options or on Power Query View ribbon for Power BI and Excel.

Once this is ticked when you now go to Get Data we get a drop down next to the text box. Now we can select New Parameter and enter in the details of the new parameter. When we click OK it returns to the first dialog with the new parameter selected.

As a side note I recommend that you always give parameters a Type rather than leaving it on Any. There are a few features such as Power BI deployment pipelines that will not work with parameters of type Any.

Conclusion

This is a really small feature, possibly doesn’t need a whole blog post. But that little drop down making the creating a new parameter that few clicks easier means I am more likely to follow my own advice and use parameters from the start.

Trigger Microsoft Fabric Data Pipeline using Power Automate

Sun, 06 Oct 2024 14:18:24 +0000

Microsoft Fabric Data Pipelines can be scheduled to happen at regular times. What happens when you want trigger Microsoft Fabric data pipelines to run as a reaction to an event? For example someone pressing a button in an app. For this we can use Power Automate to start a pipeline using a http request.

Fixed

This post has incorrect details for the connector. These have now been fixed.

YouTube Version

Workspace and Pipeline GUIDs

The flow is going to use a http action that uses a URL that includes GUIDs for both the workspace and the data pipeline. Open the data pipeline that will be triggered. Look at the URL in the browser. The Workspace GUID is the string after /groups/ and the data pipeline GUID is the string after /pipelines/.

Start Flow and Add Variables

Now we have the required GUIDS we can start writing the flow. For this example we are going to use a manually triggered flow. Then we add an actions to initialise variables WorkspaceID and PipelineID and select Type of string and put in the values from the previous step.

URL of the Trigger Microsoft Fabric Data Pipeline request

Next I recommend you use a compose step to calculate the URL for the HTTP call. Interestingly the api uses workspaces rather than groups. Make sure you have the corrects /s and no spaces between. If you have called your variables WorkspaceID and PipelineID you can just copy the code below into a Compose action.

1

https://api.fabric.microsoft.com/v1/workspaces/@{variables('WorkspaceID')}/items/@{variables('PipelineID')}/jobs/instances?jobType=Pipeline

The above url including all the flags is documented by Microsoft here – https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#run-on-demand-item-job

HTTP Request

For this action we are going to use HTTP with Microsoft Entra ID (preauthorized) connection’s Invoke an HTTP request action. Search for HTTP and scroll down till you find it.

If this is the first time you’ve used this action you will need to setup the connection to connect to this api you will need to set it up. The two urls required are below, sometimes they appear the other way around.

Base Resource URL

1

https://api.fabric.microsoft.com/

Microsoft Entra ID Resource URI

1

https://analysis.windows.net/powerbi/api

Once you have the connection sorted you can enter in the details of the request. The Method is POST and the Url of the request is the output from the compose step.

Testing Trigger Microsoft Fabric Data Pipeline Flow

The flow requires no inputs so can just be run. When the flow runs it appears to fail, but if you check the data pipeline run history you will see it was triggered just fine.

If you look in the documentation found at https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#run-on-demand-item-job. There is a note that says “There is no body returned currently”. Eventually it will return the job id of the pipeline run, so I am guessing because that is not returned Power Automate assumes it failed.

Conclusion

Data pipelines are the orchestration tool of Microsoft Fabric so being able to connect it to the Power Platform orchestration tool makes perfect sense. I look forward to the day I can mark this post as redundant and there is a Power Automate connector with an obvious action for this.

Power Automate – Save a File to OneLake Lakehouse

Thu, 03 Oct 2024 14:48:09 +0000

Currently there is no Microsoft Fabric connector in Power Automate. When we want Power Automate to save a file to OneLake Lakehouse, we need to use a workaround. This post walks through the demo I did for the Microsoft Fabric European Conference 2024. You need to be aware that this uses a premium connector!

YouTube Version

Getting the File Content

Before we can save a file anywhere we need the file content we are going to save. For this post I am going to work with the flow triggering from a new file in a SharePoint library and getting the file content action.

From the SharePoint connector triggers, select the When a file is created (properties only) and populate the site address and library name. Next add a SharePoint action, Get file content. Populate the Site Address same as the trigger. For the File Identifier, select Identifier from the dynamic content from the trigger.

Lakehouse Folder URL

Next we need to know the url of the Lakehouse folder where we want the file to go. This can be found in the Lakehouse by clicking on the three dots next to the required folder and from the menu selecting properties. This will open the properties pane on the right hand side. Click the copy button next to the URL.

To make life easy for ourselves we put this into a compose step followed by name of the file that triggered the flow. This is in the dynamic content from the trigger and is called File name with extension. Please note there is a / before the file name.

Create the file in the LakeHouse

Set up the connection

We are going to use the HTTP with Microsoft Entra ID (preauthorised) action to interact with the Lakehouse to create the file. When you add the new action search for http and then scroll down to find the right action.

If this is the first time you’ve used this action you will need to set-up the connection. Also if this is the first time you’ve connected to the Lakehouse in this way you need to setup a new connection. It asks for two URLS. Once connected you are ready to save a File to OneLake Lakehouse

1

https://onelake.dfs.fabric.microsoft.com/

1

https://storage.azure.com/

Create Blank File

The first stage is to save a file to OneLake Lakehouse that is empty. This is a simple PUT action with a parameter of ?resource=file after the path created in the compose. So select PUT for the method. In the URL of the request but the Output from the compose and add on ?resource=file. I also renamed the action.

Append Content

Finally the last action is to append the file content from SharePoint onto the file. Change the Method to PATCH. The Url of the request is the output from the compose followed by three parameters (copy from below). Expand the Advanced parameters and into the body of the request select File Content from the dynamic content.

1

?action=append&position=0&flush=true

Test Save a File to OneLake Lakehouse

Save your flow and add a file to your SharePoint library and wait a minute or 2 for the flow to trigger. A file should appear in your Lake House.

Conclusion

There are plenty of business cases of use cases to be able to store a file into OneLake. Various legacy systems email daily exports which will need to be saved and processed in the LakeHouse. When / If we get a Fabric connector I look forward to making this post as redundant!

Inherited Value in a Parent Child Pattern

Mon, 23 Sep 2024 08:03:22 +0000

In the previous post in this series we created a Parent Child hierarchy using a Dax Pattern from SQLBI. There are times it is useful to calculate an inherited value from parent item to child. The assigned to person on a User Story to be passed onto the child tasks that are not assigned to anyone, the target date of a feature to be passed down to the child user story and then onto the child tasks. The child items need to be completed before the parent target date.

Power BI and DevOps Series

This post is part of a series:

Get DevOps Data into Power BI
Add Parent Child Hierarchy using DAX Patterns
Inherited Value in a Parent Child pattern
Add conditional formatting icons the easy way

The Logic

We are going to add this as a calculated column into the work items table. If you were doing this by hand you would go through the items in the path column in reverse, looking up each target date until you found a non-blank one. In DAX we do this by creating a table of the WorkItemIDs in the path and looking up their target dates. Then we filter to only rows with target dates and take the first one.

Inherited Value DAX

Lets work through an example.

WorkItem 46 has an ItemPath of 38|39|45|46, which means it has a Path length of 4. So we start by using GENERATESERIES to create a column 1-4

1

GENERATESERIES( 1 , PATHLENGTH( WorkItems[ItemPath] ), 1 )

Value1234

We put that inside a GENERATE function and use PATHITEMREVERSE to get the work item ids from the path and then LOOKUP to use that WorkItemID to get the TargetDate of each item. GENERATE function and nested VARs is one of my favourite combinations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


VAR tabWorkItems = 
 GENERATE(
 GENERATESERIES( 1 , PATHLENGTH( WorkItems[ItemPath] ), 1 ),
 VAR tableRow = [Value]
 VAR _WorkItemID = VALUE( PATHITEMREVERSE( WorkItems[ItemPath] , tableRow ) )
 VAR _TargetDate = LOOKUPVALUE( WorkItems[TargetDate] , WorkItems[WorkItemId] , _WorkItemID )
 RETURN ROW ( 
 "WorkItemID",_WorkItemID,
 "TargetDate",_TargetDate
 )
 )

ValueWorkItemIDTargetDate1462453392024-09-094382024-11-1

Then we filter the table to rows with a TargetDate and take the first row ordered by Value. This gives us one row of data.

1
2
3
4
5
6


VAR tabFiltered =
 TOPN( 
 1,
 FILTER(tabWorkItems,[TargetDate]>BLANK()),
 [Value],ASC
 )

ValueWorkItemIDTargetDate3392024-09-09

Finally we use MINX to extract the TargetDate of that single row. The final calculated column DAX looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


Inherited Target Date = 
// Create a table of Value, WorkitemID and TargetDate
VAR tabWorkItems = 
 GENERATE(
 GENERATESERIES( 1 , PATHLENGTH( WorkItems[ItemPath] ), 1 ),
 VAR tableRow = [Value]
 VAR _WorkItemID = VALUE( PATHITEMREVERSE( WorkItems[ItemPath] , tableRow ) )
 VAR _TargetDate = LOOKUPVALUE( WorkItems[TargetDate] , WorkItems[WorkItemId] , _WorkItemID )
 RETURN ROW ( 
 "WorkItemID",_WorkItemID,
 "TargetDate",_TargetDate
 )
 )
// Take the first row of the filtered to non-blank
VAR tabFiltered =
 TOPN( 
 1,
 FILTER(tabWorkItems,[TargetDate]>BLANK()),
 [Value],ASC
 )
// Return the Target Date
VAR Result = MINX( tabFiltered , [TargetDate] )
RETURN Result

Inherited Value Conclusion

The Parent Child pattern is very useful and adding in the inherited values adds extra possibilities. I wanted to be able to identify the child items that were making the parent items late.

References

This post builds on the last post in this series and using the SQLBI Parent Child found at https://www.daxpatterns.com/parent-child-hierarchies/.

Power BI Series

DevOps Parent Child Hierarchy in Power BI

Mon, 02 Sep 2024 08:27:43 +0000

Azure DevOps is built around the pattern than items have child items and so a backlog view will show the items in a hierarchy. When the work items some into Power BI that is not shown. So this gives me the chance to use one of my favourite resources from SQLBI, DAX Patterns and their Parent-Child pattern.

I am using the Agile process so the hierarchy is Epic – Feature – User Story – Task. I have made sure my project includes all the layers correctly. You can work around oddities but for this post we will assume all layers of the items are as we expect.

Power BI and DevOps Series

This post is part of a series:

Get DevOps Data into Power BI
Add Parent Child Hierarchy using DAX Patterns
Inherited Value in a Parent Child pattern
Add conditional formatting icons the easy way

DAX Patterns – Parent Child Hierarchy

DAX patterns comes in multiple formats, a book, a website and videos. They are written by SQLBI and have saved me hours and hours. For this post we are going to adapt the Parent-Child pattern to DevOps data. I am going to use the report built in the previous post in this series. The pattern is fully explained here https://www.daxpatterns.com/parent-child-hierarchies

Adding the Path

The pattern uses a function called path. This function takes an row id and the parent row id and walks up the path until it reaches a row with no parent. Work Items in DevOps have WorkItemID and ParentWorkItemID. So following the instructions we add a column to the WorkItems table.

1

ItemPath = PATH(WorkItems[WorkItemId],WorkItems[ParentWorkItemId])

The ItemPath gives us the list of IDs in the hierarchy above the current row. It does requires the path to be complete so it might not work if you have filtered out some of the items.

Adding DevOps Hierarchy Level Columns

From the above post these are the Level 1 to Level 4 columns, we just need to swap out the right column names. We also know the the names of our levels from DevOps so we can use slightly more meaningful names, Level 1 = Epic, Level 2 = Feature etc. Add the 4 columns using the pattern below, copied straight from the SQLBI post.

1
2
3
4
5
6
7


Epic = 
VAR LevelNumber = 1
VAR LevelKey = PATHITEM ( WorkItems[ItemPath], LevelNumber, INTEGER )
VAR LevelName = LOOKUPVALUE ( WorkItems[Title], WorkItems[WorkItemId], LevelKey )
VAR Result = LevelName
RETURN
 Result

Adding a Matrix

The quickest way to see the hierarchy in action is to add a matrix visual. Into the rows we put the levels of the hierarchy to whatever levels we want. Into the values I put # Work Items. It looks great we can expand the different levels and see the 72 work items split out into features etc.

But we get blank rows at the top of each section. Not to worry of course our friends at SQLBI have a solution.

Fixing the measures

In order for this to work we need to compare the maximum path length with the level being displayed. The path length we add as a column Depth to Work Items and then add a measure MaxRowDepth. Then using ISINSCOPE function we calculate the display depth in a measure called ItemBrowseDepth.

Column Added to WorkItems table

1

Depth = PATHLENGTH( WorkItems[ItemPath] )

2 Measures Added to WorkItems table

1
2
3
4
5
6
7


MaxRowDepth = MAX( WorkItems[Depth] )

ItemBrowseDepth =
 ISINSCOPE ( WorkItems[Epic] ) 
 + ISINSCOPE ( WorkItems[Feature] )
 + ISINSCOPE ( WorkItems[User Story] )
 + ISINSCOPE ( WorkItems[Task] )

Now we can add measures that use these values to decide if to display the row or not. The logic being we only show rows where the browse depth is less than or equal to the max row depth. So I fix the # Work Items using the example from the SQLBI Pattern.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77


# Work Items =
VAR Val = COUNTROWS ( WorkItems )
VAR ShowRow = [ItemBrowseDepth] 

This now fixes the matrix to not show blank rows

![Matrix with no blank rows](image-11.png)

## Conclusion on Parent Child Hierarchy

This makes the DevOps data easy to report on. The measures roll up the data to the different levels so the total hours can be calculated etc. Handling DevOps plans with missing layers is hard work, I’ve done it where some User Stories were children of the Epic. That made the pattern clunky so I’d probably push back now and insist the DevOps plan was kept clean.

## More Power BI Posts

- [Conditional Formatting Update](https://hatfullofdata.blog/power-bi-conditional-formatting-update/)

- [Data Refresh Date](https://hatfullofdata.blog/power-bi-data-refresh-date/)

- [Using Inactive Relationships in a Measure](https://hatfullofdata.blog/power-bi-inactive-relationships-in-a-measure/)

- [DAX CrossFilter Function](https://hatfullofdata.blog/power-bi-dax-crossfilter-function/)

- [COALESCE Function to Remove Blanks](https://hatfullofdata.blog/power-bi-coalesce-function-to-remove-blanks/)

- [Personalize Visuals](https://hatfullofdata.blog/power-bi-personalize-visuals/)

- [Gradient Legends](https://hatfullofdata.blog/power-bi-gradient-legends/)

- [Endorse a Dataset as Promoted or Certified](https://hatfullofdata.blog/power-bi-endorse-a-dataset/)

- [Q&A Synonyms Update](https://hatfullofdata.blog/power-bi-qa-synonyms-update/)

- [Import Text Using Examples](https://hatfullofdata.blog/power-bi-import-text-using-examples/)

- [Paginated Report Resources](https://hatfullofdata.blog/paginated-report-resources/)

- [Refreshing Datasets Automatically with Power BI Dataflows](https://hatfullofdata.blog/refreshing-datasets-automatically-with-dataflow/)

- [Charticulator](https://hatfullofdata.blog/charticulator-simple-custom-chart/)

- [Dataverse Connector – July 2022 Update](https://hatfullofdata.blog/power-bi-dataverse-connector-july-2022-update/)

- [Dataverse Choice Columns](https://hatfullofdata.blog/power-bi-dataverse-choices-and-choice-column/)

- [Switch Dataverse Tenancy](https://hatfullofdata.blog/power-bi-switch-dataverse-tenancy/)

- [Connecting to Google Analytics](https://hatfullofdata.blog/power-bi-connecting-to-google-analytics/)

- [Take Over a Dataset](https://hatfullofdata.blog/power-bi-take-over-a-dataset/)

- [Export Data from Power BI Visuals](https://hatfullofdata.blog/export-data-from-power-bi-visuals/)

- [Embed a Paginated Report](https://hatfullofdata.blog/power-bi-embed-a-paginated-report/)

- [Using SQL on Dataverse for Power BI](https://hatfullofdata.blog/using-sql-on-dataverse-for-power-bi/)

- [Power Platform Solution and Power BI Series](https://hatfullofdata.blog/power-platform-solution-and-power-bi-part-1/)

- [Creating a Custom Smart Narrative](https://hatfullofdata.blog/power-bi-creating-a-custom-smart-narrative/)

- [Power Automate Button in a Power BI Report](https://hatfullofdata.blog/power-automate-button-in-a-power-bi-report/)

## Power BI Series

- [SVG in Power BI series](https://hatfullofdata.blog/svg-in-power-bi-part-1-svg-basics/)

- [Power BI and Project Online series](https://hatfullofdata.blog/power-bi-connecting-to-project-online/)

- [Slicers series](https://hatfullofdata.blog/power-bi-slicers-introduction/)

- [Dataflow series](https://hatfullofdata.blog/power-bi-create-a-dataflow/)

- [Power BI SVG series](https://hatfullofdata.blog/svg-in-power-bi-part-1-svg-basics/)

- [Power Automate and Power BI Rest API series](https://hatfullofdata.blog/power-automate-and-power-bi-rest-api/)

- [Power BI and DevOps series](https://hatfullofdata.blog/devops-data-into-power-bi/)

Get DevOps Data into Power BI

Tue, 27 Aug 2024 09:09:57 +0000

Azure DevOps is full of data that management want to to report on. Adding a Power BI report is an obvious way to do this. There are plenty of sample reports and Microsoft posts giving you guidance how to get DevOps data into Power BI. This post is my take on how I would build the reports. My focus is making them easy to extend and easy to point at a different organisation and project.

Power BI and DevOps Series

This post is part of a series:

Get DevOps Data into Power BI
Add Parent Child Hierarchy using DAX Patterns
Inherited Value in a Parent Child pattern
Add conditional formatting icons the easy way

YouTube Version

Microsoft References for DevOps Data into Power BI

https://learn.microsoft.com/en-us/azure/devops/report/powerbi/odataquery-connectGreat post showing you to use Visual Studio Code to write the query, test it and then use it in Power BI
https://learn.microsoft.com/en-us/azure/devops/report/powerbi/sample-odata-overviewLots of sample reports

OData Queries

We get DevOps Data into Power BI using an OData query. From the first Microsoft link above it shows the query is made up of different parts. The first part is a path and then extras that select columns, filter the data and sort the data.

The simplest form though is just the URL, that will bring through all the columns and all the rows of data. We will start with that.

Simplest Query for Work Items

1

https://analytics.dev.azure.com/{Organisation}/{Project}/_odata/v3.0-preview/WorkItems?

The path includes the organisation and the project names. So we start in Power Query, by clicking on Transform data on the Home ribbon in Power BI. Then click on Manage Parameters to open a dialog. You need to add 2 parameters, Organisation and Project. Click on New to start a new parameter, enter in a name and description (yes add that description, future you will thank you), select type text and enter in the current value for the Organisation and the Project.

We could now use Get Data and select OData and build the string… But that is hard work and not easy to read! So I’ve written a template to walk you through. Click on New Source and select blank query. Click on Advanced Editor and paste in the code below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


let
 // Path using 2 parameters
 Path = "https://analytics.dev.azure.com/" & Organisation & "/" & Project & "/_odata/v3.0-preview/WorkItems?",
 
 // URL made up of parts
 URL = Path,
 
 // Do the OData query
 Source = OData.Feed(URL, null, [Implementation="2.0",OmitValues = ODataOmitValues.Nulls,ODataVersion = 4])
in
 Source

The Path line uses the 2 parameters and points to the WorkItems. The URL line looks redundant now but will make more sense when the query has more parts. The Source line uses the URL and includes the flags recommended by the link at the top of this post to prevent throttling.

When you click OK and set up the connection if required you should get a table of data with lots of columns.

Selecting the columns

The above query gives us more columns than we want. Rather than removing the columns in Power Query we can add a select line to the query which will make the OData query faster. So we need to pick the columns we want while we can see them all. I selected and put them into a comma separated string to make up a $select string to add to the Power Query

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


let
 // Path using 2 parameters and select columns
 Path = "https://analytics.dev.azure.com/" & Organisation & "/" & Project & "/_odata/v3.0-preview/WorkItems?",
 Select = "$select=WorkItemID,Title,WorkItemType,State,TargetDate,ParentWorkItemID,OriginalEstimate,AssignedToUserSK",

 // URL made up of parts
 URL = Path & Select,
 
 // Do the OData query
 Source = OData.Feed(URL, null, [Implementation="2.0",OmitValues = ODataOmitValues.Nulls,ODataVersion = 4])
in
 Source

When you finish editing the query you will get a table of the work items from DevOps.

You could add more parts to the query for filtering and sorting if required.

User and Calendar Table

DevOps has other tables including a user list and a calendar. These can be pulled through to Power BI using the same pattern.

Users

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


let
 // Path using 2 parameters and select columns
 Path = "https://analytics.dev.azure.com/" & Organisation & "/" & Project & "/_odata/v3.0-preview/Users?",
 Select = "$select=UserSK,UserName,UserEmail",

 // URL made up of parts 
 URL = Path & Select,

 // Do the OData query
 Source = OData.Feed(URL, null, [Implementation="2.0",OmitValues = ODataOmitValues.Nulls,ODataVersion = 4])
in
 Source

Calendar

This includes a filter for the Year to be greater than or equal to, ge, 2023.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


let
 // Path using 2 parameters and select columns
 Path = "https://analytics.dev.azure.com/" & Organisation & "/" & Project & "/_odata/v3.0-preview/Dates?",
 Select = "$select=Date,MonthName,MonthOfYear,Year",
 Filter = "&$filter=Year ge 2023",

 // URL made up of parts 
 URL = Path & Select & Filter,

 // Do the OData query
 Source = OData.Feed(URL, null, [Implementation="2.0",OmitValues = ODataOmitValues.Nulls,ODataVersion = 4])
in
 Source

Some Modelling

Once the queries are written and loaded into Power BI desktop we are ready to do some modelling. I’m keeping it very light as I have another post coming that will implement the Parent Child hierarchy needed.

Relationships

The relationship model is only three tables, fact table WorkItems and the 2 dimension tables Users and Calendar. The relationships are

TableColumnRelationshipTableColumnUsersUserSK1 to ManyWorkItemsAssignedToUserSKCalendarDate1 to ManyWorkItemsTargetDate

Measures

I only created two measures. The first was for the total hours in the original estimate column. If we had included completed and remaining hours I would have created measures for them as well. The second one was for the count of items

1
2
3


Total Hours = SUM(WorkItems[OriginalEstimate])

# Work Items = COUNTROWS(WorkItems)

Create Visuals

I kept it simple. Not all the tasks had Target dates so we could not do much via dates. So I had total hours and number of tasks and Users and Item States. I created a table and a tree map visual.

Conclusion on DevOps Data into Power BI

The above model is limited, I can’t see the structure of the project, I can’t rollup the tasks into user stories etc. It does have the data in it to get there though. The next post on adding the Parent-Child hierarchy will help.

Power BI Series

Editing Azure DevOps items in Excel

Tue, 20 Aug 2024 08:30:38 +0000

One of the easiest ways to edit a project in Azure DevOps is to connect Excel to the project. This post walks you through connecting to a project and updating the DevOps items in Excel. Its mostly for selfish reasons as I need to remember how to do this.

Microsoft also has a post about this at https://learn.microsoft.com/en-us/azure/devops/boards/backlogs/office/bulk-add-modify-work-items-excel

Add Excel Extension

This needs an Excel extension added from the Visual Studio site. You need to download Azure DevOps Office Integration 2019 from here and execute

Download Visual Studio Tools – Install Free for Windows, Mac, Linux (microsoft.com)

Once you have done this, in Excel you will have a Team ribbon.

Create a Tree Query

If you want to edit many items including child and parent hierarchy, then the easiest way is to write a tree query in DevOps that loads the right tasks and then use that to pull the right tasks into Excel. In DevOps under Boards select Queries. Then select New Query. Change query type to Tree of work items and type of tree to Parent/Child. Select the right filters to pull the items you want and then save your query. Save and run the query to make sure you get a hierarchy of tasks. In my example I wanted all the items in the project.

Connect to a DevOps Project

On the Team ribbon, click on New List. If this is the first time you’ve connected you need to add a server first.

Adding a Server

Click on the Servers button. Then in the next dialog, click on Add. Into the next dialog enter in the url to the organisation, for example https://dev.azure.com/HatFullOfData/. Click OK to finish adding the server, you will be prompted to login. Click Close to return to the original dialog.

Connecting to a project

You start from the connect to Azure DevOps dialog, which will open from adding a server or opened by clicking on New List on the Team ribbon on a new sheet. Select the server and select the right project and click connect. Select your tree query from the drop down and click OK The tasks from the project should load as a table into Excel.

The table format needs some explaining. Title 1 is the name of the highest level tasks and Title 2 etc are children of the Title 1 task above it. You can add another level of hierarchy by clicking on Add Tree Level from the Team ribbon. You can also add other DevOps columns by clicking Choose Columns and using the middle buttons to add or remove columns in the Choose columns dialog.

Adding and Publishing DevOps items in Excel

Items can be added to the table just typing in new rows. The ID column leave blank, the Title 1 – 4 fill in making sure you understand the hierarchy. Some fields such as Target Date are read only for some Work Item Types. When you want to see the results in DevOps click on Publish. Then open DevOps to see the work items.

Conclusion

I needed to create a project plan in DevOps from a list of tasks I had in a text file ready for a demo. There were 80 tasks and I knew I could it them quicker in Excel than I could in DevOps because I am more familiar with Excel. I also wanted to note down how I then loaded those tasks into DevOps from Excel.

Other related series include:

DevOps with Power Automate posts

DevOps updates on behalf of another with Power Automate

Mon, 19 Aug 2024 09:34:27 +0000

I had a need to use automation to do DevOps updates on behalf of another person so when the DevOps adoption metrics were done we the stats were slightly more accurate. This can only be done using REST API and will need permissions. This post is part of the DevOps and Power Automate series.

DevOps with Power Automate posts

YouTube Version

It’s on the backlog!

Permission required to makes updates on behalf of another

The trick to being able update on behalf of another is to update the ChangedBy field to another persons email address. This is not normally allowed, it breaks the rules. So the permission we need is to bypass the rules and make illegal updates.

Permissions are managed in Project Settings, you then click on permissions. Click on Users at the top and then click on the user being used in the flow to show their permissions. Under Boards check the permission for Bypass rules on work item updates. This needs to be set to Allow.

Make Updates on behalf of another

Now we have permission is ignore the rules we can test out a flow. In the REST API documentation for updating an item https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/work-items/update you will see there is a boolean parameter bypassRules. This can be added to the relative URI using & to separate the parameters.

In the body section I have added an update to the ChangedBy field. This update updates the State to active and performs it on behalf of Dan. In the task history we can see the update and see I did it on behalf of Dan.

Word of caution

By passing the rules, it also means that the updates behind the scenes that happen are turned off. If we change the state to closed using the same technique the we get an error when we open the work item.

So make sure you have checked the updates you are doing and include all the fields you need to update, even the hidden ones.

Second word of warning is no checks are done on the updates. So you can assign a task to an email address that doesn’t have access, or update to a state that is not valid. So treat this one with caution.

Conclusion

Updates on behalf of another is a very powerful trick. But should be treated with extra caution and it has made me double check permissions on a ew projects so access to by passing the rules is limited.

Update DevOps without Notifications with Power Automate

Fri, 16 Aug 2024 07:52:49 +0000

When a work item is assigned to you, by default you are sent a notification. Notifications can also be sent regarding updates to your items. Now imagine creating over 10 thousand tasks automatically and assigning them to people. I recently did this on a projects and potentially each person was going to get 100s of emails. So I needed Power Automate to assign the tasks without notifications. This post is part of the Power Automate and DevOps series.

DevOps with Power Automate posts

YouTube Version

They are on the backlog!

Permissions for updates without notifications

Updating DevOps without notifications is not something that everyone should be able to do. Or at least the admins will want to be able to control who can do it.

In the Project settings, click on Permissions. Then click on Users to list your users. When you click on a person you get to see all their permissions. They can be Allow, Deny or Not Set for all the different actions. (inherited) means it is set by being a member of a group. If you hover your cursor over the (i) you will get a description telling you which membership the permission as inherited from. A Deny overrides an allow, so you might need to manually override the inherited permission.

The permission we are interested in is Suppress notifications for work item updates.

Updating without Notifications

Now we have the permission we can test it out in an update. This can only be done on updates done by a REST API call. If you look on https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/work-items/update there is a URI parameter suppressNotifications which is a boolean.

URI parameters get added after the ? and are separated by &. So the code to update item 36 without notifications would be

1

{project}/_apis/wit/workitems/36?suppressNotifications=true&api-version=7.0

Item History

Suppressing the notifications does prevent automated updates filling inboxes with emails that will just get deleted. It does not however supress the history of the item being updated. So who did the update and when is still recorded.

Running a Wiql DevOps query in Power Automate

Wed, 14 Aug 2024 11:55:17 +0000

Wiql stands for Work Item Query Language and is the language used to define the queries in Azure DevOps. Its has a very similar syntax to SQL. Power Automate can use a REST API send a query to DevOps to execute the query. This post post is part of the Power Automate and DevOps series.

DevOps with Power Automate posts

The full reference can be found at https://learn.microsoft.com/en-us/azure/devops/boards/queries/wiql-syntax?view=azure-devops

Writing the Query

You can write it by hand, and I’m sure some people do. I use the query tools in DevOps to build the query and then view the Wiql code. For this post the example we are going to is to assign all the items in a hierarchy to one person. For example all the items in the Design Shiny feature.

Wiql Editor

Before we can work with WIQL we need to install the extension Wiql Playground in the organisation. You will need to be an organisation admin to do this. Head to the marketplace for Visual Studio, found here

Extensions for Visual Studio family of products | Visual Studio Marketplace

Search for wiql and click on the Wiql Editor. When the extension information page opens click on Get it free. Then on the next page, select the right organisation and click Download. It will now appear under boards in your project.

Construct Query

Now in Queries construct the query. I am not here to teach you building queries in DevOps, there is a great post to get you started here https://learn.microsoft.com/en-us/azure/devops/boards/queries/using-queries. My query is a tree of work items, matching the top item first and only has 2 criteria of the top item is ID 17 and the child item is not closed or completed.

Once you have your query working and you have saved it you can view the Wiql. Click on the three dots in the top right. Then from the menu select Edit query wiql. This will open a pane containing the code that you can copy to your clipboard.

Note you have to save after any changes for the changes to be in the Wiql code.

Running the Wiql in Power Automate

Now we have the Wiql code we can write the flow. The flow will have the parameters of the Parent Work Item ID and the email address to assign all the tasks to. We will then use a compose action to the Parent Work Item ID and Project name into the Wiql.

The next step is to run the query. The documentation for the REST API call can be found here https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/wiql/query-by-wiql The method is Post and the URI will be the same for all Wiql

1

_apis/wit/wiql?api-version=7.1-preview.2

The body is JSON with only one field query with the output from the compose action.

When we run the flow and enter in the Parent ID 17 and the email Dan@lgb123.onmicrosoft.com it executes and returns us the data.

Handling the Output of the HTTP call

In the successful action we can click on Show raw output in the Outputs section to open a pane and explore the Json output. Scroll past the headers and then in the body section, past columns to find workItemRelations. This will list all the work items returned by the query, not their details, just the id and url in the target section.

To make this easy to work with we need to use a Parse JSON so copy all the output JSON and return to editing the flow. Add a Parse JSON step and add the Body returned by the HTTP to the Content. Next we need to add a schema, this can be generated. Click on Generate from sample. Then paste in the JSON you copied and click Done. And a schema will be added.

The output schema is now known to the flow. So we can use the output to loop through all the items and update the work items.

Looping through workItemRelations

We can loop through the workItemRelations using an Apply to each action. The piece of information I want from each item is the target – id. So I add a compose and add the following expression

1

item()?['target']?['id']

I would then test to make sure you get the number of items you expect and that the compose does return the ids you expect

Finally we can add a step to assign each task to the email given in the parameter. I use a Send HTTP method rather than the Update a Work Item so I can as explained in a future post turn off notifications.

We can then run the flow and check the results in DevOps and see the tasks have been assigned.

Conclusion

this was one example of using Wiql inside Power Automate. I have used it to duplicate hierarchies and delete a hierarchy. The skill comes in being able to write the Wiql to achieve what you need.

Using DevOps REST API in Power Automate

Tue, 13 Aug 2024 16:37:38 +0000

DevOps REST API is a well documented API that we can easily use in Power Automate to perform a huge range of actions. The connector includes an action to make it easy. The API documentation can be found at

https://learn.microsoft.com/en-us/rest/api/azure/devops

In this post we will take some simple examples to get you started. This is the third post in the DevOps and Power Automate series.

DevOps with Power Automate posts

YouTube Version

Its coming honest!

Get Task Details using DevOps REST API

Power Automate includes an action to get the details of a Work Item. When we look we can see it has 4 required fields. Organization name, Project Name and Work Item Id are easy. We will have them. I had problems with Work Item Type, I didn’t always have that. So I looked to DevOps REST API for an alternative.

If we look in the documentation and then search for Get Work Item, you will find the syntax for fetching the details of one work item. The URL needs three pieces of information, shown in the {} brackets, organization, project and id. So no Work item type is required.

https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/work-items/get-work-item

We start with an instant flow. Then we add the a parameter to the trigger of the work item id. Then we add the action Send an HTTP request to Azure DevOps. The Organisation we can use our environment variable (or from the drop down). The method is GET. The Relative URI is everything after the organisation in the url we got from the documentation.

1

{project}/_apis/wit/workitems/{id}?api-version=7.2-preview.3

When we run the flow, we get prompted to enter the number of a work item id. When it completes, we get returned the JSON of that work item details. If you need to extract information from that JSON, you will probably need a Parse JSON action. (Another post!)

Update a Work Item using DevOps REST API

We can update an item using the REST API. For this example the flow will update a given task to have the state Active and be assigned to the given email. So, we add Work Item ID and Email as two parameters to the trigger.

Then we add a Send an HTTP request to Azure DevOps action. The Organisation is the same as before, the method is now Patch as we are updating data. The Relative URI is exactly the same as before. In the headers we need to add Content-Type and application/json-patch+json

In the Body we need to add a JSON object that includes the fields we want to update. For this we need to know the field names, see the previous post in the series for ways to find them. It is just text so we can insert dynamic content, e.g. the email address. The above example picture and code below shows changing the State to Active and assigning it to the Email address

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[
 {
 "op": "add",
 "path": "/fields/System.State",
 "value": "Active"
 },
{
 "op": "add",
 "path": "/fields/System.AssignedTo",
 "value": "@{triggerBody()['text_1']}"
 }
]

Conclusion

Rest API is a very powerful way to work with Azure DevOps and the action in Power Automate does make it very easy to use. It is worth having a test project to try out some of the options. Other posts will look at using Rest API to make changes without notifications and on behalf of another person. Also a bulk delete process, which needs a post of its own due to a few oddities.

Power Automate Update Fields in Azure DevOps

Mon, 29 Jul 2024 08:09:46 +0000

In a previous post we created items in DevOps. Although the Power Automate create item action had lots of fields, it did not have all the fields. This post shows you how to find the names for those fields and update fields in Azure DevOps using Power Automate.

DevOps with Power Automate posts

YouTube Version

Its coming 🙂

The Problem

If you compare the fields available in a standard Epic item in DevOps with the fields available in Power Automate create work item action you will notice some fields are missing. The obvious missing ones being Start Date and Target Date and other fields like Business Value etc and any custom fields you’ve added will not be there so we need to update fields in Azure DevOps.

The observant ones amongst you will have noticed the Other Fields option in the Power Automate action. It wants a key for your field which is the name of the field. And guess what that name is not going to be StartDate. So our first job is finding that name.

Finding Field Names v1

This is the first method and I think the simplest method. Start by creating an item in DevOps that has values in the fields you would like the names of. I’ve populated Effort, Business Value, Time Criticality, Start Date and Target Date.

Then in a Power Automate instant flow I add an action to get the work item details of that item, notice it is number 21. In the Get work item details I populate Org1, Project, Type=”Epic” and Id=21. Then I run that flow and look at the outputs from that step.

If you scroll through the fields you will eventually find Business Value etc with their full field names. You need to replace the _ with . as I assume JSON objects to dots in field names. Note down the ones you need.

Create Work Item and Update Fields in Azure DevOps

Now we have the field names we can create a work item and populate those fields. Note the date formats are YYYY-MM-DDThh:mm:ssZ.

When this flow is run, it creates an item and will successfully update fields in Azure DevOps.

The Update a work item action has exactly the same Other fields option so you can use the same technique.

Find Field Names v2

Azure DevOps can be accessed via a Rest API2 and this includes the list of fields available in a project. There is quite good documentation for this which can be found at https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/fields/list . From the first code block on that post remove the GET and populate your org and project names. Then in a browser navigate to that url and it will return lots of JSON showing the definition of all the fields.

1

https://dev.azure.com/{organization}/{project}/_apis/wit/fields

Then search for the fields you want and look for the referenceName and use that. As my project got complex I saved this data and then used Power Query to make a simple list of fields (perhaps a resource I should make public).

Conclusion on Update Fields in Azure DevOps

The Other Fields section in Create and Update work item actions gives us the option to update items however we want to. Some fields of course are read only and updating some fields will update other fields, for example if you change the state to Closed it will then populate the Closed date field.

Connecting Power Automate to DevOps

Thu, 25 Jul 2024 10:54:26 +0000

I know Azure DevOps from being on projects. Mostly for tracking tasks, sprints etc. I recently had to automate some item creation and editing across 1000s of tasks in DevOps. So I had to explore connecting Power Automate to DevOps for item creation and item editing. This is the first post I have in a long list of posts regarding DevOps with Power Automate and Power BI, welcome to my way of remembering how I did something.

DevOps with Power Automate posts

YouTube Version

DevOps Connector

While editing a flow if you add action and search for DevOps you will find the Azure DevOps connector. If you look at the actions in the classic format1 you will notice its a premium connector, i.e. you need a premium license. If you have not used this connector before it will prompt you to login. It assumes your DevOps login matches your Power Automate login.

Create an item

Let’s start with a simple action to create a work item in a project. We start by adding the action Create a work item.

The Organisation Name and the Project Name drop downs will contain the organisations and projects your login can see. (See further down about making this solution friendly). The Work Item Type drop down will contain all the types that the project allows from its process. The title is plain text and the Description is rich text, i.e. html.

When you test the flow you should get a work item created that matches the above.

Troubleshoot Connecting Power Automate to DevOps

If the flow fails with an error regarding authorisation, you need to first check you are allowed to add an item in that project in DevOps. Then you need to check the organisation settings in DevOps. Look in Policies and make sure Third-party application access via OAuth is turned on.

Advanced Options in Create an Item

The Create a work item action includes advanced options, really they are just optional options. Two of the options are Link URL and Link Type. In this example I used the URL from the previously created item and the Type of Hierarchy-reverse to make this item a child of the previous item.

Another option is the Assigned To field. I added my email so that when I look in the created item I see that it has been assigned to me.

The first option is Return all fields, which the default is blank. This means the action returns all the item details when it successfully creates the item.

Making it Solution Aware

If this flow is part of a solution that is going to move from a development environment to test and production, you will need to point the DevOps actions to the different Organisation and Project values. Thankfully the text values of the Organisation Name, e.g. HatFullOfData can be used.

So I create 2 environment variables DevOps Organisation and DevOps Project. Then in the create a work item action, in Organisation Name and Project Name drop downs I select Custom Value and then the right value from Dynamic content.

And yes of course after making these changes I test the flow again.

Conclusion on Connecting Power Automate to DevOps

For when you need to automate the creation of items, this is great. It does not replace importing an Excel file to create a whole plan but it does give an extra set of options to consider.

Power Automate – Create Power BI Workspace

Fri, 21 Jun 2024 20:18:44 +0000

We’ve setup the service principal, saved the details securely in Azure Key Vault and used Power Automate to fetch them securely and finally setup the permissions to allow that service principal to use the rest api. This post will finally create the flow to automatically create Power BI workspace making use of all the preparation.

This post is part of Power Automate and Power BI Rest API series.

Start the flow and get service principal details

There will be an instant flow using a manual trigger. Once the flow editor appears, expand the trigger and add a text input for the Workspace Name.

The first action will be to call the child flow that fetches the tenantid, clientid and secret. Select Get Client Tenant and Secret flow from the dropdown and then use PBI-Workspace as the prefix value.

HTTP Call to automatically create Power BI Workspace

FInally we can do the action to create Power BI workspace, for this we add a HTTP action. I’m going to break this down into 3 parts, the URI, the body and the authentication.

From https://learn.microsoft.com/en-us/rest/api/power-bi/groups/create-group#examples we can see the HTTP starts with POST so we pick that as the method. It also gives us the URI. To save you going to fetch it here it is

1

https://api.powerbi.com/v1.0/myorg/groups?workspaceV2=True

The next part is the body. This is a very simple piece of JSON with only one field name. Get the value of name to come from the trigger input Workspace name.

1
2
3


{
 "name": ""
}

The final part is the authentication. Show the Advanced Options to see the boxes to fill in. Firstly Authentication is Active Directory OAuth. Then the Authority can be left blank. Next the Tenant, Client ID and Secret are dynamic values from running the child flow action. Finally the Audience should be the following

1

https://analysis.windows.net/powerbi/api

Your flow is now ready to test. Hopefully when you do this it all works and you get green ticks. But when you go to Power BI the workspace is not listed in workspaces you can see. This is because the workspace was created by the service account, so the service account is the admin, not you. So the next step is to add a workspace admin.

Add Workspace Admin

This is going to require another HTTP call. So if we look at the second example show at https://learn.microsoft.com/en-us/rest/api/power-bi/groups/update-group-user#examples we can see we need 2 pieces of information, the workspace ID and the email address of the workspace admin.

The workspace ID must have been returned when we create Power BI workspace. If you look in the dynamic content it won’t be listed though so we need either code by hand or we use my favourite trick of Parse JSON.

Get the Workspace ID

Go to the test run you did and expand the HTTP call that created the workspace. In the Outputs section there is a Body section which will contain the JSON that was returned. Copy this JSON as this will help complete the Parse JSON.

Then return to editing the flow and add a Parse JSON action. For the content use the Body returned by the HTTP action. JSON schemas are tricky to write by hand so you can generate it from what we copied. Click on the Generate from sample to open a dialog. Paste the output body we copied earlier and click Done. The Parse JSON step will now have a schema.

Request the Admin Email

The flow might not be run by the requester of the workspace, so it makes sense to ask for the admin’s email. Add this as an extra input in the trigger.

New HTTP Request to add User

Add a new HTTP action, and I recommend renaming it. (Yes that should happen to the first one as well!). Again its a POST method and the URI needs id from the Parse JSON step slotted in. The Body is a simple JSON again. Code for both of these can be copied from below. The authentication is exactly the same as the previous HTTP step.

URI

1

https://api.powerbi.com/v1.0/myorg/groups//users

Body

1
2
3
4


{
 "emailAddress": "",
 "groupUserAccessRight": "Admin"
}

Conclusion to Create Power BI Workspace

You should now have a complete flow that when run asks for Workspace Name and Admin Email and creates a workspace with the admin added.

This is the MVP of the solution. There are obvious additions such as approvals, notifying the new admin and other various extras that should be included. They may get blogged, lets see.

Power BI Series

Allow the use of Power BI Rest APIs

Mon, 17 Jun 2024 18:13:23 +0000

In the first 3 posts of this series we created Service Principal, stored the details in Azure key Vault and created a flow. We have one more step to do, we need to allow the use of Power BI Rest APIs in our tenancy and then we will be ready to actually set up the automation for creating workspaces.

This post is part of Power Automate and Power BI Rest API series

Find the Setting

This setting can be found in the Admin portal for Power BI under Tenant settings, you need to be a Power BI admin. If you search for Fabric API you’ll find it quickly. If you click the enabled toggle to on it will give you 2 options. The entire organization would allow the use of Power BI Rest APIs by anyone if they had the right credentials, this would not be good practice. So I recommend selecting Specific security groups. This means the service principal needs to be a member of a security group.

Tenant settings can be found at https://app.powerbi.com/admin-portal/tenantSettings

Create Security Group

To create a security group you will need the right level of security in Entra, in many organisations you will not. Go to https://entra.microsoft.com/ and click Groups on the left hand menu. Make sure you are looking at All Groups and then click New group.

The group type should be Security. Enter in a Group name, please follow any naming conventions you should be following. I highly recommend putting in a group description, future you will thank you. Click on No members selected to open the Add members pane. Search for the name of the service principal and tick next the name. Then click Select to close the pane.

No members will now be updated to 1 member, so you can now click Create.

Back to the setting to allow the use of Power BI Rest APIs

Now we have the security group setup we can change the setting to allow the use of Power BI Rest APIs. Back into Power BI admin portal and find the setting.

Enable the setting and select Specific security group. Then add your security group into the box, the search is pretty good in my experience. The click Apply.

Finally!

We have now ticked all the boxes, done all the prep to write a flow to create a workspace. The next post will be my final post in this series! (Well for now as I already have a few ideas for taking it further.)

Power BI Series

Use Power Automate to Fetch Secrets from Key Vault

Wed, 29 May 2024 08:34:09 +0000

Once you have stored security details in Azure Key Vault as described in the previous post in this series (Create Azure Key Vault to store Tenant ID, Client ID and Secret) you need a way to fetch secrets when you need them. Power Automate connector for Azure Key Vault includes the Get Secret action. This post shows you how to build a reusable child flow to fetch secrets.

This post is part of Power Automate and Power BI Rest API series.

Flow design

The flow we are going to create will be called by other flows, so is a child flow. The flow will have one parameter of the prefix of the ClientID and Secret store in the Azure Key Vault. If you have completed the previous post this would be PBI–Workspace. It will return 3 values, TenantID, ClientID and Secret which will be secured

Because we are using child flows we need to create our flows inside a solution. I recommend creating a solution for these group of flows. I don’t want to make this blog post even longer so please go google creating Power Platform solutions.

Setting up the Trigger

For a child flow we use a instant flow. Type in a descriptive name and select the trigger Manually trigger a flow. When you are in the flow editor, expand the trigger and add an input for the prefix. Adding hint text will help you and other users remember what the input is about.

Adding Actions to Fetch Secrets

The flow is reasonably simple with three repeating actions to fetch secrets. Add an action and search for Key Vault and select it. Then from the list select Get Secret. If this is the first time you have connected to a Key Vault you will need to type in the key vault name and click Sign in. Be aware that every key vault has its own connection.

The action has a very simple drop down. For the TenantID action you can just select it from the drop down. For the ClientID and Secret we want to the prefix value from the trigger so that this flow is reusable. So for the ClientID and Secret select Enter custom value. Then use the dynamic value Prefix followed by -ClientID and -Secret.

Securing the Secret Value

The secret value should be kept secret. So the Action output needs securing. Click on the ellipse (3 dots) on the Get secret – Secret action. Turn on the toggle for Secure Outputs and then click Done at the bottom of the dialog.

This could be done on ClientID as well if required. Each action that has a security setting applied will show a padlock. When a flow run execution is looked at the value of the secret will not be available.

Returning the Values

The final part for this flow is to return the values. And for this flow to run as a child flow it must have a respond action. Click on add action and search for the Power Apps connector. It only one has one action Respond to a Power App or flow. Add three outputs to return TenantID, ClientID and Secret.

Handling the Connections

When an instant flow runs the connections are usually based on who clicked the start on the flow. This does not work for a child flow, so a connection must be selected that will be used for when it runs.

The changes need to be made from the flow details page, save and click the blue arrow in the top left to get there. Click on Edit in the Run only users box. When the pane appears, under connections used select a connection under Azure Key Vault. A message will appear to state users only get access in this flow, click OK to close it and then click Save to save the changes.

Testing the flow

When you first write a child flow I suggest you write an instant flow to test it. The Run a Child Flow action only is available if your flows are in a solution. In order to protect the secret value you also need to protect the outputs of the Run a Child flow action.

This tests that the flow runs a child flow and returns values even though we can’t see them.

Conclusion

We now have an app that can fetch secrets, we’ve secured the authentication details and now written a flow to securely retrieve those values. Next step is to make sure Power BI will allow us to do this.

Power BI Series

Create Azure Key Vault to store Tenant ID, Client ID and Secret

Sat, 06 Apr 2024 20:14:00 +0000

Azure Key Vault is a cloud service that provides a secure store for secrets. And in the previous post on creating Power BI Service Principal Profile we added a secret. The Tenant ID, Client ID and Secret values give access to the Service Principal and whatever permission they have. So for that reason we are going to create an Azure Key Vault.

This post is part of the Power Automate and Power BI Rest API series

Create Azure Key Vault

The first step is to create the Azure Key Vault. Start at https://portal.azure.com/. From the Home screen, navigate to all services and find Key Vaults. Once on the Key Vaults page click on Create to start the process. This will navigate you to a screen to select Subscription and Resource group. Key vaults do cost money, in my case

Add Role Assignment

Now we have create a new key vault to store our secrets the obvious place to go next is to Secrets. But if you take a look there you will see you do not have permission to play with secrets yet. And if you try to create one it will fail.

You do have the rights to give yourself permission. On the left hand side menu, click on Access Control (IAM). Click on +Add and from the drop down options select Add role assignment.

When the next screen appears, we need to select a role based on Secrets. In the search box enter Secret and from the list select Key Vault Secrets Officer, that can read and write secrets. Click next to move to the Members tab. Click on + Select members and select yourself from the list and anyone else that needs access. When all the members are listed, click Review + Assign twice.

You can look at the list of role assignments or click on Check my access for confirmation. If you want a user to be able to read the secrets but not write them you would pick Key Vault Reader.

Add Secrets

When we now head to Secrets on the left hand menu, there are no warnings about being unauthorized. Click on + Generate/Import to open the Create a secret form. Upload can stay as Manual, enter in a name and the value. For naming I will use a common start for the Client ID and Secret so that in the flow in the next post I only need to ask for one parameter.

I also add a secret to store the Tenant ID. This probably doesn’t need to be as secure but it makes it easy if everything is in the same location.

Conclusion

I now have a key vault with 3 values. I can now use either environment variables or a flow to pull through the values. I can add more secrets to the vault for other Power BI actions.

Create a Power BI Service Principal Profile

Sat, 17 Feb 2024 18:18:23 +0000

For Power Automate to use the Power BI Rest API it needs a service principal profile. This is created by creating an app registration that has the right permissions. This allows an app, e.g. Power Automate, to establish a trust relationship with the Microsoft identity platform. This is done using Entra Admin Center (previously known as Active Directory)

This post is part of the Power Automate and Power BI Rest API series

Create App Registration for Service Principal

Navigate to https://entra.microsoft.com/. Then in the menu expand Identity and under that Applications. Finally click on App Registrations to open the list of App Registrations. It defaults to list registrations owned by you.

Click on New registration to start creating your App registration. Enter a name for your application, remember to make clear as over time the list will grow. Leave rest of the options as default and press the Register button.

Change App Registration Ownership

It is recommended that you add owners to the registration. Click on Owners, then click on Add Owners and select the owners. App owners and administrators can view and edit app registrations.

Adding API Permissions

Permissions need to be given to the App Registration based on what you want to do. The Power BI Rest API documentation at https://learn.microsoft.com/en-us/rest/api/power-bi/groups/create-group states you need Workspace.ReadWrite.All.

Click on API Permissions and then click on Add a permission. When the list of APIs appear, select Power BI service. Then select Delegated permissions and from the list of permissions select Workspace.ReadWrite.All. It is good practice to only add the permissions you need.

			## Adding a Secret

The final step in App Registration is to add a secret. This will be required by anyone using the app registration.

Click on Certificates & secrets, then click on New client secret. Enter in a description, I chose to keep the default 6 months expiration. Click Add to finish creating the secret.

Once the list appears you must copy the secret value. This is the only chance you get.

Important Information

You will need the Secret value copied from the last section and the Tenant ID and Client ID. The final 2 values can be found by clicking on Overview. These values especially the secret need to be kept secure. So the next post is how to create an Azure Key Vault to store the values.

Conclusion

Understanding how App Extensions and Service Principals work is very useful not only for Power BI Rest api. The Microsoft Graph api can also be accessed using a service principal but that is a whole different series.

Power BI Series

Power Automate and Power BI Rest API

Thu, 15 Feb 2024 14:28:37 +0000

The Power BI Rest API provides the ability to create a workspace. Power Automate includes an action to make a HTTP call, i.e. a Rest API. So we should be able to get a flow to create a workspace. So that was my goal. Writing the first flow and getting it to work involved so many little steps I had to write blog posts so I would remember the steps.

Power BI Rest API Documentation

The documentation refers to workspaces as groups and can be found at https://learn.microsoft.com/en-us/rest/api/power-bi/groups/create-group. It states that the API can be called by a service principal profile and the required scope is Workspace.ReadWrite.All.

Other Settings

You also need to make some changes to Power BI tenant settings as well and I’d recommend creating a security group and using Azure Key Vault to store the credentials. That would make a very long blog post so I’m breaking it down into separate posts.

Create a Power BI Service Principal Profile
Create Key Vault to store Tenant ID, Client ID and Secret
Write a flow to fetch the Key Vault Values
Update Power BI Settings and Create a Security Group
Write a flow to create the workspace
Extras for the workspace

Final Power Automate Flow

The final flow, after post 5, fetches the Service Principal credentials using a child flow, then uses HTTP action to create the workspace and with a bit of parse json action magic uses another HTTP action to change the permissions.

The above flow has flaws, no error handling, no checking the workspace name is allowed and no notifying the admin they have a new workspace.

Next Steps

This is a simple example of using Power Automate with Power BI rest API to perform some admin tasks. This could easily be extended to include other boring admin tasks.

Power BI Series

Microsoft Fabric – Notebook and Data Wrangler

Mon, 13 Nov 2023 16:22:16 +0000

Before I started with Microsoft Fabric I had never created a spark notebook. This doesn’t stop me though. By using a combination of Notebook and Data Wrangler means I can transform data and write to table writing very little code by hand.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Create and rename a Notebook

Inside the Lakehouse, on the ribbon drop down Open notebook and select New notebook. This will create an empty notebook with a default name. We can rename the notebook by clicking on the down arrow next to the name and entering a new name.

Adding a Markdown Title cell

Notebooks use 2 different types of cells, Code and Markdown. The Markdown cells allow for documentation to be included in the notebook. Any cell in the notebook can be converted into a Markdown cell. Below are screen grabs of converting the default cell into a Markdown cell, by clicking on the M button so giving a title cell to the notebook.

Markdown includes syntax for lots of different text formatting. A good reference can be found at https://www.markdownguide.org/basic-syntax/

Load the file into a data frame

Notebooks use data frames to handle blocks of data. The next task is to load a csv file. On the left hand side select Files to list the right file. Then on the Products.csv, click on the three dots menu. Next select Load data and then Spark. This will add a new code cell to the notebook.

Run a code cell

Each code cell has a blue arrow on the left, clicking this runs the code in the cell. The function display lists the data frame contents below the code cell. Being able to run each cell in turn really helps with development. There is a run all option on the toolbar.

We can limit the display function to only list a few lines by using a method on the data frame called take. df.take(4) just returns 4 rows of data. Another method that is useful to examine a dataframe is dtypes. This returns all the datatypes of the data frame. So I edit the code and re-run the cell.

Data Wrangler

Before I can write the data frame to a table there are transformations that need to be done. Firstly there are spaces in some of the column names. Next the ID and Price columns need to be numbers rather than strings. Finally I do not want to include the Notes column.

I could do the searches and find the Spark code to do those transforms. There is an alternative introduced in Microsoft Fabric called Data Wrangler. This will write the code for the three changes we need. We launch Data Wrangler from the Data ribbon – Transform Data Frame in Data Wrangler. This will list your data frames – which means you need to have run the cell that populates the dataframe first. Then we get to see the Data Wrangler screen.

Rename Columns

Product ID column needs to be renamed. Clicking on the three dots on the Product ID column heading. Then select rename column and the Operations pane changes to show rename options. Enter the new name Product_ID and the data pane will update to show the old in red and the new in green. Finally click Apply to save the changes.

The same needs to be done on the Base Price column to change it to Base_Price.

Changing Column Types

The next requirement is to change the column data types. This is very similar to renaming and can be found on the column heading menu. I change the Product_ID column to Int64 and the Base_Price column to Float64.

Dropping a column

Finally we need to drop the Notes column. Again this is found on the column header menu.

Add Data Wrangler code to Notebook

Once we have finished all the transformations we are ready to add the code into the notebook. Click on the Add code to notebook. When the dialog appears, check the code and click Add.

Run the Data Wrangler code

When the new code cell appears, we can see it ends by creating a new data frame df_clean and displaying it. So if we run the code we can see the result of the transformations.

Write data frame to a table

After the transformation code has run, the data frame is ready to be saved into a table. This is the single line of code we have write by hand. A data frame has a write method which accepts format and mode options. Using Ctrl + Spacebar will get the autocomplete to help. Here is the required code.

1
2


# Save df_clean to a table called Products
df_clean.write.format("delta").mode("overwrite").saveAsTable("Products")

Once the code has run successfully, we can look in the Lakehouse and see the new table products.

Conclusion

For the newcomer, notebooks are daunting. With the addition of Data Wrangler we have a tool to help write the transformation code. One quote I heard was Data Wrangler is the Power Query tool for notebooks. I have yet to work out when I would use a notebook over a dataflow but I’m assuming there will be pros and cons that vary depending on the data involved.

Microsoft Fabric – Create tables with Dataflows

Fri, 10 Nov 2023 09:09:14 +0000

As part of Microsoft Fabric, we have new Gen2 Dataflows and they have a new ability. A Gen2 Dataflow entity can have a destination. This means using a dataflow we can create a table in Fabric from another source, such as a folder of files.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Creating Gen2 Dataflows

Open the Lakehouse and then from the Get data drop down on the Home ribbon select New Dataflow Gen2. This will create an empty dataflow with a default name. Clicking on the name in the top left, then a pop-up appears allowing you to change the name.

For this blog post we are going to load the csv files in Sales folder, loaded in an earlier post and video. The files have been loaded into the Lakehouse so the data source is OneLake. In the Power Query window, on the Home ribbon, click Get data. From the menu that appears, we select More.. When the Choose data source appears, the bottom half lists OneLake sources, select the Lakehouse.

When the Choose data appears navigate to find the Sales folder. Then click Create. After a few moments a query will appear showing a row for every file in the folder.

Power Query includes a feature to combine files, it requires the Content column to be binary. On the Content column, click on the datatype ABC/123 in the top left of the column and select binary. This will give you a column of binary

The Content column is now the correct data type to combine the files. When you click on the top right icon it will start creating the helper queries to do the file combining. It will open a combine files dialog, showing you the content of the first file. When you click OK it will build the helper queries and the Sales query will now show the data from the files appended.

Fixing the date column

The CSV files contains dates in American format, MM/DD/YYYY. The dataflow is looking for dates in my format DD/MM/YYYY so makes the date column string format. This needs fixing before saving to a table.

Click on the ABC in the top left of the Order Date column. Then from the data type options select Using locale. When the dialog appears, I select Date for the data type and the English (United States) for the locale. When I click okay it changes the strings into dates. This feature is awesome and probably deserves a post of its own!

Adding a Destination

Now the data is ready we can set up the destination. In the bottom right of the screen it shows Data Destination and should show No data destination.

Click on the plus next to Data destination and from the options select Lakehouse. You will get a connections window appear and you can just click Next. Then the next step asks you to select which workspace and then which container in that workspace. Once you have selected the right place you can update the Table Name. Now you are ready to click Next.

The last step update method and column mapping. The update method is a choice between adding data to the bottom of the table, Append, or overwriting the data, Replace. We are going to use Replace in this instance. The column mapping takes the columns from the query and tries to use those for column name. Column names can only contain letters, numbers and underscores. So if you have column names that don’t fit that it offers to fix that. Once you are happy click Save Settings

Publishing and Refreshing

Now we have a query that shapes our data and we have a destination set up we can Publish the dataflow. This will save the dataflow and will refresh it. If you wanted to just save the dataflow and not refresh you can click the arrow next to Publish and select Publish later.

Once the refresh has completed, we can open the Lakehouse and see the new Sales table has been created.

Gen2 Dataflows Conclusion

Dataflows having a destination to write to is an exciting update. Power Platform has dataflows that can write to Dataverse but they have more issues than I’m willing to list here. Probably 90% of the transformations you need in Dataflows you can do by clicking rather than writing M. So this is mostly a low-code solution for getting data into Fabric. A later post will talk about how to handle not just overwriting the table.

Microsoft Fabric – Create table from CSV file

Thu, 09 Nov 2023 08:41:51 +0000

In a previous post we uploaded a csv file into the Lakehouse. In this post we take the next step to create table from csv file loaded into the Lakehouse. This is the simplest method to create a table and requires no coding. The simplicity means that there are limitations.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Create Table from CSV file

From within the Lakehouse, navigate to view the filename of the file you want to load into a table. Click on the 3 dots next to the filename. Then click on the arrow next to Load to Tables and select New table. When the next dialog appears, by default the table name will match the csv file name in lowercase. The name can be updated. Tick or untick the box regarding the headers, and click Load.

After a short while the new table will appear in the Explorer pane. When you click on the table name, a preview will appear.

Limitations

This method of creating a table leaves the determination of column type to the process. So we have no control of that. For some files this will cause an issue. To avoid any issue on the date format I made sure the date format was YYYY-MM-DD.

You can see what the column types are in the table by clicking on the arrow next to the table name to expand it.

You cannot change the column types from this view. The table is stored in delta format and clicking on the 3 dots menu will give you options to explore the properties and view the actual files that contain the data.

Updating a Table

When selecting to load into a table and selecting Existing table instead of New table will give a slightly different dialog box. You need to select the table and then select if the file data is to be appended or will overwrite the existing data. Be aware there is no undo.

Conclusion

For a clean, well structured CSV file, this is a great place to start. With a slow moving dimension table this could be a good solution as it just requires maintaining a csv file. For more complex files that require transformation there are better methods such as a dataflow or a notebook.

Microsoft Fabric – Upload a CSV file and folder

Tue, 07 Nov 2023 16:50:43 +0000

The simplest way to add data to a Lakehouse is upload a csv file. In this blog post we will upload a csv file and upload a folder of csv files. You can upload other file types, but they take more work than csv files to quickly load into tables.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Upload a CSV

We open the Lakehouse, see previous post on how to create one. Then in the Explorer pane on the left next to Files and click on the three dots. On the popup menu click on Upload and then Upload files. When the pane appears on the right, click on the folder icon in the input box. Once you’ve selected the file, click the Upload button.

Once the file is uploaded it can be found by clicking on Files. If you click on the file it the content will be displayed.

Multiple files could have been selected to be uploaded at the same time. All the files would have been uploaded into the same location.

Upload a Folder of Files

Uploading a folder is very similar to uploading a single file. In the explorer pane on the left, click on the three dots next to File, or a subfolder and select Upload. Then select Upload folder and the Upload folder pane will appear on the right. Clicking on the folder icon will allow you to select a folder. Finally you can click Upload and you will a progress line for every file in the folder.

The list of files that have been uploaded can be seen by clicking on the folder name in the explorer pane on the left

Updating a File or Folder

Data is rarely the same forever so you will want to upload an updated file or folder. If you select a file or folder that already exists you will not be able to click the upload button unless you tick the Overwrite if files already exist.

Conclusion

Manually uploading files and folders is not a great solution for a large number of files but it is a good place to start. The OneLake app will be a great alternative when its fully available and using pipelines to orchestrate file management is also a good alternative and future posts will cover both.

Microsoft Fabric – Create a Lakehouse

Thu, 02 Nov 2023 22:43:20 +0000

Lakehouses are one of the main building blocks in Microsoft Fabric. A lakehouse allows you to store structured and unstructured data in a single location. In this post we are going to create an empty one. Future posts will populate it.

Microsoft Fabric Quick Guides

Create a Lakehouse
Load CSV file and folder
Create a table from a CSV file
Create a Table with a Dataflow
Create a Table using a Notebook and Data Wrangler
Exploring the SQL End Point
Create a Power BI Report
Create a Paginated Report

YouTube Version

Empty Workspace with a Fabric Capacity

We start in an empty workspace that has a capacity assigned to it. A jewel logo next to the workspace name or checking the workspace settings are indicators that it has a capacity. A premium per user capacity does not include all Fabric features.

Create a Lakehouse

There are two ways to start creating a lakehouse. First method is to click New in the top left of the workspace area, by default it shows Power BI options so click on More options. When the screen updates, on the second row is Lakehouse.

The second option is switch the experience in the bottom left of the screen to show all the Experiences. When the menu pops up, select Data Engineering. Then the screen will update and show a slightly different Lakehouse button.

The next step in the create a lakehouse process is to name the lakehouse. The name can only include letters, numbers, and underscore. It will display an error message to correct you. Enter the name and click Create. Then wait a few moments and you will get your empty lakehouse.

What is created?

Every lakehouse created in Fabric also creates an SQL Endpoint and a Power BI Dataset. Even though we have not added data the containers are already there to handle the data.

References

Microsoft’s Lakehouse Overview – https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-overview

Power Automate – Write Me a Flow

Wed, 09 Aug 2023 16:44:09 +0000

Power Automate create includes a feature called Describe it to design it which will write me a flow if I can describe it. In Aug 2023 its still in preview, and is not 100% and only copes with simple requests. It doesn’t require a co-pilot licence though!

I started to explore this feature for a session for Scottish Summit 2023, trying to do 3 demos in a 25 minute session, I needed my first demo to be quick.

YouTube Version

Getting Started

On Power Automate site click on Create and then click on the Describe it to design it tile. This starts a 3 step process. The first step is to describe the flow. My simple request was when I added a new file to a SharePoint library that the Power BI report that was based on that library got refreshed.

Once Step 1 appears I enter in my request using a “When X then Y” pattern. It will show you a list of suggestions as you type. Click the arrow to submit the description. If it works you get a suggested flow and you can click Next to move onto Step 2

When it cannot translate your description into a flow, it shows a message of No flow suggestions. This feature is in preview currently so its not yet working 100%. There are links to provide feedback, they can’t fix what you don’t tell them.

Step 2 – Connections

When you click next it takes you to step 2 of 3. Here you are asked to review the connections the flow will use.

Step 3 – Settings

The final step is to add some settings to the flow steps. The drop downs are populated just like in the full flow editor. In some of my explorations it didn’t give me everything I needed but it works mostly. The you click Create flow.

Power Automate did write me a flow!

Very quickly you get presented with a flow. I recommend editing the name in the top left and of course pressing save in the top right. Then test your flow out.

Conclusion

For getting started, asking Power Automate to write me a flow is great. For developing a long complex flow its not going to help much. It does allow you to explore some ideas and with the right description it will build longer flows.

Power Automate Button in a Power BI Report

Mon, 07 Aug 2023 20:39:39 +0000

You can add an action to a Power BI report by adding a Power Automate button. The button will trigger a flow to run and perform actions. You add data to be used in the flow. In this post I will show to use a flow to add a row to a table that the report is connected to.

YouTube Version

Problem Definition

The report is connected to a table in a database that contains the start times of sections in a bike ride. The section times are used to for calculations and colouring charts. There are three measures that calculate the values to put in a new row of data. I want a Power Automate button to insert a row into my database table.

Add a Power Automate Button

From the Insert ribbon tab in Power BI desktop, we select the Power Automate visual. It’s right down the bottom under Other. This will add you a visual that includes instructions. The first step is adding data to the visual that will get passed into the flow. I have added my three measures.

Creating the flow

After you add the data, the next step creates the flow. Click on the … link on the top of the visual and select Edit. This will open a window in Power BI desktop showing you templates and any existing flows that you have.

In the top left, select New and select Instant cloud flow. Then the window moves to give you Power Automate flow editor inside Power BI desktop. Be aware it will be using the credentials you logged into Power BI with.

Then you can click New step and add your actions. In this example I am using SQL Server connection Action to insert a row to a table. I populate the Server name, Database name and Table name. Then the table fields appear. When I click in ActivityID, the first field the Dynamic content shows me the fields available including values I added to the Power Automate button.

When I click on Power BI data Selected Activity ID, measure from my report, it populates the box and it also puts the action into a Apply to each. This is because the values are sent to the flow in a table. When writing the flow it knows its a table so assumes it could be multiple values so therefore adds the apply to each.

There are multiple ways to handle this and convert that table into just three values, or we can accept we know the Apply to each only ever get a single row. Add a comment in your documentation to save future confusion.

Save and Apply the Flow

When you have completed your flow. You need to add a name in the top left. Then click Save and then click Save and Apply. Yes you need to click both!

After you click Save and Apply you should get a success message in a green bar. Then you can click Back to Report in the top left.

The visual in your report will now have changed to be a blue button. And you can test it out. In Power BI desktop you need to Ctrl + click, when you publish you can just click the button.

Formatting the Button

Similar to other Power BI visuals formatting can be applied. The blue isn’t a theme colour so I recommend selecting a theme colour so it will change with themes. You also can tweak padding, shadows and the text to make it look part of the report. I could not find a way to change the arrow icon.

Flow Environments

A Power Automate flow lives in an Environment and developers usually get to select which environment and get to put flows into solutions ready to go through a development life cycle. When writing that flow in Power BI desktop you do not get a choice on environment. It goes into Default environment and therefore cannot be put through any development lifecycle.

One workaround / hack exists. Thank you to James Reeves who pointed it out in a LinkedIn Post. It works currently but its a hack and should be treated as such.

Flow Permissions and licences

When the flow runs it runs as the user who clicks the button. This means firstly the user must have access to the flow, so don’t forget to share it and if like in my example it uses premium connectors the user needs the right licences.

Conclusion

Adding a way to react to the data and do write is great and Power Automate does it pretty well. There are some missing features though, I’d like to select environment for the flow and be able to select a flow from the solution I’ve added the Power BI report to. And then include that in the solution lifecycle management.

Power BI Series

Power Automate – Get data from a Power BI dataset

Tue, 11 Jul 2023 16:05:12 +0000

Power BI datasets contain a rich useful data resource that would be really useful in a Power Automate flow. In this post we will walk through how to get data from a Power BI dataset and then email that data. This is just one example of how you could use data fetched from a Power BI dataset behind a report.

YouTube Version

Starting the Flow

In this example we want to send a monthly quick summary. So I started by creating a scheduled email, repeated every month. After I enter the flow name and change the repeat to every month rather than the default every minute! Now I can press Create, which after a few moments creates a flow with the recurrence step.

Fetch Data using a Power BI step

Click on New step to start the process. In the search box type in Power BI and click on the Power BI tile when it appears. Now you can search the Power BI actions. In the search box enter in query and this will filter to 2 actions. The action we want is Run a query against a dataset.1

When the step appears, I select the Workspace and Dataset I am interested in. The query text is expecting a DAX statement. Don’t worry you don’t have to write the DAX by hand, we can get Power BI desktop to do it for us.

Writing the Query Text

Power BI desktop can write the query text for you. You need to open the Power BI report or a report connected to the dataset. On a blank page add a table visual. Populate the table with the data you want to use in Power Automate remember to apply the correct filters. I also recommend you remove totals unless you want the totals row in your data.

In my example the table shows Total Sales and Sales Growth for products in the previous month, using Offset Month column from my calendar.

After creating the table, on the Optimize ribbon tab click on Performance analyzer2. When the pane appears on the right hand side, click on Start Recording. Then you click on Refresh Visuals. This tool is usually used for analysing how long each visual takes to refresh and part of that is to provide the query being run.

Expand the Table line by clicking on the +. Then click on Copy Query. Now you can return to the flow in Power Automate and paste the query into the query box.

Sending the Email

The final part of this process is to create the email to send the details. The Power BI step returns an array of the rows of data. The Send an Email action would like this array converted into a html table. So we need 2 actions, create HTML table and Send an email. I am not going to make the table or email pretty. That is a separate post!

Testing the flow

I would test as I added each action to the flow, even on a short flow. Obviously once completed I’d test again.

Resources

The report used for this example can be found on my GitHub

Power BI Report – https://github.com/Laura-GB/DemoData/raw/main/Sales%20Data.pbix
Excel file https://github.com/Laura-GB/DemoData/raw/main/Sales%20Data.xlsx

Conclusion

Remember its Power Automate not Power Integrate. So yes you can load a long list into Power Automate but this is not a good tool to load large quantities of data into a new system. This is a brilliant feature though that unlocks data in a dataset for an automation process.

Power BI Series

SVG in Power BI series
Power BI and Project Online series
Slicers series
Dataflow series
Power BI SVG series
Power Automate and Power BI Rest API series
Power BI and DevOps series
The other action I have no idea how it works and have yet to find any documentation [↩]
In earlier versions of Power BI it was on the view ribbon [↩]

Power BI – Creating a Custom Smart Narrative

Fri, 07 Jul 2023 12:15:53 +0000

Power BI desktop includes the Smart Narrative visual. This will summarise a page or visual in your report for you. Often though the summary it produces is not the summary you would have written. This post will show you how to write your own custom smart narrative that updates and filters exactly the same as the standard smart narrative.

YouTube Version

What is the Smart Narrative?

With a report open in Power BI desktop, you can add a standard Smart narrative by selecting Smart Narrative from the Insert ribbon. It will create text based on the visuals on that page. It doesn’t support all visuals and although can be very good it is very generic.

More instructions and information can be found at https://learn.microsoft.com/en-us/power-bi/visuals/power-bi-visualization-smart-narrative

Create your own custom smart narrative

The smart narrative is a text box with some generated text added. The clever part is adding values to that text box.

Start by adding a text box to your report. Position it and start to add some text.

When you want to add a dynamic result to the text click on the Value button on the Text box ribbon. Describe what you would like to calculate. If the dialog offers suggestions as to what you mean click on the correct one. When you click Save, the value will be entered into the text box.

Formatting to numbers could be applied when creating a text box value. Weirdly you cannot use the same name for values in different text boxes unless you copy and paste the text box then its fine. The values cannot transfer into any other visual either, use measures for that.

Reviewing the Values

When you added values to your custom smart narrative you hopefully remembered to name them. This helps when you go to review the values. Click on Review to list the values. You can click on edit on any value to confirm what question you asked, rename or apply extra formatting

Resources

The report and the data source used in this blog post can be found on my Github

Power BI Report – https://github.com/Laura-GB/DemoData/raw/main/Sales%20Data.pbix
Excel file https://github.com/Laura-GB/DemoData/raw/main/Sales%20Data.xlsx

Conclusion

Know your audience, does a summary paragraph work for them? If it doesn’t then stick to visuals that work for them. Some clients love a description others just prefer a batch of cards showing the same values.

Power BI Series

Power Platform Solution and Power BI – Part 3

Mon, 22 May 2023 22:17:42 +0000

In Part 1 we added Power BI to the Power Platform solution and then in Part 2 we added a dashboard. This is post we will embed the report into a form and filter the report to match the context of the form. For this we will use the Accounts form and filter the expenses report to only show the expenses related to one account.

Series

YouTube Version

Embed the report into a form.

In my Power Platform solution, the Accounts table has one form included. When I edit the form I can see that I previously added a tab for expenses. On that tab there is an empty section, ready to insert the report.

In the left hand side menu, select components and expand Power BI. Drag and drop Power BI report into the blank section. An pane appears asking lots of questions regarding showing various parts. For now leave all of them as default and click Done. The unfiltered report should appear.

Filtering the embedded report.

The report needs to be filtered to only show the expenses of the selected account. With the report selected you will get the properties pane showing on the right hand side. Expand Components and click on Power BI Report. In the pane that appears, same as before find JSON filter string. You need a JSON string to put in the Static value.

The filter we are going to apply is straight forward match the account id on the form with the account id in the report. We need 3 bits of information

Power BI Report table name – Account
Power BI Report column name for the account id – AccountID
Dataverse logical column name for account id – accountid

You must get the case and of course the spelling right. We then insert these into a json string. The whole string must be on one line, no returns and tabs to make it pretty. Below is an image to explain the parts and below that is the code I used.

1

{"Filter": "[{\"$schema\":\"basic\",\"target\": {\"table\":\"Account\",\"column\":\"AccountID\"}, \"operator\":\"In\",\"values\":[$a],\"filterType\":1}]", "Alias": {"$a": "accountid"}}

When editing a form in Dataverse, the form is in new record mode so if you have the filter right the report should filter to nothing. When it is working click Done and Save and Publish your form.

Testing the form

Make sure you test your form. Open your app, open an account and look at the expenses and the report. If your report is a direct query report you should be able to add a new record and the report update.

Conclusion

The three posts combined gives us the building blocks to include Power BI in our Power Platform Solutions. I’m hoping more Power BI artefacts such as dataflows also get included in time. There will be future posts, let me know what you want me to do.

Here are the resources I usedAbout Power BI in Power Apps Solutions – Power BI | Microsoft Learn

Power BI Series

Power Platform Solution and Power BI – Part 2

Sat, 20 May 2023 19:08:20 +0000

Intro – Add a Dashboard

In Part 1 we added a Power BI dataset and report to a Power Platform Solution. In Part 2 we will create a dashboard and add it to the model driven app.

Series

YouTube Version

Create the Dashboard

In the solution click on +New and select Dashboard and then Power BI embedded. When the pane on the right hand side loads, enter in a name for your dashboard. Then make sure Power BI report is selected and select your report from the drop down.

Add the Dashboard to the App in the Power Platform Solution

After the dashboard has been added, you can add the dashboard to a model driven app. Edit the app by clicking on the name of it in the Power Platform Solution. Then click on Add Page and the New page dialog appears. On the first page select Dashboard and then click Next. Then on the next page, select your report, probably at the bottom of the list. Finally click Add to add the dashboard to your app.

Note

At the time of publishing there is a small bug that the report doesn’t show correctly in edit mode but if you publish all customisations it fixes it in play mode. Remember this feature

Testing the Report

As with any addition to a development you should test the report works. In my example the report is a direct query. So my test is check the dashboard, then add a new expense record and then return to the dashboard to check the new record is showing.

Conclusion

For some projects this is enough, an embedded dashboard in the model driven app. The next post will be to add an embedded report into a form.

As in Part 1 here are the resources I’ve used

About Power BI in Power Apps Solutions – Power BI | Microsoft Learn

Power BI Series

Power Platform Solution and Power BI – Part 1

Fri, 19 May 2023 08:26:43 +0000

Power BI reports can be an important part of a Power Platform solution. Microsoft has added a new feature to Power Platform Solutions to add a Power BI dataset and report. Then you can add the Power BI report in as a dashboard or as an embedded report in a form and add context filtering. Part 1 will cover adding the Power BI dataset and report to the solution.

Series

YouTube Version

Preparation

For this post I have a Power Platform solution already. I have a Power BI report connected to the environment and published to a workspace.

Add Power BI to Power Platform Solution

First task is to add the Power BI report to the solution. Start by clicking on Add Existing and from Analytics selecting Power BI report. This will bring the underlying dataset in as well. Then click on From Power BI which will show you a workspace drop down*.

From the drop down, we select the right workspace and then the right report. Then we click on Add.

It takes a while for the dataset and report to be added. Behind the scenes a new workspace is created and the report is copied to there. If you don’t have permission to create a workspace you will get errors.

Checking the New Workspace

I recommend that you check the workspace. Check the credentials in the dataset and refresh the dataset and finally make sure the report opens and shows data. Remember to setup a refresh schedule if your report needs it.

Note the message across the top telling you that this is a workspace connected to a solution and the workspace icon is the Power Apps icon.

Finishing the connection part

The report is connected to this environment’s Dataverse. When the solution is exported and imported the report needs to be connected to the new environment’s Dataverse.

In the solution, I clicked on the three dots and selected Edit. Then a pane will appear on the right. Then select Yes from for the existing connection question and click Save.

Conclusion and Resources

Congratulations, you now have a solution that includes a Power BI report.

This is just Part 1. The future parts will cover adding a dashboard and an embedded report. Here are the resources I have used.About Power BI in Power Apps Solutions – Power BI | Microsoft Learn

Power BI Series

Using SQL on Dataverse for Power BI

Wed, 26 Apr 2023 16:55:19 +0000

SQL (Structured Query Language) has been used to get data in and out of databases for decades. Dataverse is a database. So lets use SQL on Dataverse to fetch data into a report. This can simplify our queries and make a direct query report easier to write.

YouTube Version

Microsoft SQL Server Management Studio

There are multiple tools available to write and test your SQL, for this post I am using SQL Server Management Studio (SSMS), which is a free tool from Microsoft. You can download SSMS from

https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms

After you have downloaded SSMS, you can start it. Then you need to login to the specific environment. A dialog should pop up to fill in or you can click Connect. The Server type is Database Engine. The server name is in the url path from running a model driven app.

For this post the authentication I am using is Azure Active Directory – Password. So I also supply a User Name and Password. Then you can click Connect.

When connection has completed, the Object Explorer pane will show the hierarchy of items. Expand Databases to find your environment database and then expand that database and Tables to see the complete list of tables. Expand the table to see the list of columns.

Write your first query

This post is not to teach you SQL. There are plenty of sites to do that. Here are some very simple queries to get you started. For our first query, we will write a query to fetch a list of accounts. Click on New Query and a blank page appears. In here we can type a simple query to list all accounts. Click Execute to run the query and see the results.

SQL Code

1
2
3
4
5
6
7
8


Select
	accountid as AccountID,
	name as "Account Name",
	industrycodename as Industry,
	statecode,
	statecodename
from
	dbo.account

Filtering the data

The previous example includes an account that is not active. Contoso Pharmaceuticals has a statecode of 1 and statecodename of Inactive. For this example we are going to filter to only show the active records. The active records have 0 in the statecode column. The WHERE clause applies the filter. The statecode and statecodename columns can be removed.

1
2
3
4
5
6
7
8


Select
	accountid as AccountID,
	name as "Account Name",
	industrycodename as Industry
from
	dbo.account
where
	statecode = 0

Using SQL on Dataverse in Power BI

Now we have an SQL statement we can use it in Power BI. Start POwer BI and then click Transform data to open Power Query. Then we create a parameter by clicking Manage Parameters and then click New and fillin in the details. The current value should be the environment path you used earlier.

Now you have the parameter we can create the query. Start by creating a blank query by clicking New Source and then Blank Query. Then on the Home ribbon click Advanced Editor and paste in this code. I have adapted a pattern created by Scott Sewell.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


let
 Dataverse = CommonDataService.Database(Environment, 
 [CreateNavigationProperties=false]),
 SQL = "Select
	 accountid as AccountID,
	 name as ""Account Name"",
	 industrycodename as Industry
 from
	 dbo.account
 where
	 statecode = 0",
 Source = Value.NativeQuery(
 Dataverse,
 SQL,
 null,
 [EnableFolding=true]
 )
in
 Source

Note 1 – the CreateNavigationProperties and EnableFolding are important flags to help with performance. Scott Sewell explains both topics in these posts.

https://www.linkedin.com/pulse/speedtip-enablefolding-option-dataverse-native-sql-power-sewell/https://www.linkedin.com/pulse/stuck-evaluating-dataverse-source-power-bi-try-speedtip-sewell/

Note 2 – If your SQL includes any ” you will need to double them up inside the string. In this example the ” is needed as the column Account Name has a space in it’s name.

For other queries you just replace the SQL = step with a different SQL statement. You can add further transformations if you need them into Power Query. Remember to rename your query to a clear name.

Permission to Run a Native Database Query

SQL is a powerful language. On some databases, not Dataverse, it can edit or delete the data. So Power Query will check the first time it runs any query. A message in a yellow bar appears stating Permission is required. When you click Edit Permission, a dialog appears showing you the query and you can click the Run button. Then the data will appear.

Loading into Power BI Desktop

Now you have a query that loads data you can load it into Power BI desktop by clicking Close & Apply on the home ribbon. Before it will load the data it will ask you to set the storage mode. For this post I am selecting DirectQuery. This means the data is live and data security is applied using the report viewer’s security. (See publishing to get this 100% right)

It will ask you confirm running the query again, just click Run. Once the data has loaded we add relationships if required, measures and visuals into the report. Be aware direct query will need to refresh and so will be slightly slower.

Publishing

Once you have completed the modelling and visuals, you can publish the report. You need to check the credentials of the report. Even if credentials look okay, confirm the viewers credentials are used. In the workspace, open up settings of the dataset. Under Data source credentials click on Edit credentials. Authentication method should be OAuth2 and select the appropiate security level.

If you are using direct query then tick the bottom option so that report viewers credentials get used to fetch the data. Security within Dataverse is now applied to your report.

Click Sign in and complete the sign in to now make your report ready for the report consumers.

Conclusion

Dataverse is a great data source for business apps and being able to report on the data in there is important. Building a simple efficient pattern to do this is important. Hopefully this post helps get you there.

Power BI Series

Power BI – Embed a Paginated Report

Thu, 30 Mar 2023 19:24:23 +0000

Within a Power BI report you can embed a paginated report. Values can be passed to the paginated report to populate parameters. The paginated report can be exported by a report consumer.

YouTube Version

Paginated Report

For this post we are going to use a paginated report connected to restaurant sales data. The report shows daily total dishes sold and takings. It uses 2 parameters, StartDate and EndDate to filter the report.

Prepare Measures for Parameter Values

The Power BI report in this example uses the same data source of restaurant data. It doesn’t have to. The report requires 2 parameters. The easiest way to fill these vales would be to create measures. We create 2 measures to define the date range.

1
2
3


StartDate = MIN('Date'[Date])

EndDate = MAX('Date'[Date])

Embed a Paginated Report

In a report in Power BI desktop drop down the visual selection from the Home or Insert ribbon. Then select the Paginated Report visual from the Other group.

When the visual appears, click on the Build Visual button to display the data options. Add the measures to the visual. Then click Connect to report button to select the report.

The next dialog shows all the paginated reports you have access to and includes a search box. Select the correct report and then click Set Parameters. Add the relevant measures from the dropdown. The current values of the measures will appear next to each measure.

Click See report to see the report. In this report there is also a chart showing sales per week.

Filtering the Paginated Report

When a filter is applied that changes the measures used as parameters, a Apply changes button appears. Clicking this will filter the report.

We can remove Click on the report and click the format button. Then tick the Auto-apply. Now when the measures change the report will refresh immedeatly.

Exporting the Report

After the Power BI report is published, consumers can view it from the workspace or workspace app. When viewing the report the viewer can click on Export and select a format. Then all pages will download as a single file.

The toolbar at the top of the report allows the viewer to navigate the pages. We can open the report with the current filters applied by clicking Open Report.

Restricting Viewer Options

When designing the report you can add restrictions to the embedded Paginated report from the format pane. We can hide the complete toolbar, or remove access to export, navigate pages and viewing the report alone. We can restrict exporting or the different export formats.

Conclusion

This was a very simple example with 2 simple parameters. The paginated report could have optional parameters with default values and could accept multiple values. The tricky part there is setting up the paginated report rather than embedding it.

One great option would be to use bookmarks to save the common parameters.

Power BI Series

Power BI – Clear all Slicers Button

Sat, 25 Mar 2023 16:34:22 +0000

The March 2023 Power BI update included a new button, Clear All Slicers. This allows you very quickly add a reset on a report for the slicers. It partly replaces the previous method explained in https://hatfullofdata.blog/power-bi-resetting-slicers-with-a-bookmark-button/

YouTube Version

Adding the Button

Head to the Insert ribbon and under Buttons select “Clear all slicers”. Nothing else needs to be done, although I would recommend formatting the button. The button will reset slicers to have no filters applied.

Formatting

The default styling for this button does not include a border. My opinion, which is potentially wrong, is there should be a border of some sort.

The March 2023 update also included formatting updates. So to access the formatting pane of the button you might need to click the ellipses (…) on the button and select Format.

In the format pane on the right of the screen look for Button Style. In that section you will find Text and Border. You can change and format both the display text and the border details.

Word of Caution

The button to reset slicers back to no filter. You need to be clear what that excludes. It will not remove filters applied by clicking on a visual and will not change a slicer that has a single select. You will need to use the previous method linked above to achieve that.

Conclusion

For most reports this is a great addition. Too many reports with lots of slicers do not include a quick obvious way to reset the report filters. Combined with an apply all filters button this will greatly assist on filtering and unfiltering a direct query report.

Power BI Series

Power BI Adoption in Your Business

Sat, 18 Mar 2023 18:07:15 +0000

So you’ve found Power BI, you’ve created amazing reports that answer every question your business has. Yet when you look at the stats your reports are not being looked at. People are still returning to the old spreadsheets and creating their own pie charts. What has gone wrong with your Power BI adoption?

You might have adopted Power BI but your business hasn’t. You need to take your business on the Power BI adoption journey. No they don’t need to learn M and DAX, but they do need to understand how they can view reports and how to use them. This post is to go along with a set of videos looking at ways to deliver the reports to your business report consumers. Click on the pictures to view the videos.

Report Delivery

Workspaces are not user friendly, you need to find the workspace and then the report etc. So what options are there for delivering the report to where the viewer already is. Here are four videos to get you started

My favourite update for 2022 was being able to embed a Power BI report into PowerPoint with full functionality. This bring Power BI into the meeting without any fuss, window switching. Embedding is fast and simple assuming everyone has licenses and access.

For outside the meeting but still in the areas your users work embedding in a SharePoint page could be the answer. If your users are working every day in SharePoint then this is the obvious place to embed your report.

Most Office workers use MS Teams for almost everything. So its obvious to pin that Power BI report into the relevant Teams channel for everyone to be able to quickly find it when they are working in that channel.

When you have a complete report suite that should be delivered together then a Power BI App is your best option. Using Audiences you can tailor the experience based on the viewer. This is my second favourite update from 2022.

Report Experience

The average manager does not believe they have time to learn the complexities of how a new report works. So you need to make the transition easy. Give the 5 minute tour, listen to feedback and show them how to make it work for them.

When a report covers multiple teams and slicers give users the specific results they need, show them how to use Personal Bookmarks. Now the report view is under their control and its quick and easy.

When that multi-team report is exposing data too widely then Row Level Security should be considered. This is the simplest version. Another post will take this further.

Old habits die hard

Excel is here to stay, do not make your Power BI adoption project a battle you cannot win. As much as Power BI can do much of the reporting that Excel does there are times when exploring the data in Excel is the best option. Showing your users how to connect Excel to a Power BI dataset will at least make sure the data they have in Excel should be the same as the data in Power BI.

With the proper controls users can pull data from a Power BI dataset into a Pivot Table or into a table. However much you’d prefer they didn’t, they will want the data in Excel. So let them do it, with it controlled by you.

Conclusion

In order for your reports to be used, trusted and promoted across the business you need to deliver them. Hopefully you have a few more ideas now on how you could do that. Please let me know if you have any other ideas to add to the list.

Power BI Series

Export Data from Power BI Visuals

Sun, 05 Mar 2023 22:52:49 +0000

Excel has been the backbone of finance departments for decades. Almost every company has multiple spreadsheets that are business critical. Many Power BI adoption programmes have aimed to reduce the reliance on such spreadsheets. But we may still need to export data from Power BI into an Excel spreadsheet as old habits die hard and Excel is an amazing tool and Power BI will never replace it completely. This is one possible option and I will cover creating pivot tables from a dataset in another post.

YouTube Version

Export data from a visual

When viewing a report in the Power BI Service, the viewer can click on the three dots usually in the top right of the visual. When the menu that appears, they can click Export data. (See Admin and Report Settings sections if its not there) Then it will display a dialog giving potentially multiple ways to export data.

Different options will be available based on the visual selected and the report options. The details for each option will be covered.

Data with current layout

This option is only available on Table or Matrix visuals. It creates an Excel file with almost exactly the same layout. The file is a pure export and does not include a refresh.

Summarized Data

Assuming the admin and report creator have allowed export, this option is available for most visuals. When selected it has three possibilities. The live connection is only available if you have contributor or higher access, i.e. its not available if you have Viewer access.

Excel with live connection, 500,000 row max.Live connection means the data can be refreshed at a future time.
Excel, 150,000 row maxExport without refresh.
CSV file, 30,000 row maxCSV cannot include a refresh.

Underlying Data

This option is only available if the report creator has enabled it. (See Report Setting Options below) It exposes more columns than summarized. In this example it exports all the columns from the fact table, the 2 columns from the dimension table used and the measure used.

Applied Filters in Export Data

When the data is exported, the exported data will be filtered with the report filters applied. Then the filter details are included in Excel files. CSV files will be filtered but will not include the details.

Report Setting Options

A report creator can control if report readers can export data from the report. These settings are found under File – Options, CURRENT REPORT – Report settings. Then look under Export data. The first option is the default and will allow Data with Current Layout and Summarized data. When option 2 is selected, underlying data export is enabled as well. Option 3 disables all exports from this report.

Admin Options

Power Admins have ultimate control over this feature. There are 3 features they can restrict, enable or disable. These options are Export to Excel, Export to csv and Users can work with Power BI datasets. Disabling the third option will disable files having a live refresh.

Also by using security groups, you can restrict any of these features to limited users across the tenancy.

Data Export Conclusion

Educating users in good practice on using these features will help your strong Excel users feel more comfortable. I can understand companies restricting parts of this, but I would hope it is not just a blanket ban hoping this will speed up adoption of Power BI. In my experience it will just make some people more stubborn. Stubborn workarounds will be shadow IT and increase your technical debt.

Resources

The UK Weather pbix file is on my github site – https://github.com/Laura-GB/DemoData#video-and-blog-resources

Power BI Series

Power BI – 5 Stars SVG

Sat, 25 Feb 2023 22:57:38 +0000

In the Feb 2023 Power BI Update they introduced image width in a table formatting. This means SVG images now don’t have to be square, so we can now do rectangle images. So I ported Stars SVG code I had in my Power Apps series and added this post to my Power BI SVG series

For this post we are using a table of events that have a rating score. I want to add a visual version of the rating to the table visual.

Resources for this post and video can be found at https://github.com/Laura-GB/DemoData/

YouTube Version

Draw 1 Star SVG

The first step is a measure to draw a single star. That starts with the SVG essentials similar to previous examples and the basic star shape. We need to remember to change Data Category the measure is an Image URL under data category and then add it to the table

1
2
3
4
5
6
7
8
9


Stars = 
// svg essentials
VAR svg_start = "data:image/svg+xml;utf8,"
VAR svg_end = ""
// star shapes
VAR Star1 = ""

RETURN
svg_start & Star1 & svg_end

The default image size will be too big. In the formatting for the table, change the image size. I’ve used 30 px, pick a size that works for you.

Draw 5 Stars SVG

In order to draw 5 stars in a row the view box needs to be expanded to fit them all in. So the svg_start needs to change to 500 100. This will mean that the above width will also need to change. One star is 30 px, so 5 stars will need a width of 150 px.

Then we need to draw 5 stars in a row. We can use GENERATESERIES to create a table with numbers 0,100..400 so we can draw a star at 0,0 and 100,0 etc. That can be used inside a CONCATENATEX to create 5 groups. Each group will use a transform, translate to position each star.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


Stars = 
// svg essentials
VAR svg_start = "data:image/svg+xml;utf8,"
VAR svg_end = ""
// star shapes
VAR Star1 = ""
VAR Star5 = CONCATENATEX(
 GENERATESERIES(0,400,100),
 "" & Star1 & ""
)
RETURN
svg_start & Star5 & svg_end

Grey Star Outlines

The first layer of stars is outline stars so we need to put the 5 stars in a group with a white fill and 1 point black outline.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


Stars = 
// svg essentials
VAR svg_start = "data:image/svg+xml;utf8,"
VAR svg_end = ""
// star shapes
VAR Star1 = ""
VAR Star5 = CONCATENATEX(
 GENERATESERIES(0,400,100),
 "" & Star1 & ""
)
// Defs

// Coloured Stars
VAR GreyStars = "" & Star5 & ""
RETURN
svg_start & GreyStars & svg_end

Define Clip Path

The gold stars need clipping to match the rating score. 5 stars is 500 wide so a rating of 2 needs the gold stars clipped at 200, and 3.5 at 350 etc. We use variable ClipWidth to store this. To clip an image we use the and tags. The clipPath includes an id so it can be used later. The clipped area is a rectangle from with a height of 100 and a width of ClipWidth.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


Stars = 
// svg essentials
VAR svg_start = "data:image/svg+xml;utf8,"
VAR svg_end = ""
// star shapes
VAR Star1 = ""
VAR Star5 = CONCATENATEX(
 GENERATESERIES(0,400,100),
 "" & Star1 & ""
)
// Defs
VAR ClipWidth = [Avg Rating] * 100
VAR Defs = "
 
 "
 
// Coloured Stars
VAR GreyStars = "" & Star5 & ""
RETURN
svg_start & GreyStars & svg_end

Draw Clipped Gold Stars

The last step is to draw the 5 gold stars using the clip path. So we use another group, apply a fill of gold and the clip path. The clip path name

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


Stars = 
// svg essentials
VAR svg_start = "data:image/svg+xml;utf8,"
VAR svg_end = ""
// star shapes
VAR Star1 = ""
VAR Star5 = CONCATENATEX(
 GENERATESERIES(0,400,100),
 "" & Star1 & ""
)
// Defs
VAR ClipWidth = [Avg Rating] * 100
VAR Defs = "
 
 "

// Coloured Stars
VAR GreyStars = "" & Star5 & ""
VAR GoldStars = "" & Star5 & ""
RETURN
svg_start & Defs & GreyStars & GoldStars & svg_end

Conclusion

The width addition to images in tables and matrix opens up lots of possibilities to use images in tables. There are lots of ideas for visuals to add to a table or matrix.Kerry Kolosko has some brilliant templates to get you started at https://kerrykolosko.com/portfolio/

Power BI Series

Power BI – Take Over a Dataset

Thu, 24 Nov 2022 14:24:00 +0000

A published dataset in the Power BI service is owned by one account, often a person. People change jobs, move projects so that dataset is no longer part of their job. So another accounts need to take over that dataset so the connected reports can get refreshed data. This post will walk you through the simple steps with a few words of caution along the way.

YouTube Version

Initial Dataset

Click the three dots on a dataset and select Settings. At the the top of the Settings window we can see AmyA currently has control of the Account report dataset. So this account, DanJ cannot edit scheduled refreshes, edit parameters etc. If Amy’s account gets deactivated then the refreshes will fail.

Perform Take over

Before Dan clicks Take over it is very important to make sure Dan has access to the data sources that this report uses. If Dan does not have access and continues, they could break the connections and therefore break the refresh on this report.

When Dan clicks on the Take over button, it displays a warning message.

When Dan clicks Take over the stored credentials are deleted so the credentials will show an error. In this example there is only one data source so only one set of credentials. Click on Edit credentials for each set of credentials and login or enter credentials.

Once all the credentials have been updated Dan can now configure everything on the dataset. If Amy looks at the dataset they will be able to see Dan has completed the take over. You must have at least Contributor access to the workspace to do this.

Take Over – Word of Caution

There is no undo. If you do not have credentials for the data sources used in the dataset the refresh will no longer work for import sources and the whole report won’t work for direct query reports. So I highly recommend doing this before your report builder leaves the company and a full handover is completed.

Power BI Series

Power BI – Connecting to Google Analytics

Tue, 22 Nov 2022 09:24:52 +0000

When I started my WordPress blog I added a Google Analytics plugin. This allows me to track visitors to my blog. So now I have a few years worth of data to load into a Power BI report. Power BI has a Google Analytics connector. This blog post is to walk through building a simple report on my blog visitors.

YouTube Video

For those who prefer a video guide.

https://www.youtube.com/watch?v=AdvAq-B6ZWg

Creating the connection to Google Analytics

In Power BI desktop the connector can be found under Get Data > More and then Online Services.

The connector relies on a third party, i.e. Google. Therefore Power BI shows a message warning the connector user that Microsoft can’t guarantee the connector will always work correctly. Google will also confirm that its okay for Power BI to connect to your Google Analytics.

Selecting Fields

When the connection is established the Navigator dialog lists all the fields grouped in folders. The fields have two types, Dimensions and Measures. Every report must include at least one of each. For this example I select from the Time folder, Date and Month of Year as the dimensions. Then from the Page Tracking folder I select the measure Pageviews. Finally I click Load to get the data into my report.

Quick Chart

A quick column chart, using Month of Year and Pageviews shows the page views over the past few years.

Adding more fields from Google Analytics

Google analytics has many more dimensions and measures to explore. After you click on Transform and return to Power Query you can see the applied steps. My first attempt to add more dimensions to the query was to click on the cog next to Added Items. This gives you can error so you cannot just modify that step.

The error is not a problem. Instead of modifying the existing step, the easiest method to add new dimensions is to add a new step. On the Cube Tools – Manage ribbon, click on Add Items. For this example I added the following:

Geo Network

Continent

Country
Platform or Device

Browser

User

Users

After adding the new items, I return to the report and add slicers and more visuals.

Power BI Series

Power BI – Switch Dataverse Tenancy

Tue, 01 Nov 2022 12:47:36 +0000

Connecting to Dataverse with Power BI to report on the business data is becoming easier and more common. One difference with connecting to Dataverse is there are two connectors involved. One to list the available environments and the second to connect to the data. Occasionally you need to change to point to a different tenancy which means changing the first connector to switch Dataverse tenancy.

YouTube version

Starting Point

When I click Dataverse on the Home ribbon the navigator shows me where I last connected. I need to connect to a different tenancy for this report.

Switch Dataverse Tenancy

You need to change the login for the Dataverse login. From the Home ribbon, click the down arrow on Transform data and select Data source settings. In the Data source settings dialog select Global permissions. Then select Dataverse and select Edit Permissions.

In the Edit Permissions dialog click on the Edit button. This will prompt you to login to the different tenancy. If you do not have a Privacy Level selected I recommend you select one, often Organizational will fit most data sources. It will depend upon your company’s policy.

Result

Now when I click on the Dataverse button I get the list of environments on the different tenancy.

The Dataverse connector is only used to populate the Dataverse Navigator window. Switching the tenancy will not break any other reports. If all you do is report from one tenancy you will never have to do this. Though I’ve been asked to explain this multiple times.

Power BI Series

Power BI & Dataverse – Choice Columns

Sat, 24 Sep 2022 19:24:07 +0000

Choices and Choice Column Introduction

When a choice column is created in a Dataverse table the developer can either select a pre-made list of choices or create a custom list of choices. In a Power BI model it helps to add the dimension table of those values.

The difference between a choice and choices column is how many choices can be picked. A choice column only allows one choice and a choices columns column allows multiple values to be added. So if we take dish from our menu then the menu section, eg starter, main, side would be a choice but the allergens such as gluten, dairy etc could be a choices column.

Choice Column in Power BI

When the table is loaded into Power BI there are two columns for the choice column. The first column is the number that refers to the choice in the choices list and the second column is the value from the choice list. The limitation of using the name column is only selected values will in your data and it will slow down the refresh of the data.

Choices Column in Power BI

When the table is loaded into Power BI there is a single column for the choices column. This column contains numbers separated by commas. Each of these numbers refers to a choice in the choices list. Using the Dataverse connection, there is no easy to get the choice values.

Using the Web API to get choice lists

Choice are part of the table and columns definition and one of the four paths exposed is Global Option Set Definitions. Choices and Options are the same thing in this context. Microsoft give us the path on this page

https://docs.microsoft.com/en-us/powerapps/developer/data-platform/webapi/use-web-api-metadata

The path given is

1

[Organization URI]/api/data/v9.0/GlobalOptionSetDefinitions

If you navigate to that url in your environment you will get JSON data. Then exploring the JSON we can find the Name of the choice list. Expanding Options we get the Value which matches the values in the choices column in our table. Expanding Label and then UserLocalizedLabel we get the Label value.

Using the Web Connector to link to the JSON we can connect. Power Query decides to expand everything for you. I deleted all the steps except Source and expanded only the columns I needed.

Before I add the list I create a parameter called Environment to store the path to the my Dataverse environment, eg orgdaae212f.crm11.dynamics.com.

Then this code gives you the list of all the choice lists in your environment. I save this as one query which is not loaded and then I add referenced queries for each choice list that I want to include in my report.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


let
 Source = Json.Document(Web.Contents("https://" & Environment & "/api/data/v9.0/GlobalOptionSetDefinitions")),
 value = Source[value],
 #"Converted to Table" = Table.FromList(value, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
 #"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Name", "Options"}, {"Name", "Options"}),
 #"Expanded Options" = Table.ExpandListColumn(#"Expanded Column1", "Options"),
 #"Expanded Options1" = Table.ExpandRecordColumn(#"Expanded Options", "Options", {"Value", "Label"}, {"Value", "Label"}),
 #"Expanded Label" = Table.ExpandRecordColumn(#"Expanded Options1", "Label", {"UserLocalizedLabel"}, {"UserLocalizedLabel"}),
 #"Expanded UserLocalizedLabel" = Table.ExpandRecordColumn(#"Expanded Label", "UserLocalizedLabel", {"Label"}, {"Label"}),
 #"Changed Type" = Table.TransformColumnTypes(#"Expanded UserLocalizedLabel",{{"Name", type text}, {"Value", Int64.Type}, {"Label", type text}}),
 #"Renamed Columns" = Table.RenameColumns(#"Changed Type",{{"Name", "Choice List"}, {"Value", "Choice Value"}, {"Label", "Choice Name"}})
in
 #"Renamed Columns"

Result

After you add the above code, you get a table of all the options available. Then using reference and a filter and the choice dimension tables can be created.

Conclusion

Its not pretty, its not clean and I’d prefer a straight forward connector. It works for now though and until there is another method that works this is the method I’ll use. If there is another method please correct me via the comments below and I’ll correct my post!

Power BI Series

Power BI Dataverse Connector – July 2022 Update

Thu, 01 Sep 2022 16:33:32 +0000

In the July 2022 Power BI update, they updated the Dataverse connector. The connector now uses the display names from a Dataverse table. Dataverse columns have internal name and display names and previously the display names were not available.

YouTube Version

For those who prefer a video version.

New Connection

Let’s start with a brand new Dataverse connection. When you click on the Dataverse button the dialog appears as before. When you select a table it still shows the internal names of the columns.

When you click Transform and Power Query opens it still shows the internal names. So I select a few columns and then click Close and Apply. When the data appears in Power BI desktop it now shows the display names. If a column does not have a display name such as statecodename it shows the internal name.

Power Query and Power BI Desktop views

When I experimented I found that even if you renamed the columns within Power Query the name within Power BI desktop did not change. And if you changed the name in Power BI desktop it did not push back into Power Query as it does with other connectors.

Exploring with Tabular Editor

Opening the model in Tabular Editor 2 we can see that the column has two names. So the source column has a different value to the Name property.

Updating an Existing Model

I created a Power BI model in an earlier version of Power that used the old Dataverse connector. It contains the Account table with columns renamed in Power Query and a measure to count the Account table records.

Report before refresh

When I opened the file in the July 2022 version of Power BI with the new Dataverse Connector, it looked unchanged. When the report is refreshed in Power BI desktop the connector is updated. The column names change and measures update and still work.

Report after refresh with updated column names

Exceptions

If the table from Dataverse is transformed using a reference the display names do not come through. I have not explored all the combinations as to which break the new update so please be aware.

Conclusion on Dataverse Connector

The update to the dataverse connector makes this connector work differently to other connectors. This update works well if all the transformation does is reduce the columns and filter the rows. This will work for many reports.

Power BI Series

Power Automate Child Flow

Thu, 14 Jul 2022 12:34:07 +0000

Occasionally it is useful to break your flow apart into separate flows. For this you need to create a child flow or two. There is one tweak I always forget, hence this post.

Create the Child Flow

When building child flows you need to follow these rules

Child flows need to be built inside a solution
The trigger needs to be a Manually trigger a flow. Add inputs to the trigger.
To return values back to the parent flow the last action should a Respond to PowerApp of Flow action.

Above is a very simple example. The trigger has an input of location and the child flow will return 3 values.

Calling the Child Flow

Now you have created your child flow and tested that it works you can now call it. In the parent flow, add the action Run a child flow. Select your child flow and populate the inputs.

As soon as you add this you will probably get an error in the flow checker. Stating “Update the child flow for action ‘Run a child flow’ to not use ‘run-only user’ connections”

Fixing the Child Flow

Return back to the child flow information page. In the bottom right of the window look for Run only users box. When you have found it, click Edit. For each connection change the Provided by run-only user to the connection listed. You get a dialog stating the run-only users only get access to the connections for this flow. Click OK.

When you have updated all the connections remember to click save.

You can now return to the parent flow. You might need to re-add the action but you will no longer get an error.

Conclusion

Child flows are a great way to re-use logic and break a process down into manageable parts.

Power BI Charticulator Resources

Mon, 27 Jun 2022 08:26:01 +0000

Creating custom visuals using the Charticulator visual in Power BI is not 100% intuitive. So here are some links to some Charticulator resources I’ve used to skill up and some resources I’ve created.

Microsoft’s Resources

Charticulator was created by Microsoft Research and is Open Source. So they have some great Charticulator resources to help you get started.

Charticulator Website – https://charticulator.com/
Microsoft’s Announcement – https://powerbi.microsoft.com/en-us/blog/announcing-the-new-charticulator-visual-public-preview/
Microsoft’s GitHub of Resources – https://github.com/microsoft/charticulator
Research Team Video – https://www.youtube.com/watch?v=5EZq4QolNy4

Blogs and Videos

One great resource to help me get started was the tutorials by Burning Suit. They’ve even written an brilliant book on the topic. Also other content creators have videos and blogs all with some great ideas and different ways to get you started.

BurningSuit – https://www.burningsuit.co.uk/charticulator-in-power-bi-1/
And their book – https://www.burningsuit.co.uk/our-book-introducing-charticulator-for-power-bi/
Paul Turley’s Slide Deck – https://sqlserverbi.blog/wp-content/uploads/2019/09/can-you-say-charticulator-paul-turley.pdf
Enterprise DNA Video – https://www.youtube.com/watch?v=ORKTLUAFXnI
RADACAD’s Introduction Video – https://www.youtube.com/watch?v=xtyxTNnEvnw

My Charticulator Resources

How I learn new topics is by creating content to share. So I’ve created a blog post and a few videos. I need to create more and they are coming.

Simple Chart Post – https://hatfullofdata.blog/charticulator-simple-custom-chart/
Simple Chart Video – https://www.youtube.com/watch?v=G7YtRPnoc7M
Grid Chart – https://www.youtube.com/watch?v=Hb56crKEJN4
Heat Map – https://www.youtube.com/watch?v=kz2b1tchFJo

Conclusion

After reading this, please let me know if this was useful. Then please point out the links I’ve missed out. There are other great custom visual options out there and Charticulator has its place amongst the great ones.

Power BI Charticulator Simple Custom Chart

Fri, 07 Jan 2022 10:24:29 +0000

In late 2021, the Charticulator visual was added to Power BI desktop. This post walks through creating a Charticulator Simple Custom Chart. The chart will show the drink preference split of 20 office colleagues. The steps assume you already have connected to the data. The data is available for download or direct connection here:

https://hatfullofdata.blog/wp-content/uploads/2021/12/Drinks.xlsx

Charticulator Series

This series will build a number of different visuals using the Charticulator visual in Power BI desktop. Each post will include links to data sources, sample Power BI reports and any related YouTube videos.

Simple Chart
Exporting and Importing Templates
Allowing Export of Templates

YouTube Video

It’s coming!

Adding the Charticulator Visual

The Charticulator visual is a custom visual so needs to be added. In the visualisations pane, click on the three dots and then click on Get more visuals. Search for Charticulator. Then click on the icon to see the full description. Click Add to add the visual to this report.

Start creating Charticulator Simple Custom Chart

Add the Charticulator visual to a report page. This visual needs the count of the people who prefer a drink. So I created a measure that counts the rows.

Next I add Drink field and People Count measure to the Data of the visual. Once you have data added you can create the visual. So click on the three dots on the visual and select edit.

When the Charticulator pane appears, click Create Chart. Then the Charticulator editor window appears.

Creating Charticulator Simple Custom Chart

The Charticulator editor window has different areas. We are going to start by working with the Glyph area to define the shapes that will appear in the plot area.

Adding Shapes

The chart is made of a rectangle per drink with a width that corresponds to the number of people who prefer that drink. From the toolbar make sure the Marks button shows a rectangle and then drag the rectangle into the glyph area. The glyph area should now have a rectangle in it and the plot area 3 rectangles, one for each row of data from the three drink types.

The 3 rectangles in the plot area are all the same size. We want the width to be linked to the people count value. Make sure you still have the shape selected. Then click on the link button next the Width box and select People Count. The plot area rectangles will now be different widths.

Next we want the rectangles to be different colours. Further down the attributes pane there is a Fill property. Click on the link and select Drink. You can change the colours here as well if you want to.

The final step before me move on to text is to remove the gap between the rectangles. This is a property of Plot Segment. Start by selecting the PlotSegment1 in the Layers. Then scroll down the Attributes to find Gap. Reduce the Gap value down to 0.

Adding Text

A good alternative to adding a legend to a chart is to add labels. We are going to add 2 text objects to show the drink name and the people count. Drag a text item, show by A, into the glyph area. Keeping Text1 selected, we can update the attributes. Make the following changes

Link the text to Drink
Increase the Font size if required
Change the Anchor Y to be bottom
Change the color to be white

Note you can click on Glyph title to minimise that section of the pane to see more Attributes.

Add another text object. Make very similar changes except link the Text to the People Count and the Anchor Y link to be Top so that the number is under the drink.

Formatting Numbers

When you link a numeric column to a text object the calculation includes the formatting for the number. In the example below the second pair of {} includes the formatting. f = fixed point notation and the .1 means show 1 decimal place, so {.2f} would show 2 decimal places etc.

1

${avg(`People Count`)}{.1f}

You can change this to {.0f} to show no decimal places or you can use {d} which is decimal notation, rounded to integer. This notation is called d3 and more information can be found here.

https://github.com/d3/d3-format#format

Changing Plot Size

By default the chart is 600 x 400. This would make the chart very square. Click on the Chart layer and reduce the height. You can also reduce the margins to make the chart fill more of the area.

Finishing the Charticulator Simple Custom Chart

Finish editing the chart by clicking on Back to report. Click on Save to save your changes to the chart. Be aware as you re-size the chart the font size is fixed.

Conclusion

This is the first post in a series on creating charts using the Charticulator visual. I will be looking at creating templates and how to create custom visuals. On the bottom of every post I will be including useful references. If I’m missing any please reach out

Resources

Power BI Series

Refreshing Datasets Automatically with Power BI Dataflows

Sun, 29 Aug 2021 18:58:12 +0000

I have a report, Open Issues, that has a Power BI dataflow as a data source. If I just use scheduled refreshes, my options are limited. I need to be refreshing datasets automatically after the dataflow has finished so my reports are up to date as soon as possible to avoid a situation where I schedule the dataflow to refresh at 10am and the next slot available might be 10:30am, for example. This can be done using Power Automate.

Create a flow in Power Automate

Head to Power Automate at https://flow.microsoft.com/and click on Create on the left hand side menu. The flow will be triggered by the dataflow refresh finishing. This means it is automated so click on Automated cloud flow.

A dialog appears asking for details of the flow. You need to enter in a name for the flow. Then in the search box enter in power to find relevant triggers. Select When a dataflow refresh completes and click Create.

After you click Create the flow editor appears with the trigger already added. For Group Type select Workspace. Then you can select the workspace name for Group. Then you can select the Dataflow.

We then need to refresh the connected report dataset. So we click New step. In the Choose an operation dialog, we type dataset into the search box. This will find you the action Refresh a dataset. We click on the action to add it to the flow.

Then you need to select the Workspace and Dataset in the action. The flow is complete, so we can click Save.

Testing the Flow

The flow triggers when the dataflow finishes refreshing. If we refresh the dataflow that will test the flow. I clicked refresh on the dataflow at 3pm and it triggered the flow which then refreshed the dataset.

In the workspace we can see the date and time of the refreshes and if the flow has worked correctly the dataset should be refreshed just after the dataflow. If you look at the history of the flow runs there should be a matching flow run just after the dataflow finished refreshing.

When refreshing datasets automatically fails

It is worth pointing out that if the dataset refresh fails the flow will not reflect that and will show succeeded in the flow runs. So it is worth checking that the dataset can refresh successfully.

Conclusion

This idea can be extended to refresh multiple datasets over multiple workspaces that are connected to the dataflow. This simplifies the process of scheduling refreshes. It also could allow for notifications to interested report owners to be told of the refreshes etc.

Power BI Series

How much does that report cost the business?

Tue, 06 Jul 2021 15:04:41 +0000

In nearly every business across the world, staff give reports to managers. How many people assess the report cost? The reports will show how well or badly the business is performing. Accurate reporting is important in every organisation. Management needs to be able to make data-driven decisions.

Every report comes with a cost. Is the report worth that cost? Could there be improvements in the process to lower the report cost? The biggest cost is probably the effort from the report builders. In this post I am not covering how to make a report useful, I am just looking at the report cost.

When I teach intro to Power BI I use a diagram to show the effort split. Far too often getting access to the data and transforming it into a structure you can model takes up more time than anything else.

Report Building Effort

In an ideal world the above diagram would be wrong. Anyone can access the data they need, it is all easy and quick and the data is well structured so only needs the very basic transformations. If that is the case, superb, well done. If you are guessing that is the case, please do go and ask your report builders, you might be surprised at the effort they put in.

Get

In so many companies the above diagram is accurate and it doesn’t have to be. So let us look at the “Get” stage. Often the process of getting access to the data is hard work and involves politics galore and the bigger the company the worse it is.

Is there a process of requesting access to company data? Is there a self-service data option? Are the limitations on who is allowed to connect properly governed? Do your staff even know how to find the required raw data?

In one company there were 2 options, option one pay a yearly £10K per daily csv extract or option two extract the data in 10,000 rows blocks manually once you’ve worked out the queries you need to run. The option of connecting to an actual database wasn’t even an option. Its not a huge surprise they picked the 10,000 block extract option. So there were hours wasted every week of people extracting many blocks of data and sticking it back together. The hours were being done by highly skilled engineers who had better things to do.

In another company the report builder wasn’t actually allowed access to the data but was asked to create a report. So shared screens, remote control and lots of faff and a report was built, every change involved the same games. All in the name of security.

When I suggested the engineers time was worth more than 10K or pointed out the security was already broken, they shrugged their shoulders. No-one thought the management would or could do anything.

Transform

Once you have access to the data, how much transformation is required to make the data usable?

Let us start with a story. In this company big bookings are done on accounts, so the account needs to be recorded for discounts to be applied and account managers to get commission. The box to fill in the account name is free text and the business locations are across the world so the names are foreign to some. You can imagine the variations in spelling. So, every month the reporting team spend days refining their transformation routines to cope with new variations. Account managers spend hours doing multiple checks to make sure they have all their account booking assigned to them.

All it needed was a drop down box. The booking system design team and the reporting team were in no way connected except by very senior management and no-one was telling them. So every month the hours were wasted and the account managers stressed. Before you assume this is a small company in 2019 their turnover was 21 billion dollars.

This is one small example, I could bore you with hundreds of stories as could many of the data community. Badly captured data, extracts that are impossible to work with, constantly changing data structures are daily problems for report builders and often its due to a lack of big picture planning and people un-aware that data is used in a report.

It only takes 20 minutes

My final story is from a company which had daily stand-ups on every production line. They were an important part of how that business functioned. There were a few charts produced daily and when I asked about them the reply came back “It only takes 20 minutes”.

So over the 5 day week that’s 100 minutes which isn’t bad and was easily absorbed. Now take a step back. That factory site had 12 production lines, now that becomes 1200 minutes, which is 20 hours. That is half a week. The reports were almost identical except filtered to a different production line, but they were all built separately and slightly different so not 100% comparable. Then you find out rolled up versions of the report were produced for every level of management.

Across that production site I worked out there was roughly 0.75 of a person being used to build that report. Was that a good use of that headcount?

Telling the management the report cost

I was the external consultant being brought in to help. It was often assumed that the problem was a lack of skills. The problem was usually a lack of communication and an assumption that management wouldn’t fix it. When I had the opportunity to ignore the internal politics and talk to the senior management, things often got fixed. Even the grumpiest manager cares about time being wasted on pointless tasks. Sometimes it takes an external consultant to point out the obvious and reduce the report costs.

So managers please do the following :

Look at the reports, if you don’t need them say so.
If reports need changes, please say so. People like to hear changes because it means you value their work.
Find your reports builders, no not their manager, find the actual builders. Ask if there is anything that could make it easier that you could initiate. Ask what their biggest challenge is, could you fix it?
Is work being repeated? Do three of your team give you the same report just filtered differently? Perhaps you could facilitate a joint effort.

And report builders please do the following :

Record the actual time it takes, including all the extras you forget and the problems you had last month.
Talk to your report readers and ask for feedback, let them know the challenges.
If ignored, brush up your CV.

Last Word

The stories given in this post are just a very small sample of the issues I have seen that increase report cost. The solutions to fix many of the boring problems were not complex, they just required someone to to step back, ignore company politics and look at the bigger picture.

There are plenty of reporting challenges that cannot be fixed easily, that is where skilled data engineers and report builders do the magic they do. Copying and pasting data from one system to another or pressing that extract button every 3 minutes for 2 hours or manual typing in the list of 1000 exchange rates for this month is not that magic they love. Someone will offer them a better job with more chances to do data magic, and now your report cost just got much higher.

Power BI Series

Power Query – Get Previous Row Data

Tue, 08 Jun 2021 08:55:16 +0000

Occasionally I get asked a question via Twitter or LinkedIn that prompts me to write a post. The question asked was how to get previous row data. In this post I will describe how to get a value from the previous row that matches a criteria. The example data is peoples scores on different dates. I want to know if they are improving. For this I need their previous score.

Starting Data

Getting Previous Row Data

The first step is to sort the data. You need the previous score to be in the previous row, so we sort by name and then date.

Then from the Add Column ribbon we add an Index column. And then a custom column, “Prev Index” which is Index-1

Now we can merge a table to itself (yes a touch head screwy but its fine it works) by linking Prev Index to Index. This though will give Bob Alice’s last score as his previous score so we add the Name column to the merge as well. Holding down the Ctrl key will allow you to select multiple columns and make sure at the bottom of the dialog you have some matching rows..

This creates a column that contains tables. Press the expand button and only select the Score column. Click OK to give a column of the previous scores ready to rename.

I then rename Score.1 to Previous Score and use Choose Columns to remove Index and Previous Index to end up with a table of name, date, score and previous score.

Power Query – Dynamic Data Source and Web.Contents()

Tue, 04 May 2021 13:01:31 +0000

This is the first post in a series to create a Star Wars report. This post handles the issue of using a parameter as the url in a web Power Query. This type of data source is referred to as a Dynamic Data Source. It will not refresh in the Power BI service unless you do some tricks.

Star Wars Series

Starting on May 4th a series using Star Wars data had to be done. Its based off an api that has some great data that is paged in 10 row blocks. I’m looking to create complete report from queries to modelling to visuals and theme etc.

The data comes from https://swapi.dev/api/

Handling Dynamic Data sources
Fetching number of pages
Using next page url

Fetching Data

I started by looking at the people table. In a new Power BI report I select Web from Get Data and use the url https://swapi.dev/api/people/ and click OK. This throws you straight into Power Query where is has created a few steps to get you a table of data.

It only gives 10 rows of data, which does include the url to the next 10 rows and also how many rows there are in total. So I use a technique I’ve blogged about before and create a URL parameter, change the source step to use the URL and then convert the query into a function.

Look here for more details https://hatfullofdata.blog/power-query-fetch-web-data/

Test Refresh

Then as a test I invoke the function to fetch the first 10 rows into a table, close and apply Power Query and then publish my report to the Power BI service.

Then I do the important test, does it refresh? When I click on refresh it fails, and when I click on the small red triangle it gives an error stating the dataset uses a dynamic data source and these can’t be refreshed.

Part of the error includes a URL https://aka.ms/dynamic-data-sources , so I go and look at the web page. And it explains how you can identify if you have a dynamic data source that won’t refresh by looking in the Data source settings in Power Query. Sure enough, when I look, it gives a message which I now know means a dynamic data source so refresh won’t work. I assume hand-authored queries means dynamic data sources.

This doesn’t tell me how to to fix it though. So off onto the internet I go and search around my favourite Power Query bloggers and sure enough Chris Webb comes up trumps with a few posts.

The second post was the one I used the most in modifying my query. It also includes searching for cow data, which I think is superb!

Removing the Dynamic Data Source

So the Web.Contents used in the source of my function needs the base url to stay the same and then any dynamic parts put into the options. The base url must be a valid web address you can authenticate to as anonymous and the shorter the better as it makes the function more flexible. I delete the invoked function query and the function from my report.

So for the Star Wars data the base url is – https://swapi.dev/api

The people part and the page=1 can be added in as part of the options. So the new solution has 2 parameters, TableName and PageNum and they are both text. Then we have to edit the source step by hand to write the second parameter of Web.Con (the whole hand-crafted fix to get rid of hand-crafted error is just amusing). The options are a record so put inside square brackets [ .. ]. The query is another record so is a nested [ .. ].

Web.Contents options allow for two things to be set, a relative path, eg People and query parameters eg Page=1. APIs are fussy so make sure you get the case right in the query part. This is my final code for the Source step. (This is the reason I blog, so in 6 months time I know I have a working example piece of code.)

1
2
3
4
5
6


Source = Json.Document(Web.Contents("https://swapi.dev/api/",
 [
 RelativePath=TableName,
 Query=[page=PageNum]
 ]
 )),

Then I right click on the query and create a function. Invoking this function creates a table ready now to close and apply back into Power BI desktop. I publish and try a refresh and YES! it works. I now have a function that will take a table name and page number and give 10 rows of data.

In the next post we will use this function to get all the data for a table.

Power Query – VBA to Edit a Parameter Value

Wed, 21 Apr 2021 17:04:16 +0000

This post is to document how to use Excel VBA to edit a parameter value without using the a cell reference in Power Query. My query was using Web.Contents which works with parameters in Excel Power Query1 but doesn’t like a function as part of the path.

I started with Chris Webb’s great post on using an Excel named range and Web.Contents gave me problems. I would try Chris Webb’s method first which is found here https://blog.crossjoin.co.uk/2014/07/22/working-with-excel-named-ranges-in-power-query/

So I want a button to take the value from a cell and update the value in a parameter. I searched the web and found one solution buried in a MrExcel forum so I’m documenting the solution here mostly so I can find it again later.

VBA Code to Edit a Parameter Value

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


Sub ChangeParameterValue(ParameterName As String, ParameterValue As String)

 Dim qry As WorkbookQuery
 Dim formula As Variant
 
 '=== Get the query
 Set qry = ThisWorkbook.Queries(ParameterName)
 
 '=== Split the formula into 3 parts and update the second one
 formula = Split(qry.formula, Chr(34), 3)
 formula(1) = ParameterValue
 
 '=== Update the parameter value
 qry.formula = Join(formula, Chr(34))
 
End Sub

Description

The formula property of a query is really weird and hard to construct so the above function splits on Chr(34), which is a “. It then updates the middle value to the new value and then sticks the three values back together again.

The original forum post I got the solution from can be found at https://www.mrexcel.com/board/threads/vba-code-to-edit-power-query-data-source-settings.1146964/

Footnotes

1 – Web.Contents varies in different versions of Power Query so test carefully and do not assume you can copy and paste queries.

Power BI Desktop Connecting to Dataverse

Wed, 07 Apr 2021 08:46:05 +0000

Dataverse is a secure flexible data storage solution that is part of the very popular Microsoft Power Platform family. So connecting to Dataverse is an obvious request. Dataverse was previously known as Common Data Service. This post is from getting very frustrated mid-hackathon and trying to connect to Dataverse and just getting an odd error.

If you want to learn more about Dataverse you can find more information at Microsoft Dataverse documentation

Finding the Environment Domain

To connect to Dataverse you need the environment domain. This can be found by going to Power Apps. Then open a model driven app and look at the address bar in your browser. The domain is the XXXXX.crmX.dynamics.com part.

Connecting to Dataverse

In Power BI desktop Dataverse is one of the connectors found on the Home ribbon and found in the drop down under Get Data. If you are looking in the dialog it comes under Power Platform. Here is where you put the Environment domain of your Dataverse.

Use the domain without the https:// and without the / after it. If this is your first time connecting it will ask you to login and then will offer you the list of tables available in your Dataverse environment. I strongly advise transforming the tables to remove the columns you don’t need from your Dataverse tables.

You can connect to Dataverse using DirectQuery if you must. Often its not required and makes the report slow to edit. Yes I have opinions 😊.

Weird Error

If you include the https:// or the final / or any other extras in the environment domain it will take ages to try and connect and then will come back with an error. The error refers to trying to connect to SQL server which will confuse users.

Power BI Series

Paginated Report Resources

Sun, 21 Mar 2021 18:39:32 +0000

Important Update!

On 14th Nov 2022 Microsoft announced that Paginated reports will no longer require premium capacity and can be created with a Power BI Pro license.

Here is a link to the article https://powerbi.microsoft.com/en-us/blog/announcing-support-for-paginated-reports-in-power-bi-pro/

Introduction

Here are resources to help you learn how to use Paginated reports. You will need a premium capacity or Premium per user license. If you have any extra resources or questions please reach out to me on Twitter @Laura_GB.

Install Report Builder

Install from the Microsoft Store if you can or download

https://www.microsoft.com/en-us/download/details.aspx?id=58158

Power BI Series

Power BI – Reactor – Last Year’s Sales

Sun, 31 Jan 2021 22:49:25 +0000

This post is to support my virtual session at London Microsoft Reactor on 1st Feb 2021. The session is an introduction to Power BI using sales data for 2020 from an imaginary gift shops.

YouTube Recording

Resources

The Excel file used as the data can be found at:https://hatfullofdata.blog/wp-content/uploads/2021/01/Sales.xlsx

The background image used in the demo can be found at:https://hatfullofdata.blog/wp-content/uploads/2021/01/Gift-Background.png

The page from SQLBI used to create the calendar table can be found at:https://www.sqlbi.com/articles/creating-a-simple-date-table-in-dax/

Useful Resources Mentioned in the Session

Create a calendar in Power Query by Radacad.https://radacad.com/create-a-date-dimension-in-power-bi-in-4-steps-step-1-calendar-columns

Guy in a Cubehttps://www.youtube.com/channel/UCFp1vaKzpfvoGai0vE5VJ0w

Chris Hamill’s Backgroundshttps://alluringbi.com/2020/05/05/themeable-backgrounds-for-power-bi/

Next Session

The next session will be 22nd Feb 2021, when we will look at UK population data. Closer to the date you’ll be able to find joining instructions at https://www.meetup.com/Microsoft-Reactor-London/

London Reactor Posts

Power Query – AI Insights Error

Fri, 15 Jan 2021 22:09:52 +0000

AI Insights in Power Query is a great Premium feature in Power BI. An AI Insights Error can occur if you are not fully aware of the settings you need to check. This foxed me for 2+ hours today. So I hope this post helps someone not waste the 2 hours like me.

From the Add Columns ribbon I selected Text Analytics and then in the Text Analytics I selected Extract key phrases, but this is true for any of the AI functions.

It thought for a while, and I assumed it was processing. And then it gave this error. I tried every combination of what I could do to get rid of the error.

The issue is when you do an AI function it creates a data source with a Public privacy level. So if your data that you are analysing has a different privacy level you get the error above.

Fixing Privacy Levels

This issue has a really quick fix once you realise what it is caused by. On the home ribbon in Power Query, click on Data source setting. This opens the Data Source setting dialog box.

Click on each data source and click on Edit Permissions. Change the privacy level to be the same, I use Organizational usually.

When you have checked all your data sources you can click close in the Data Source Settings dialog and refresh your data to get rid of the error.

Conclusion on AI Insights Errors

This is a classic case of when you know the cause it’s easy to fix. I hope it helped.

Power BI – Reactor – Christmas Carol Text analysis

Mon, 14 Dec 2020 14:58:39 +0000

This post is to support my session at London Microsoft Reactor. The session series have all been introduction to Power BI. This session uses some Christmas Carol song lyrics and then using some AI premium tools to analyse the text.

YouTube Recording

Resources

The Excel file used as the data can be found athttps://hatfullofdata.blog/wp-content/uploads/2020/12/Christmas-Carols.xlsx

The background image and theme file can be found athttps://hatfullofdata.blog/wp-content/uploads/2020/12/Christmas-Carol-Background.png https://hatfullofdata.blog/wp-content/uploads/2020/12/Christmas-Carol-Theme.zip

Theme file is zipped for security reasons (WordPress had its reasons apparently).

London Reactor Posts

Power BI Dataflow – Create Dataflow from Export

Sat, 28 Nov 2020 23:15:04 +0000

There are times when you need to copy a dataflow from one workspace to another workspace. Power BI service provides a simple way to export the definition as a json file and then import.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Export dataflow as json file

In the workspace of the original workspace, click on the three dots next to the dataflow and select Export json. When the file is ready a message will appear in the top right.

Create dataflow from json file

In the destination workspace click on the New dropdown and select dataflow. From the options then displayed select Import Model. Select a your json file. Once the json file has imported a message will appear in the top right.

Although your dataflow has been created you are left on the dataflow creation page. You need to return to the workspace to see your new dataflow.

Check Credentials

The dataflow will not have refreshes setup as that is not part of the model. It is also recommended that you confirm that the credentials are correctly loaded. If you the dataflow is from another user or another tenancy you will need to re-login each credential.

To check the credentials you need to click on the three dots next to your new dataflow and select Settings. In settings, expand Data source credentials. If any of the credentials have a cross next to them click on Edit credentials to re-login.

Conclusion

Part of my work is to build dataflows to assist people fathom the depths of Microsoft Project data. The ability to export a dataflow means I have been able to transfer a solution between different teams across the company.

Power BI Series

Power BI Dataflow – Refresh History

Mon, 23 Nov 2020 16:20:50 +0000

You can find out which entity took the longest to refresh or which entity caused the error in a failed refresh by looking at the refresh history of your dataflow in the Power BI workspace.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Refresh History List

In the workspace, click on the three dots next to the dataflow and from the menu select Refresh History. This will open a dialog that has a row for every refresh.

Every refresh will show when it happened, how long it took and if it completed or failed. In the above image we can see the last 2 refreshes failed but the 4 previous refreshes completed.

Refresh History Files

The details of the refresh for each entity that loads from the dataflow can be found by downloading the csv file. You can download the csv file for a refresh by clicking on the down arrow on the right of each refresh row.

Calculating the Refresh Time

Excel will happily open the CSV file. The first column might say Invalid Date this is due to the Power BI service trying to treat a dd/mm/yyyy formatted date as mm/dd/yyyy. I have reported the issue to Microsoft.

A simple calculation of (End Time – Start Time) will give you the duration in days so in this example I then multiplied it 246060 to give seconds. Adjust the calculation to minutes or hours to fit your durations.

Showing Errors

If you download the file of a failed refresh you will either get a single line why the whole refresh failed or details of which entity failed. there are occasions when the error is not clear, for example the second one shown.

In the first example the sales entity has failed due to a problem with the “Qty” column. This causes the other entities’ refreshes to be cancelled.

Conclusion

The Refresh History is an important part of understanding how a dataflow is performing. Hopefully at some point there will be an API to fetch the history directly, last time I looked there wasn’t.

Power BI Series

Power BI Dataflow – New Diagram View

Sat, 21 Nov 2020 22:32:08 +0000

In November 2020 a new diagram view was introduced to the online Power Query editor. The new feature produces a brilliant diagram of how the queries and parameters relate within the dataflow.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Displaying Diagram View

When editing your dataflow, click on the View ribbon tab and select Diagram View. The diagram view will appear between the ribbon and the formula bar.

Object Types Shown in Diagram View

The entities with a blue border are loaded entities, i.e. listed when connecting and the entities with a grey border are not loaded, so they refresh but are not exposed to the report builder. Parameters and functions have a relevant symbols in the top left and a grey border.

Query Steps Preview

If you hover your mouse pointer over the step counter, it will display a list of the query steps.

Expanding Display View boxes

In the top right corner of each query and function box there is a double-headed arrow button. Click this button to expand the query and show all the steps. The button is a toggle to switch between collapsed and expanded views.

If multiple queries are expanded, you can collapse all query boxes by using the toolbar in the bottom right of the canvas. It has collapse all and expand all buttons.

Displaying Step Names

By default an expanded query displays step labels, i.e. the name of the action being applied. You can display the names of the steps by clicking on arrow below Diagram view button and selecting Show step names.

Related queries can be highlighted using the model diagram button in the top right of the query box. It will highlight all related objects blue or grey based on their borders. In a complex dataflow this can be useful to understand the complexities.

Adding Steps

When a query is selected and expanded it displays a plus sign. Clicking on the plus button displays a list of possible steps to add, it will be based on the column you have selected. There is a search box to assist in finding the right step.

Clicking on the three dots in the top right hand corner offer multiple query steps to add to the query plus some other options.

Conclusion

Diagram view is a great addition to the online dataflow editor. Diagrams are a great way to explain a structure and this should help explain transforms within a dataflow. Personally I doubt I will be using the plus button to add steps, but then I am not the target audience.

Power BI Series

Power BI Dataflows – Endorsement as Promoted and Certified

Sat, 21 Nov 2020 20:55:18 +0000

Inform your users as to which dataflow to use by using Endorsement. This allows you to label a dataflow as promoted or certified.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Before Endorsement

When a report builder selects dataflow as the data source, the list of dataflows only shows the title of the dataflows. It gives no indication of the last refresh date or owner so if there are similar names your report builders could easily pick the incorrect dataflow.

In the above example there is Biscuits and Biscuit Data and no information to help a report builder decide which dataflow is the correct one.

Adding Endorsement

Endorsement has to happen in workspace. Click on the three dots next to the dataflow and select Settings. In settings, expand the Endorsement section to see three options.

None – default setting with no endorsement.
Promoted – this level can be added by the dataflow owner.
Certified – this level can only be added by owners who have been given permission by the tenancy admins. Details of how an admin can add be found in Endorse a dataset post.

Once you have selected the level, click Apply to save it.

After Endorsement

Now when a user selects dataflow as their connector they will see the same list of dataflows but they will be tagged as Promoted or Certified.

Conclusion

When someone has done the hard work to create a dataflow to assist the users in building reports it is important to let your users know which dataflow is the one to use.

Power BI Series

Power BI – London Reactor – Tea & Biscuits Session

Sun, 15 Nov 2020 23:07:36 +0000

This post is to support my virtual introduction to Power BI session for London Reactor on 16th November 2020.

Resources

Tea Production – https://teahow.com/where-is-tea-grown/Favourite Biscuit – https://hatfullofdata.blog/wp-content/uploads/2020/11/Biscuit-Data.xlsx

The biscuit data is fictitious using randomly generated names and random selection of biscuits.

Session Recording

London Reactor Posts

Power BI – Scheduled Refresh for your Dataflow

Mon, 09 Nov 2020 17:45:23 +0000

Most dataflows needs to have scheduled refresh setup so that the data will be up to date ready for the reports connecting to it. This post walks through the basic steps to set up the refresh.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Setup Scheduled Refresh

In the workspace containing the dataflow, there is no Next Refresh on the dataflow. Therefore the dataflow will not refresh automatically.

Click on the three dots on the dataflow row, they appear when your mouse hovers over the row. Select Settings from the options.

Expand Scheduled refresh.
Click on the toggle to turn on scheduled refresh.
Select refresh frequency, daily means every day, weekly means you can select which day of the week.
Select your time zone.
If you have selected weekly, select days.
Click on Add another time and enter the hour, minutes and am or pm for every time you want the refresh to happen.
Leave the tick so you you will be notified if the dataflow refresh fails.
Click Apply.

Once the above has been completed, we can now return to the workspace. The dataflow now has a Next Refresh date and time for next Wednesday at 8:30am.

Conclusion

It always fascinates me how people are not confident in setting up the refreshes of datasets and dataflows. Hopefully this quick simple guide will give enough confidence to setup the refreshes to happen without people logging into Power BI to refresh the dataset or dataflow manually.

Power BI Series

Power BI – Create a Dataflow for Reusable Transforms

Tue, 03 Nov 2020 14:18:01 +0000

This post is a quick guide to create a dataflow and connect to a dataflow. Dataflows are a great way to give your report writers reusable data source that includes any complex transformations required and doesn’t require unique data logins for every report writer. It just requires access to the workspace that contains your dataflow.

Dataflow Series

This post is part of a series on dataflows.

YouTube Version

Create a Dataflow

Staring in the workspace online in the Power BI service, click on New and then select Dataflow.

In the next screen click on Add new entities button to start creating your dataflow.

The next screen lists all the data sources supported for dataflows. Select your data source. There is a search box in the top right if required. In this example I selected SQL Server database, although I could have picked Azure SQL Server.

As soon as you click in the data source, you will be prompted to enter in connection settings. When complete click Next to continue.

You will then be prompted to select the tables or views to add to the dataflow. You can see previews if that helps. When ready select Transform data button.

This will open an online version of Power Query. Add the transforms you need to each query. When you have finished click Save & close to finish creating your dataflow.

It will then prompt you for a name for your dataflow and an optional description.

Refresh the Dataflow

Although you saw data in the dataflow as you were building it, its not there until you refresh. You get a message for a short while when you save it asking if you want to refresh in the top right hand corner. If you miss that you can return to the workspace and when you hover your mouse point over the dataflow the refresh icon appears.

Connect to a Dataflow

A report, in Power BI desktop, can connect to the dataflow as a data source. Click on Get Data to open the dialog, click Power Platform to filter the options and then select Power BI dataflows. Click Connect to start the connection to dataflows.

When the Navigator dialog appears, expand the correct workspace and dataflow to select the tables. Click Load to load the data into your report. You can click transform to add extra transformations into your data if required.

Note on Refreshes

If a report is based on a dataflow, the dataflow needs to refresh before the report refreshes. Care needs to be taken on setting refreshes to co-ordinate correctly.

Conclusion

Dataflows are a great part of the data reporting structure that can reduce the hit on databases, reduce the number of accounts accessing the data and give the report writers clean well structured data to build their reports on.

Power BI Series

Power BI – Reactor – Use AI Visuals to Analyse Titanic Data

Mon, 12 Oct 2020 14:50:28 +0000

This is a post to support my session at the Reactor on 12th October 2020. The session was aimed at newcomers to Power BI and the AI visuals. Below is a link to the video on Reactor’s YouTube channel

Data Sources

The session uses 2 data sources, the passenger list for the Titanic and an Excel file to add some extra information such as port locations.

https://hatfullofdata.blog/wp-content/uploads/2020/10/TitanicLocations.xlsx

https://hatfullofdata.blog/wp-content/uploads/2020/10/titanicdata.csv

Transformations

The passenger list data requires some transformations to make it easier to work with. The two main skills demo’d are Column by Example and Selecting columns. These are covered in these 2 videos.

Page Background

In order to make page layout easy I use page backgrounds created in PowerPoint. This idea I got from Chris Hammill’s brilliant blog post found at https://alluringbi.com/2019/10/21/background-concepts-for-power-bi/

AI Visuals

In this demo I use 3 AI visuals Key Influencers, Decomposition Tree and Smart Narrative. Here are videos to help you get started with those visuals.

London Reactor Posts

Power BI – Reactor – Weather Report Resources

Wed, 02 Sep 2020 21:43:14 +0000

Thank you everyone who attended my virtual session at the London Reactor on Wednesday. Here are links to resources I used for the presentation.

Weather Data

The met office very usefully provides data in a text format for stations across the UK. The first link is to the summary page that gives a map link to all the stations. The second link is to Whitby’s station data.

https://www.metoffice.gov.uk/research/climate/maps-and-data/historic-station-data

https://www.metoffice.gov.uk/pub/data/weather/uk/climate/stationdata/whitbydata.txt

I have typed up the list of station with Latitude and Longitude values into an Excel file which can be accessed here

Background

Chris Hamill has written a great post on designing backgrounds for reports that make a report load quicker and make it easier to read.

https://alluringbi.com/2019/10/21/background-concepts-for-power-bi/

Calendar

SQL BI team have a huge range of videos and blog posts on many DAX topics. The example we used in this session was creating a simple calendar using code from this page..

https://www.sqlbi.com/articles/creating-a-simple-date-table-in-dax/

Turn Off Time Intelligence.

Another great source of educational material on Power BI is is Guy in a Cube. One recent video explains the reason to turn off the Time Intelligence.

Drill Through Button

During the presentation a question was asked regarding drill through. I mentioned the possibility of using a drill through button but I did not have enough time to demonstrate. Here is a video to explain the feature.

Resources

The final Power BI report files with the other resources can be found here.

YouTube

As soon as the Reactor posts the video the link will be added here.

London Reactor Posts

Power BI – Import Text Using Examples

Sat, 22 Aug 2020 22:22:45 +0000

Import text using examples was introduced in the 2020 August update for Power BI. It writes the Power Query steps to transform a non-tabular text file into a table.

YouTube Version

https://youtu.be/6h7g-CwkeVg

Enabling the Feature

At the point of writing this blog post this feature is still in preview. The feature is enabled in File – Options in Preview features. It will like most features move to being enabled for everyone at some point.

Example Import text using Examples

For this example I have a very simple text file as shown in first picture in this post. It contains the details of journeys on different days. So when I select Get Data – Text/CSV and select the file, the dialog that shows a preview of the data in the file has a new button.

When the button is clicked a new dialog opens a dialog box with a data preview showing and an empty table. This means we need to enter in some answers into the table so a pattern can be calculated how to transform the data. When you start to type in a value it will suggest possible values.

A column is renamed by double clicking on the column title and entering a new name.

With only one value it cannot calculate the pattern required. When I have added more values it can determine the pattern. It often needs less values in future columns. In this example 2 dates were needed for the first column and then only one value for the remaining columns.

In the above example the Distance column was only part of the row of data. It calculates the ways to split the data. After this and all the columns have been added, we click Transform Data to see the Power Query steps that the import feature have been written.

Looking at the Power Query

So the magic behind this new feature is it writes a 10+ step query for you with a pretty good pattern for doing this. So the 5 steps after the source are working out how many rows make a block of data and grouping that data into one block. The next chunk of steps are splitting that block into separate columns.

How does import text using example decide the block length?

This for me is the clever part. Step three where it adds the conditional column is the crucial step. So I looked at three slightly different versions of the text file to see how it changes.

Example 1

In this example each of the blocks start with a date, which means the line starts with a number. Step 3 uses a test to see if in position 0 there is a number. So for this example, we could have as many rows as we liked in a block as long as the first row is the only row starting with a number.

Example 2

In this example the blocks start with the word “Date”. Step 3 looks for the line starting with “Date” as the first line. This is pattern is a great one as it works fine for different length blocks as long as the only lines starting with “Date”.

Example 3

In this example there is no easy way to identify the top row of the data. This means the only pattern it can determine is by length of the block so it guesses 4. This works if the data is in same length blocks.

Conclusion

I like the new import text using examples feature and I can see places where it will get used. I hope it gets progressed to recognise more patterns and be more flexible. When it works it stops the report writer needing to put together the Power Query steps to do the transform.

Power BI Series

Power BI – Q&A Synonyms Update

Sat, 15 Aug 2020 17:53:05 +0000

Q&A is a great tool for report readers to ask questions regarding the data behind the report. The natural language interpretation is improving all the time and can be improved by using Q&A synonyms. The July 2020 Power BI update added a new way to add synonyms to a report.

YouTube Version

Without Q&A Synonyms

On report I have added a Q&A button to allow a report reader to open a dialog to ask a question. Different people will use different terms for the same thing. A word that is not recognised will be underlined in red.

Adding a Q&A Synonym by a rename

A synonym is an alternative word with the same meaning. In the example report the table is called Products with a Product Name column, but the products are toys. Therefore “Toy” in the context this report is a synonym of “Product”.

The July 2020 update introduced a new method to add a synonym to the report. When a field is renamed in the visualisation pane a suggested synonym term is created.

With the suggested synonym term created the natural language interpretation will now understand Toy means Product Name. When the question is asked using Toy the question using Product Name will be suggested.

After a few moments it will quickly show you a slight alternative with what question the current result is. It will also underline the word Toys in a orange dotted line.

Modifying Q&A Synonyms

The Q&A setup button is on the Modelling ribbon tab. When the dialog opens, click on Field Synonyms to see the synonyms. Expand the relevant table and you will see the synonyms applies to each column.

In my example you can see Toy has been added to the Product Name column in the suggested terms. This means alternative questions will be offered if Toy is used in a question.

Click the X to remove a term. When you click the + the term will be promoted to a Term. Visually it will move to the left.

Re-asking the Question

Once the Q&A synonym has been promoted to a term rather than a suggested term, you no longer get another question offered.

Conclusion

Q&A is an under used feature in Power BI. As report writers we assume we know best, and our report consumers’ questions have all been answered. We should add Q&A buttons, educate our users in how to use it and then collect the stats on questions asked, but that is for another blog post!

Power BI Series

Thank You! One Year On

Thu, 06 Aug 2020 10:39:15 +0000

At the end of July 2019 I decided to posting more regularly to my own blog and on the 5th August 2019 I posted my first video on YouTube. This post is to say thank you to anyone reading this and any other post or watching a video.

I remember being highly amused when I got 25 subscribers and when people I respected in the community re-tweeted my links to blog posts and YouTube videos. I find it amazing that in July 2020 I had almost 9,000 page views of my blog and over 3,400 views of my videos. They can’t all be my dad, can they?

So I would like to thank you, the people who read my blog posts, watched my videos, subscribed, re-tweeted and even tweeted their own tweets regarding my content. I’ve been unbearably happy when my posts have been included in community leader round-ups. It has now become a passion of mine to blog and record videos and every like, comment or smile I get is what I do it for.

If you haven’t yet found my YouTube channel it is athttps://www.youtube.com/lauragbpowerplatform

The Data

But let’s be honest really I like the numbers. The pretty charts that show numbers going up, the ways you can track what is going on it is all fascinating for a data geek like me. Without even loading data into Power BI I can get charts and see progress.

Page views on my blog from Google Analytics

Google Search Console Results

YouTube views

I am no expert on Google analytics and promoting sites etc. Megan Walker has done a great series on setting it up and click here to open it

Advice

When I first started my blog I got one piece of advice from a friend, Pieter Veenstra who has a very successful blog SharePains

Every time you are asked a technical question, write a blog post as the answer.

Pieter Veenstra 2019

I’ve not managed to follow that completly but it has meant I always have topics to blog or record about. I haven’t got organised enough to promise a post every week or a video every week. One pattern I will endeavour to follow though is every video gets a supporting blog post.

So to pass the advice on, if you get asked technical questions, start a blog or YouTube channel.

Thank You to everyone!

Power BI – Endorse a Dataset as Promoted or Certified

Sat, 01 Aug 2020 13:47:21 +0000

Connecting to an existing dataset is highly recommended if the data is the same. In order to encourage report builders to select the right datasets you can endorse a dataset as promoted or certified.

YouTube

Connecting to a Dataset

When a report writer looks at a workspace they can see the datasets available in that workspace. When the report writer goes to Power BI desktop and selects Power BI datasets they are given an alphabetical list of datasets from all the workspaces they have access to.

There are two types of endorsement, promoted which any dataset owner can apply and certified that only selected people can apply. These datasets will appear at the top of the list of datasets when connecting.

Endorse a Dataset as Promoted

In the workspace, click on the ellipse menu on the dataset and select Settings. The last option in settings is Endorsement, expand this to see the options. Click on Promoted and click apply.

Enabling Certification

By default users cannot endorse a dataset as certified. This is more restricted to keep the certified label use limited to datasets that have been checked. The Certified option will be greyed out for most users.

If you are a Power BI Admin you will be able to set up who can certify a dataset in the Admin Portal. From any dataset in the Power BI website, click on the three dots in the top right of the window, select Settings and then Admin portal.

In the Admin Portal, select Tenant settings and then scroll down to find Certification. Expanding that section reveals a toggle button enable Certification. Once enabled, you can select who it applies to by entering in a security group. Click Apply to save your changes. It can take up to 15 minutes for the changes to take effect.

Note : A company that has guidance on certification of a dataset, can add the url to that guidance in the documentation page box. Any dataset owner can then click on Learn more to see your company guidance.

Endorse a Dataset as Certified

In the workspace, click on the ellipse menu on the dataset and select Settings. The last option in settings is Endorsement, expand this to see the options. Click on Certified and click apply.

Updated Connecting to a Dataset

Now when a report writer, from Power BI desktop, wants to connect to a dataset they will be offered certified datasets, then promoted datasets, with all other datasets listed afterwards.

Conclusion

One dataset to maintain and refresh is the best option, so anything that gets users to connect to the correct dataset is a great idea. Two levels of endorsing works quite well. Dataset owners promote their own datasets. Companies certify datasets that they have checked out.

Power BI Series

Power BI – Gradient Legends

Sat, 25 Jul 2020 21:11:21 +0000

Introduction

In the July 2020 update to Power BI Microsoft introduced a new feature of Gradient Legends to help annotate the colours used in a colour scale conditional formatting. When conditional formatting is applied to a visual it is important to make sure the reader of the report understands what the different colours on the report mean. Is purple good? Is orange ok?

YouTube Version

Adding Conditional Formatting

In this chart we have Turnover plotted for every month as a bar chart. The sales manager now wants to colour each bar based on the profit margin.

They select the chart and select the format section of the visualization pane. Under data colours they click the fx button next to the colour to open the conditional formatting dialog box.

Then the sales manager changes the field selected to the measure already setup of Profit Margin. They then can change the colours for the minimum and maximum values before pressing OK to apply the formatting.

View Legend

Not all visuals have the legend turned on by default. The legend can be turned on in the visualizations pane.

Changing Values in Gradient Legends

The min and max values will be calculated based on the data. These can be adjusted by returning to the conditional formatting dialog box, via visualizations pane and data colours and clicking on the fx button.

Then for the min and max values change from Auto to custom for the value and type in the values. If a plotted value is below the minimum it will be the minimum colour and the same is for maximum. In this example we have just gone to the nearest 5%.

Adding a middle colour

A mid-way third colour can added to the colour scale and that will update the legend to match.

Return back to the conditional formatting of the column colour and tick the diverging box. This will now add a third colour. After you click okay, the legend will now be updated to show all the values.

Conclusion

For those who use colour scale conditional formatting this is a great addition to make their story on the dashboard crystal clear. Adding gradient legends is crucial to annotate charts clearly.

Power BI Series

Power BI – Introducing Sync Slicers

Wed, 15 Jul 2020 08:50:19 +0000

How many reports have duplicate slicers across the pages and the report reader has to re-apply the slicer values on each page? This post covers what the sync slicers feature in Power BI is.

Slicer Series

This series will cover different topics regarding slicers.

Slicers Introduction
Resetting Slicers with a Bookmark Button
Cascading Slicers
Hierarchy Slicer
Sync Slicers
Clear all Slicers Button
Relative Date Slicer

YouTube Version

Sync Slicers Pane

On the View ribbon tab there is a Sync Slicer button which will show the Sync Slicer pane. When you click a slicer it will list the pages and show which pages already contain that slicer. In the Example shown below the ShopName slicer is on the Summary and Turnover pages but not on the Volume page.

Sync Slicers

In the above example, if the slicer on the Summary page is set to Daisy Delights the slicer on the Turnover page will not change. This is due to no ticks in the sync column.

When the two pages have ticks next to them the slicers will sync. This means that selecting one shop on Summary will select the same shop on the Turnover page.

Copy Slicers

In the example above the Volume page does not have the Shop slicer. If the sync tick is selected the Volume page will be filtered using the slicer from the other pages.

When the view slicer is ticked, a copy of the slicer with exactly the same formatting and position will be placed onto the Volume page.

Deleting the slicer from the page will stop the page being synced with other pages.

Advanced Options

The advanced options really confused me. So a google search found me a great video from Reid at Havens Consulting explaining what they mean.

Adding the same group name to each slicer, makes the slicers sync which fields they contain and the filters applied. I highly recommend watching the above video to see a full explanation.

Conclusion

Sync slicers is a great feature for reports that contain multiple pages and slicers are used cater for different people.

Power BI Series

Power BI – Personalize Visuals

Fri, 10 Jul 2020 19:49:35 +0000

In the May 2020 Power BI Desktop update they introduced the idea of allowing a reader of a report to personalize visuals and bookmarking these changes. This post walks through how to enable it for a report and how to enable and disable for each visual.

YouTube Version

Turning On Personalize Visuals

At the point of writing this blog post personalise visuals is still in preview so you need to head to File – Options & Settings – Options, and then from the Options dialog select Preview Features and turn on Personalize Visuals. This may require a Power BI restart.

Secondly you will need to turn on personalising visuals for your current report. So in Options, under Current File select Report Settings. The last option in there is Personalize Visuals, you need to tick Allow report readers to personalize visuals.

Adjusting Individual Visuals

By default, once Personalize Visuals has been turned on in a report, all visuals in the report can be personalised. You can disable it on any visual by selecting the visual, then looking under Format and searching for person and turning it off.

Personalize Visuals in a Published Report

To personalize a visual in a published report, click on the personalize button in the visual header.

The then Personalize pane will open up next to the visual and the chart type and fields used can be changed. A reader will not be able to change the formatting of the visual.

Saving Personalisations as a Bookmark

Any changes made to visuals in reports can be saved as a personal bookmark. In the top right hand corner of the browser window click Bookmarks and then Add a personal bookmark. Enter in the name of the bookmark and click Save.

Access the list of bookmarks by clicking on the drop down in the top right hand corner.

One great way to give feedback to a report designer would be to show them the personalizations you’ve done to their report. If you have person to share you can click the Share button at the top of the report.

Enter their details and add a message if required. Tick Include my changes and then click Share. They will receive an email giving them a link that includes the changes you made to the visuals.

Conclusion

This feature is great when different report readers want to adjust what the original designer did or different charts to be tried out. I know designers who will not want anyone to change anything and designers who perhaps will be more tolerant.

Power BI Series

Reactor – Power BI Introduction Resources

Mon, 22 Jun 2020 14:08:28 +0000

This post is to support my virtual session at London Reactor on 22nd June. It will include the recording and links to the data source and the final Power BI Report.

Link to Recording will go here….

Data Used at Reactorand Final File

Data in a single Excel File

Final PBIX File

Links to great Resources

London Reactor Posts

The Anxious MVP

Wed, 03 Jun 2020 18:22:41 +0000

The buzz of being nominated for an MVP is great, adding the community activities you do with passion all working towards what is built up to being a huge a goal. Hopefully eventually you get the email we’ve all seen in social media to say you’ve been awarded an MVP. You put out the messages saying how humbled, honoured, happy and other words beginning to h and everyone celebrates with you, you get followed, connected to by people you’ve never heard of and you stick the blue badge everywhere. For me that happened 1st December 2019.

You’re now a member of a community of people who do some amazing things and work extremely hard to make the Microsoft user community what it is. Behind the scenes there is huge amount of work most people never see. Congratulations, life goal achieved.

Then I fell apart. The community is huge, everyone else seemed to know what they were doing and where they were going and I didn’t. Maybe I wasn’t really invited, maybe I shouldn’t be there atall. Maybe no-one wants me here.

Why didn’t I reach out to my friends in the community? Anxiety.Why didn’t I reach out to my amazing MVP co-ordinator in the UK? Up to being awarded the MVP in my head she had been the judge I sure as heck wasn’t going to admit she might be wrong.Why did I just hide away? Anxiety

When a community is a busy vibrant community it is fantastic when you know where things are and you’ve found your seat… Until that point for the anxious amongst us, it is as scary as hell. Imagine a scared puppy at Oxford Circus in the pre-Christmas rush, that was me.

I was rescued, Claire my MVP co-ordinator was great when I admitted I was struggling. Various friends helped me out, then a community friend was my sanity and stopped me giving up. I was about to resign my MVP. Thankfully he got his MVP a few days ago, he totally gets the we rise by building up others. A smile, a simple “are you okay?”, stepping back to expand a circle wider to welcome people in, remembering someone’s name, saying hello, were simple actions that helped me. And I hope I do in return for others.

So why write this post? It is not my normal how-to post. I’m aware that I am not alone in struggling with the social side of community. I wanted to make sure people knew they weren’t alone if they felt anxious. More importantly I wanted to reach out to community members to have their radar up for the person who might be struggling. My anxiety is mild, it is real but most of the time few know it exists. I have no history to cause my anxiety, I’m employed, happy family, life is good. Anxiety is just part of my makeup.

My desire to be an MVP was to be a member of the community. What I’ve learnt is I was already a member and people around me wanted me there. No badges required. Okay I’ve learnt that logically, still a way to go to learn that emotionally. I’m not 100% “okay”, I doubt I ever will be, but this is who I am. My anxiety is part of the real me; if I was “fixed” would I be me?

Power Apps – Embed a Power BI Chart

Sun, 31 May 2020 12:57:14 +0000

This post covers how to embed a Power BI chart from a published report into a Power App, handling a tile that doesn’t appear and then filtering the embedded chart.

YouTube Video

Creating the Chart Tile

The report must be published to a workspace that the app users will have access to. When you open in the Power BI service in the visual header is a pin, clicking this will prompt to “Pin” your visual to a dashboard.

When the dialog appears select either an Existing dashboard or a New dashboard. If you are using Themes do remember to select Keep current theme on the left hand side. Click Pin to finish pinning your chart to the dashboard.

When the visual has been pinned you will get a notification in the top right of the browser window with a button to navigate you to the dashboard. The dashboard can also be found in the workspace list. On the dashboard you can see the pinned tiles.

Embed a Power BI Chart

Within your Power App, from the Insert ribbon under Chart, select Power BI Tile. This will open a pane to select the Workspace, Dashboard and Tile. When selected the chart will appear in your app.

Fixing the missing Chart

When I first did this no charts appeared. So a few google searches later I found Power Apps is looking for a title to be set. All of my charts had altered titles. When I tried a chart that just had the automatic title, it worked. A little more digging and I found you can set the title of a tile on a dashboard. Just opening the editor and pressing Apply fixed the problem.

If you have a missing chart try the following:Open the dashboard. On the tile that Power Apps does not see, in the top right hand corner are 3 dots, click to open an options menu. From the menu select Edit details.In the Tile details pane click Apply.Save and refresh your Power App and try re-adding your tile.

The above fix has worked for every problem I’ve had so far. Please add comments if it doesn’t work for you.

Filtering the Chart

Power BI is powerful because charts can be sliced easily. When you only have a single chart from a report you don’t have the option to include slicers and filters. If you examine the Power BI control in Power Apps you will see the TileUrl property for the control is a long url.

An extra parameter can be added to this url to filter the chart. The filter is written as an OData filter. You need to know the table and field names to write the filter. In this example if I look back in the Power BI report in Desktop I can see the Colour field is from the Toys table.

So the filter string to filter to Red will be:

1

"&filter=Toys/Colour eq 'Red'"

The colour could come from a drop down or a variable. Remember to include the single quotes needed around the filter value. Also spacing is important so don’t add any extras!

Conclusion to Embed a Power BI Chart

Embedding a Power BI Tile is a great addition to a Power App but it does have limitations. The dataset behind the Power BI report will need to be refreshed to reflect any changes and refreshed are limited to 8 times a day or 48 times in a premium workspace.

Power Query – Replace Values for Whole Table

Wed, 15 Apr 2020 15:40:57 +0000

Today’s challenge was to replace the values on all the columns in the query when I know the columns will change so I don’t want to name them.

Quick Answer

For those who don’t want the long explanation. Do replace values on at least one column to get the replace step. In the example below the previous step is #’Changed Type’ and the columns are Jan 20 and Feb 20.

1

= Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,{"Jan 20", "Feb 20"})

Remove the {…} section and replace it with Table.ColumnNames(Previous Step)

1

= Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,Table.ColumnNames(#"Changed Type"))

Longer Answer

Replace values is a powerful tool for coping with null values and errors. When you select multiple columns and select Replaces values a step is written in that includes a list of column names, eg {“column1″,”column2″,”column3”}. If your data structure changes that step will either break because a column is missing or will not include a new column.

The function Table.ColumnNames(StepName) returns a list of column names from the named step. To see this result and filter the list right click on a step and select Insert Step After. Edit the new step to be Table.ColumnNames(StepName)

The list of column names can be filtered, even though there is no drop down arrow. Right click on the value to remove and select Remove Item.

If I then rename this step to ColumnNames I can then use that step name in a replace step.

Note

Yes the Replaced Value step is refering to a calculation that comes after it (yup its wacky). If you reorder the steps by dragging them Power Query will decide to change the code so steps refer to the new previous step.

References

Microsoft Docs Table.ColumnNames

Power Apps – Build a Responsive App – Adding Dynamic Containers

Thu, 09 Apr 2020 16:00:16 +0000

In this post we will use dynamic containers to arrange out content so it moves to the planned location at the different screen sizes. We will need to calculate the screen size and then use it to position and size the containers.

This post assumes you’ve done everything in the first 2 posts.

YouTube version

This series is to support my YouTube video.

The posts for this series are

Calculate Screen Size

In order to make the calculations simple we are going to add a label called AppScreenSize which contains a calculation to classify the screen size to one of small, medium, large and extra large.

For this we are going to use a property of the App object, SizeBreakpoints. It contains a list of numbers for the dividing points of the screen sizes. So under 600 will be small, 600 to 900 will be medium, 900 to 1200 will be large and above 1200 will be extra large.

We then add a label, renamed to AppScreenSize and change the text property to the following formula.

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126


If(
 App.Width

Now the label shows “Extra Large”.

### Adding Title Bar Container

In my example the title bar is the same for every screen size so is the simplest to position and size. It will start in the top left hand corner so x,y will be 0,0. The height is going to be 80 and the width needs to fit the full screen width. The screen is the parent of the container so we can use Parent.Width.

Within the container I’ve placed a text label that will start in 0,0 and the width and height match are Parent.Width and Parent.Height. I also set the text to be AppScreenSize.Text & ” App”. Finally I formatted the text label.

![Title bar dynamic containers](Responsive_203.jpg)

#### MainTitleContainer

X0Y0Height80WidthParent.Width

### First Test

The app is now ready to save and publish. The main test is to check the app resizes correctly and the title bar changes.

![first test of dynamic container](Responsive_204.jpg)

## Adding Dynamic Containers

Now we are going to add the containers that will move based on the screen size. In total there are 4 containers to add. The left hand side people selector, the right hand side people information which will contain the final 2 containers that sit side by side in large and extra large and above each other on a medium screen.

The next container we add is the container down the left hand side. It needs to fit in below the title bar container and fill to the bottom of the screen. For the small screen size the width will be the whole screen width, for all other sizes it will be 350 wide.

![LHS Dynamic Containers](Responsive_205.jpg)

#### PeopleSelectorContainer

X0YMainTitleContainer.HeightHeightParent.Height-MainTitleContainer.HeightWidthSwitch(
    AppScreenSize.Text,
    "Small" , Parent.Width ,
    350
)

The next container is the People Info container for the right hand side of the screen. This will fill the remaining part of the screen for all sizes except small when it will be 0 width and not seen.

Note I coloured the containers just so we could see the sizes.

![People info dynamic containers](Responsive_206.jpg)

#### PeopleInfoContainer

XPeopleSelectorContainer.WidthYMainTitleContainer.HeightHeightParent.Height-MainTitleContainer.HeightWidthSwitch(
    AppScreenSize.Text,
    "Small" , 0 ,
    Parent.Width - PeopleSelectorContainer.Width
)

### Second Test

With the two dynamic containers added we can publish and test again to make sure the PeopleSelector resizes to full screen for the small size and the PeopleInfoContainer takes up the right hand side when for any size except small.

![testing People info dynamic containers](Responsive_207.jpg)

### Nested Dynamic Containers

The last two containers will be child containers of the PeopleInfoContainer. For Large and Extra Large screen size they will sit side by side, for Medium screen size they will be one above the other, and for Small it won’t matter because the parent container will be 0 width.

![nested People info dynamic containers](Responsive_208.jpg)

#### PeopleInfo1Container

XPeopleSelectorContainer.WidthYMainTitleContainer.HeightHeight400WidthSwitch(
    AppScreenSize.Text,
    "Medium" , Parent.Width ,
    Parent.Width /2
)

#### PeopleInfo2Container

XSwitch(
    AppScreenSize.Text,
    "Medium" , Parent.Width ,
    Parent.Width /2
)YSwitch(
    AppScreenSize.Text,
    "Medium" , PeopleInfo1Container.Height ,
    0
)HeightSwitch(
    AppScreenSize.Text,
    "Medium" , Parent.Height - PeopleInfo1Container.Height ,
    PeopleInfo1Container.Height
)WidthSwitch(
    AppScreenSize.Text,
    "Medium" , Parent.Width ,
    Parent.Width /2
)

## Final Test

We now have 4 dynamic containers that should all be responsive to the screen size, so we save and publish.

![final test](Responsive_209.jpg)

## Conclusion

Responsive design takes time, lots of time. This is why good web coders are paid the rate they are paid, it is a skilled job. Building an app to be fully responsive will take time. I hope this post gives you enough to get going along with the resources from Microsoft.

### Resources

Microsoft have some resources found at

[https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/create-responsive-layout](https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/create-responsive-layout)

## More Power Apps Posts

- [Transparency Update](https://hatfullofdata.blog/powerapps-transparency-update/)

- [Using JSON Feature to Save Pictures](https://hatfullofdata.blog/powerapps-using-json-function-to-save-pictures/)

- [AI Builder Object Detect Model](https://hatfullofdata.blog/ai-builder-object-detect-model/)

- [Function Component](https://hatfullofdata.blog/powerapps-function-component/)

- [SVG in Power Apps series](https://hatfullofdata.blog/powerapps-svg-introduction/)

- [12 Days of Components](https://hatfullofdata.blog/power-apps-12-days-of-components/)

- [Build a Responsive App series](https://hatfullofdata.blog/power-apps-build-a-responsive-app-planning/)

- [Embed a Power BI Chart](https://hatfullofdata.blog/power-apps-embed-a-power-bi-chart/)

Power Apps – Build a Responsive App – Initial Setup

Wed, 08 Apr 2020 16:09:24 +0000

A responsive app resizes the app based on the browser window and moves parts of the app to make the app work in different screen sizes. The most common sizes to handle are monitor, tablet and mobile phone.

YouTube version

This series is to support my YouTube video.

The posts for this series are

Create the Responsive App

In the video and for my example I design the app based on a the larger size and then calculate where things will move or transform to become the smaller sized app for a phone for example.

App Settings

The first change to make inside the app is to turn off Scale to fit which allow us to control sizing etc.

From under the File tab, click on Settings and then Screen size + Orientation. Scroll down to the Advanced settings and turn off Scale to fit.

Enabling Containers

As of April 2020 containers are still an experimental feature. For the html writers, containers are like the div element. They allow the developer to group items that are sized and positioned within the container, and then specify the size and position of the container. This makes a responsive design possible rather than a very complex design nightmare.

Features can be turn on from File > Settings > Advanced Settings, scroll down and find Containers

Settings Screen Size

By default the screen height of every screen in an app is set to Max(App.Height, App.DesignHeight). This assumes the app is designed small and the screen should expand. I prefer to do the reverse and design the larger size and work out how to scale down. So I set the screen height and width to be App.Height and App.Width.

Conclusion

The app is now ready to add containers to fit the plan we created earlier. The next post will be adding the containers.

Resources

Microsoft have some resources found at

https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/create-responsive-layout

Power Apps – Build a Responsive App – Planning

Wed, 08 Apr 2020 15:54:03 +0000

Introduction

Planning a responsive app is vital. A responsive app resizes the app based on the browser window and moves parts of the app to make the app work in different screen sizes. The most common sizes to handle are monitor, tablet and mobile phone. Just like a choreographer has work out the position of every dancer in a dance you need to work out the position and form of different elements at every screen size.

YouTube version

This series is to support my YouTube video.

The posts for this series are

Screen Sizes

Although the app can be made to react to even the smallest screen size change it makes logical sense to work with a few screen sizes and these are based on the width of the screen. The sizes we will work with are small, medium, large and extra large.

Layout Planning

For this series we will be working with a simple people app that will have three layouts. The Extra Large and Large size have the same layout, the Medium size rearranges the info boxes to be on top of each other and the small hides the info box completely and will have navigation to another screen for the info boxes.

Further Planning

Each container will need some planning to select how they will look in the different layouts. For example the Gallery in the Large and Extra Large layouts could container less data as you have the info boxes, where as the small only shows the Gallery on the front screen so although its a smaller screen you might want more information in the gallery.

Conclusion

You need at least a simple wire frame before you start creating the app. Each section of the screen probably needs extra planning but this can be done once you have the app structure working.

Resources

Microsoft have some resources found at

https://docs.microsoft.com/en-us/powerapps/maker/canvas-apps/create-responsive-layout

Power BI Series

Power BI – COALESCE Function to Remove Blanks

Fri, 03 Apr 2020 09:27:01 +0000

Introduction

This post walks through a simple use of the coalesce function to prevent a measure returning a Blank. The example used is a simple measure.

YouTube Version

Measure

The above image shows a card that displays the Total Sales measure and due to slicer selections on the page it returns a blank. The measure is a simple SUMX calculation of Transactions[Qty] * Related Products[Price].

Adding Coalesce

Coalesce function can take multiple parameters. It evaluates the first one, if that returns a value it returns that value, if it returns a blank the function moves onto the next parameter and continues until it finds a non-blank value. If all the calculations return a blank the function returns a blank.

In this example we would like the Blank to be replaced with zero. In other measures you might want it to be another value. So we wrap the whole calculation in a coalesce function.

1
2
3
4
5
6


Total Sales = 
// Return 0 if blank
COALESCE (
 SUMX ( Transactions, Transactions[Qty] * RELATED ( Products[Price] ) ) ,
 0
)

The card now shows a zero value. The Measure was also formatted to be a currency, so we get £0.00.

Conclusion

This is just a simple example of how the coalesce function can improve the output from measures. The zero could be a string such as “No Sales” or could be an alternative calculation followed by a zero if both calculations returned a blank.

Power BI Series

Power BI – Introducing the Hierarchy Slicer

Wed, 25 Mar 2020 19:45:03 +0000

Does your data include a hierarchy? We often can end up with multiple slicers to walk the hierarchy taking up valuable screen. This post does a quick walk through using the new hierarchy slicer.

Slicer Series

This series will cover different topics regarding slicers.

Slicers Introduction
Resetting Slicers with a Bookmark Button
Cascading Slicers
Hierarchy Slicer
Sync Slicers
Clear all Slicers Button
Relative Date Slicer

YouTube Version

Adding a Hierarchy Slicer

Add a slicer from the visualisation pane and add the top level of the hierarchy, for example ProdRange from my products table. A tick list will appear just as previously.

The add the next level down of the hierarchy into the Field box under the top level field. The slicer will now have down arrows that can be clicked to expand to show the next level down of the hierarchy.

NoteIn March 2020 you need to activate the Hierarchy Slicer in File – Options – Preview features and then save and restart Power BI. When this visual is not in preview you won’t have to.

Multiple layers can be added to the hierarchy. Of course the usability of the slicer needs to be considered when adding many layers.

Formatting a Hierarchy Slicer

This slicer can still be formatted as a drop down or changed to be single select. The single select allows a whole level of the hierarchy to be selected such as the whole year or a whole month.

Using Search

When search is turned on in a tick list the search returns the terms plus the parents or children of the items returned.

Conclusion

The Hierarchy slicer is a great addition to Power BI and I am sure will replace groups of multiple slicers in many reports.

Power BI Series

Power BI – Reset Slicers with a Bookmark Button

Mon, 09 Mar 2020 12:11:00 +0000

When your report has a bunch of slicers and charts that can also add filters to the page resetting back to no filters can be painful. So we need a quick way to reset slicers. This post walks through the steps of creating a bookmark and the using that bookmark to reset the page by clicking a button.

Slicer Series

This series will cover different topics regarding slicers.

Slicers Introduction
Resetting Slicers with a Bookmark Button
Cascading Slicers
Hierarchy Slicer
Sync Slicers
Clear all Slicers Button
Relative Date Slicer

Introduction

This post will walk through the stages to add a reset button to a report. It comes in 2 parts, firstly create a bookmark that removes filters and then to add button that applies the bookmark as its action.

YouTube Version

Create Bookmark

Bookmarks include the current status of slicers on the current page. So the first job is to clear all the slicers and make sure to selections on charts is filtering the data.

From the View ribbon, select Bookmarks
Click on the Add button and a bookmark will appear.
Double click on the bookmark to rename it

The bookmark can be tested by selecting some options on the slicers and then clicking on the bookmark name and the slicer options should be cleared.

You can now close the Bookmarks pane.

Adding Reset Button

A bookmark action can be added to a Button, Shape or Image. Select one from the Insert ribbon tab. In the format of the button / shape / image click the Action to be on, change the Type to be Bookmark anf the Bookmark to be the bookmark you created.

When viewing the report in Power BI desktop you need to hold the CTRL key when you click the button. In the Power BI Service, i.e. where your report is published you will not need the CTRL key.

You can add a message into the the Tooltip to assist your report consumers.

Updating Bookmarks

When you add new slicers or reformat your slicers it is always a good idea to update your bookmarks. Open the bookmarks pane from the View ribbon tab. Click on the … next to the bookmark name and select Update.

Conclusion to Reset Slicers

It is important that your report is easy to use. Adding a button so you are able to reset slicers is a good step towards that.

Power BI Series

Power BI – Slicers Introduction

Fri, 06 Mar 2020 11:17:56 +0000

Slicer Series

This series will cover different topics regarding slicers.

Slicers Introduction
Resetting Slicers with a Bookmark Button
Cascading Slicers
Hierarchy Slicer
Sync Slicers
Clear all Slicers Button
Relative Date Slicer

Power BI reports can be made more flexible using simple slicers to allow your report consumers to slice and dice the data down to what they are interested in. This post will look at the various options for formatting slicers in different ways.

YouTube Version

YouTube version

Simple Slicer

Click on the slicer from Visualizations pane to add a slicer to your report.

Then you need to drag in the value you would like to slice by, for example Shop Name. Your slicer is now ready to use.

You can select a single value in the slicer just by clicking. You can select multiple items by holding down the Ctrl key. To clear the slicer click the eraser icon in the top right.

Selection Options

Our slicer tick list can be tweaked in various ways to make the report fit different requirements. These options can be found in Formatting (click on the paint roller) and Selection Controls.

Single select will turn the selection boxes into round radio buttons. Only one item can now be selected and there is always one item selected, i.e. you can no longer select all.

Turning off Multi-select with CTRL will no longer require the ctrl key to be selected to select multiple options. This does mean you need to un-select items to remove filters. This is great for touch screens. You also get little ticks in the boxes.

Show “select all” adds an option at the top of the list that will toggle turning them all on or off. Showing the “Select all” is great for when report readers will want all excluding one. Some report readers find the concept of selecting none gives you all a hard one to get their head around.

Button Slicers

When there are only a few options in a list can be effective to change from a list to buttons. This option is not the most intuitive but easy to find when you know how.

Select Format – General and change the Orientation from Vertical to Horizontal and you get buttons!

When a list of options is long the tick list takes up too much screen space and becomes hard work to use. So we can change a tick list into a drop down.

In the top right hand corner of the slicer box there is a drop down, which only shows when your cursor is over the slicer. On a simple list slicer this drop down offers List or Dropdown. Selecting Dropdown will reduce the list into a smaller drop down.

Adding Search

When a slicer has lots of options it can be useful to add a search box to the slicer. The most common place to add this is probably the drop-down but it can be added to almost any slicer.

Click on the three dots at the top of the slicer window, or the the bottom if your slicer is at the top of the page! Select Search. This will add a search box to the top of the slicer options and will filter the options to match any text.

Range Slicer

Dates and numbers don’t work well for a list so when you add a number or date column to a slicer it will default to a range slicer. This slicer has two handles that allow you to modify the range selected.

Clicking the top right drop down will offer Between, Before and After as different range options as well as converting to List or Dropdown.

Conclusion

There are many options for slicers, you need to know your audience and which slicers they will prefer. So make sure it is obvious to your users what they are filtering. If you need slicers to filter slicers take a look at Cascading slicers.

Power BI Series

Power BI – Connecting to Project Online

Tue, 21 Jan 2020 12:22:15 +0000

I have spent 12 months working with a client building a Power BI report as part of their roll out of Project Online. This post walks through connecting to Project Online to visualise the data in Power BI. And the full series will document the stages and lessons learned in the past 12 months.

Power BI and Project Online Series

Connecting Power BI to Project Online data
Reducing columns returned by Project Online
Adding Relationships into Project Online Data

Connecting to Project Online

Navigate to your Project Online site. It will have a URL similar to:

1

https://mycompany.com/sites/MyProjects/default.aspx

Start Power BI Desktop, which will open a new report.From the home ribbon click “Get Data” and find and select OData.

It will prompt you for a url to the data. From the above url, replace default.aspx with _api/projectdata/. My example url becomes:

1

https://mycompany.com/sites/MyProjects/_api/projectdata/

If required sign in with the correct details, it probably is an organizational account you need and click Connect.

From the long list of tables select the tables you need. For the start of this series I’m going to just import Projects, Tasks and TimeSet. TimeSet is the calendar table.

Click Load ready to build your report.

Missing Data

When you edit a project in project online it prompts you to check in when you close.

But when you go load the data into project your updates will not be visible. If you have only checked in a project the whole project will be missing. You need to publish the project for the data to be visible within Power BI.

The Publish button can be found on the Tasks ribbon when editing the schedule of your project. (Why is it not on the Project ribbon? Another mystery!)

Conclusion

The Project Online database is huge and can contain a fascinating source of data. Due to a company being able to add extra columns and select which parts of project they use connecting to Project online can create useful reporting to enhance Project.

Power BI Series

Power Apps – 12 Days of Components

Fri, 27 Dec 2019 23:37:45 +0000

Madly I decided I was going to do 12 videos on Power App components and they have had a heavy focus using SVG. This post is to provide the code and resources for each video.

Day 1 – Create a Component

Video walks through turning on components, adding an SVG image and then adding the component to a screen within the app.

SVG Code

1
2
3
4


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 "
)

Day 2 – Add an Input

This video adds an input, called BaubleColour which is a HEX string, to the Christmas tree component to allow the component user to select a colour for the baubles.

SVG Code

1
2
3
4


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 "
)

Day 3 – Use Table Input

This video uses a table of HEX strings called BaubleColours to provide a list of colours for the baubles. The colours are cycled through using a timer control.

SVG Code

1
2
3
4


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 "
)

Day 4 – Use SVG Transform

This video does a simple transform in SVG to make the star on the top of the tree rotate.

SVG Code

1
2
3
4
5


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 "

)

Day 5 – Flexible Resizing of SVG

Using percentage values within the SVG make the image flexible when the component is resized.

SVG Code

1
2
3
4
5
6
7
8
9


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 
 
 
 
 "
)

Day 6 – Patterned Presents

Using SVG patterns and a list of patterns we can add patterns to the present wrappings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 " & LookUp(Parent.Patterns,PatternName=Parent.PatternName,Pattern) & "
 
 
 
 
 
 
 
 "
)

Excel File for SVG Patterns

Day 7 – Add a Tag using SVG Text and Rotate

Using SVG Text and rotate we can create another component to draw a tag that uses an input of the name to label each present.

1
2
3
4
5


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 " & Parent.TagName & "
 "
)

Day 8 – Star Rating

Rate your New Year Resolution progress with a star rating drawn with a little SVG and a gallery.

1
2
3
4
5
6


With(
 {StarColour:If(
 ThisItem.Value
 "
 )
)

Day 9 – Fill a shape using SVG Clip

Fill up a heart in colour to visualise a percentage value using a component containing SVG using a clip path.

SVG Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


With(
 {
 SVG: "",
 ClipWidth: 100 * Parent.Score
 },
 "data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 
 
 " & SVG & "
 " & SVG & "
 " & ClipWidth & "%
 "
 )
)

SVG Code to fill from the bottom up

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


With(
 {
 SVG: "",
 ClipHeight: 100 * Parent.Score,
 ClipY: 100 - 100 * Parent.Score
 },
 "data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 
 
 " & SVG & "
 " & SVG & "
 " & ClipHeight & "%
 "
 )
)

Day 10 – Fill Part stars using SVG Clip

Combining the star rating from Day 9 with the clip path from Day 10 we can improve the star rating to show part stars for a score of 2.5 etc.

SVG Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


With(
 {
 StarSVG: "",
 ClipWidth: 100 * ('Day10 Star Rating Improved'.Score - ThisItem.Value + 1)
 },
 "data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 
 
 " & StarSVG & "
 " & StarSVG & "
 "
 )
)

Day 11 – Draw a Gauge

Draw a simple gauge to show a percentage value using an arch and rotating a line with a simple transform.

SVG Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


With(
 {
 SVGArch: " ",
 SVGRotate: 180 * Parent.Score
 },
 "data:image/svg+xml;utf8, " & EncodeUrl(
 "
 " & SVGArch & "
 - "
 )
)

Day 12 – Add Colours to that Gauge

Lets add some colour to the gauge from day 11 of this series. We look at rotate again and how moving items partly outside the view box hides that part.

SVG Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


With(
 {
 SVGArch: " ",
 SVGRotate: 180 * Parent.Score,
 RedRotate: -180 * (1 - Parent.RedScore),
 AmberRotate: -180 * (1 - Parent.AmberScore)
 },
 "data:image/svg+xml;utf8, " & EncodeUrl(
 "
 " & SVGArch & "
 " & SVGArch & "
 " & SVGArch & "
 
 "
 )
)

Conclusion

I thoroughly enjoyed building the different components and working out the SVG code to use. I hope you have enjoyed watching. Below is the link to download a zip of the MSApp file for the components.

https://hatfullofdata.blog/wp-content/uploads/2020/01/12-Days-of-Components.zip

Power Apps – Animating SVG Fill Colour and Size

Sat, 23 Nov 2019 23:54:27 +0000

This is the second post in the SVG in Power Apps series. In this post I will introduce animating SVGs by altering the colour and size of the SVG graphic using variables and a timer control.

Series

This series is to introduce ideas for using SVG within PowerApps to add graphics to your Apps.

Introduction to SVG in a PowerApp
Animating SVG colours and sizes
Multi-part SVGs
Rotation and Clip Paths

Selecting Fill Colour

For this post we shall start with a simple circle SVG shape to modify using controls. The code shall start as one circle and the image control will have a border.

The first control will be a drop down to select the colour. We add a drop down with a list of colours as the items and rename it to ColourDropDown.

We then modify the SVG code of the shape to use the selected colour in drop down. The code is now as follows.

1
2
3
4
5


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 "
)

Now the circle changes colour based on the drop down value.

Animating SVG Size

The next stage to use a slider to resize the circle. We add a slider and change the Min and Max values to match the requirements and renamed the slicer to SizeSlicer.

We then modify the code to use the slicer value as the radius.

1
2
3
4
5


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 "
)

Animating SVG Colour

Getting a user to select a colour or size is a simple introduction idea to modifying the SVG image. The next step is to create a simple animation using a timer to modify the size. On another screen we will add a SVG shape with the fill colour and size based on two variables.

First step is to add a button to setup the variables. Then we change the SVG code to use the 2 variables

1
2
3
4
5


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 "
)

The next step is to add a timer. In order to make the shape resize smooth I change the duration to 10, ie 10 milli-seconds. Also change the Repeat and Auto start to be on, so the animation will start automatically and continue to repeat.

The next step is calculate what will happen every 10 milliseconds. The size of the shape should increase by one until it reaches a maximum value and then reduce by one until it reaches a minimum. We will need to change the code behind the setup button to set the vvChange to 1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


If(
 // Reaches maximum so start decreasing
 vvSize >= 60,
 UpdateContext({vvChange: -1}),
 // Reaches minimum so start increasing
 vvSize 

### Colour Animation

The colour of animating SVG can be specified using a hex rgb string. For example #FF0000 is red and #CCCCCC is grey. Power Apps do not include a function to translate from a number to a hex string so in a separate post I explain how to use a component as a function which can be found [here](https://hatfullofdata.blog/powerapps-function-component/) . I add this component and name it DECtoHEX_Blue.

So the we add code to the setup button to create variables vvBlueValue set to 0 and vvBlueChange set to 1. The component is to convert the vvBlueValue into a HEX string, so we put into the component custom property DEC to be vvBlueValue.

Then we add very similar code to the size changing. The final step in the code builds the colour string ready for the SVG.

```xml
If(
 // Reaches maximum so start decreasing
 vvSize >= 60,
 UpdateContext({vvChange: -1}),
 // Reaches minimum so start increasing
 vvSize = 255,
 UpdateContext({vvBlueChange: -2}),
 vvBlueValue 

The animation will now have a circle that grows and shrinks and cycles from black to blue and back again.

### Conclusion

This introduces a concept of using SVG and a timer to create animations for any app. These methods could be expanded to modify the position of elements within the SVG.

## More Power Apps Posts

- [Transparency Update](https://hatfullofdata.blog/powerapps-transparency-update/)

- [Using JSON Feature to Save Pictures](https://hatfullofdata.blog/powerapps-using-json-function-to-save-pictures/)

- [AI Builder Object Detect Model](https://hatfullofdata.blog/ai-builder-object-detect-model/)

- [Function Component](https://hatfullofdata.blog/powerapps-function-component/)

- [SVG in Power Apps series](https://hatfullofdata.blog/powerapps-svg-introduction/)

- [12 Days of Components](https://hatfullofdata.blog/power-apps-12-days-of-components/)

- [Build a Responsive App series](https://hatfullofdata.blog/power-apps-build-a-responsive-app-planning/)

- [Embed a Power BI Chart](https://hatfullofdata.blog/power-apps-embed-a-power-bi-chart/)

PowerApps – Function Component

Sat, 26 Oct 2019 18:24:41 +0000

This post is to describe how I solved a problem using a component. The problem was I wanted to have calculation I could reuse. In a more traditional programming environment I would write a function. Components are reusable so it kind of made sense to use one of them as a function component..

Calculation Details

The calculation I wanted to do was convert a decimal number to a hex number. PowerApps doesn’t have a function to do this so I worked out the calculation. I added a text input to take the decimal number and then a label to contain the calculation.

Create the Function Component

In my app I had already turned on components and added a new component. Shane Young does a great intro to components found here. I then added 2 custom properties. DEC is the decimal value as an input to my component like a parameter to a function and HEX as the returned value.

I then clicked on the HEX Value to enter in a the formula as the following code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


With(
 {
 FirstBit:(DECtoHEX.DEC-Mod(
 DECtoHEX.DEC,
 16
 ))/16,
 SecondBit: Mod(
 DECtoHEX.DEC,
 16
 )
 },
 If(FirstBit>9,Char(55+FirstBit),FirstBit) &
 If(SecondBit>9,Char(55+SecondBit),SecondBit)
)

Using the Function Component

Back on the screen of my app I can use the component using a slider value as the input. I can hide the component as it doesn’t need to be seen.

I needed a HEX value to use in some SVG for an image. In this example I used a slider to determine red value from 0 to 255. I add the component and pass in the slider value into the DEC. I then use the HEX in the SVG code I’m using in an image. See my Introduction to SVG to explain the SVG part.

I could then add a slider and component for green, blue and transparency to make my image fully adjustable.

Conclusion

Components as function opens many possibilities and as 2 of my favourite PowerApp builders Hiro and Brian Dang use them I’m going to explore them some more.

Update

Thanks to Brian Dang for giving me a tidier version of the calculation using nested WITH functions. So the calculation inside the component is now:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


With(
 {
 CalcValue: Mod(DECtoHEX.DEC,16)
 },
 With(
 {
 FirstBit: (DECtoHEX.DEC-CalcValue)/16,
 SecondBit: CalcValue
 },
 If(FirstBit>9,Char(55+FirstBit),FirstBit) &
 If(SecondBit>9,Char(55+SecondBit),SecondBit)
)

PowerApps – SVG Introduction

Fri, 25 Oct 2019 08:23:17 +0000

This is the first post in the PowerApps SVG series. In this SVG Introduction post we will cover adding a simple SVG drawing to a PowerApp screen. Future posts will show how to manipulate that image.

Series

This series is to introduce ideas for using SVG within PowerApps to add graphics to your Apps.

Introduction to SVG in a PowerApp
Animating SVG colours and sizes
Multi-part SVGs
Rotation and Clip Paths

SVG Introduction

SVG stands for Scalable Vector Graphics and is a language to define a drawing. The drawings are defined by co-ordinates and dimensions which means they can be stretched without loosing definition.

I am not going to give a full introduction to SVG here in this post. I recommend you look at my intro SVG post at https://hatfullofdata.blog/svg-in-power-bi-part-1/ for resources and a brief overview.

Using SVG in a PowerApp

Given the resources in the post above we can write a very simple SVG statement that would work on a web page with the following code.

1
2

The above code draws a circle filled in with red. It is centered in the drawing area and if the area is 100px its radius will 40px, and if expanded to to 400px square the radius will expand to 160px.

In order to use the above code we need to do two things, firstly encode the code using the function EncodeUrl and then add a string onto the front stating the format of the image data. So the above code becomes

1
2
3
4
5


"data:image/svg+xml;utf8, " & EncodeUrl(
 "
 
 "
)

We use the above code by adding an image control to the app and replacing SampleImage with the above code.

The image can be re-sized and moved and it will remain a sharp red circle.

More Complex SVG images.

The above example is a very simple red circle. SVG is a complex language that can define very detailed shapes using paths. You can search online or use latest version of Microsoft Office’s Icons saved to disk to get SVG images. For this example I’m going to draw an owl. I saved an icon from PowerPoint as a picture to create this one but I could have found one on the internet.

When I open the SVG in a text editor I can see I have similar top part and bottom parts and and between them multiple paths and other drawing elements. I have replaced all double quotes ” with single quotes ‘.

In order to keep my image code simple I am creating a variable, vvSVGOwl, to store the SVG, in the start code of the app.

We can then use the variable in an image to draw the owl in the app. (Remember to execute your start code, I always forget!)

Conclusion

This post is a quick SVG introduction for PowerApps. Due to the SVG code being flexible and including lots of other options the possibilities for using SVG are endless.

Power Query Functions – SQL Procedures

Sun, 06 Oct 2019 17:25:02 +0000

This is the fourth post in my series regarding writing custom functions in Power Query. This post describes how to create a function that will execute an SQL procedure, passing in a parameter.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Stored Procedure

My database contains a very simple procedure that given a cost center returns all the expense records.

It can be run with the following SQL

1

EXECUTE dbo.GetExpenses @CostCenter="B01"

Create Initial Query

The initial query will excute a procedure with a fixed parameter being passed to the procedure using the SQL in the previous section.

Add a new source of SQL Server. Fill in the Server and Database and then click on Advanced options to reveal the SQL statement box where you enter the SQL statement to execute the procedure.

Click OK to and then Transform Data to edit the query. When the query opens it has only one step called Source.

Create Parameter and Function

After creating the query, we need to create a parameter to store the value to pass into the procedure. So from the Home ribbon tab select Manage Parameters – New Parameter. You need to give the parameter a name and the type Text and lastly a value.

Then we need to edit the Source step of the query to use the above parameter. Remember to get the number of ” correct, you need 2 ” if the ” is inside a string.

When you alter the query you might get a prompt asking for permission to run the query. For now let it run the query, further on in this post I will discuss this further.

After we have included the parameter into the query it is ready to be converted into a function, just as we have done in previous posts. Right click on the query and select Create Function, then enter in a name for the function. As before the query and parameter get moved into a group with the new function.

This function can now be invoked on a table of cost centers to give all the related expenses.

Changing Security

When you invoke the query you will get a prompt asking for permission to run a native database query. This is a caution to warn you that some SQL is being run which could makes changes to the database.

When you click on Edit Permission you will be shown the SQL that is going to be executed, so you can check it carefully.

This request for permission can be turned off. Click on File and then Options, and then Security. The first option in the dialog is regarding Native Database Queries, by default the require approval is ticked. Be aware this changes it for all queries.

Conclusion

SQL procedures are a great way to make use of the dba to build the queries you need to fetch the data using your parameters. Obviously this will require many biscuits for the dba. I think there is plenty of scope for functions that execute an SQL procedure.

Resources

I am not the first, and hopefully not the last to write blog posts on writing functions in M for Power Query. Here are a list of the resources I found useful. (If you know of any good ones I’ve missed please let me know!)

Power Query Functions – Fetch web data

Sat, 05 Oct 2019 17:33:54 +0000

This is the third post in my Custom functions in Power Query series. Often web based data is split across multiple pages using the url to select the data. This post describes how to build a function that populates the parts of the url so multiple pages can be used to fetch web data can be combined.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Initial Query to Fetch Web Data

We start the whole process by fetching data from one page of the web site. for my example in this post I am going to use data from Formula 1 racing. The page I am using as my initial page if for the first race in 2019 in Australia. The url is

1

https://www.formula1.com/en/results.html/2019/races/1000/australia/race-result.html

I’ve used table by example to extract the data I need. (There is a blog post coming cover table by example.) So I end up with three step query.

Adding First Parameter to Web Query

The next step is to replace parts of the url with parameters in the Source step. If we look at the url it has three parts that could be made into parameters. The Race Year, Race Number and Race Name. For the the first race in 2019 the values would be :

RaceYear – 2019
RaceNumber – 1000
RaceName – Australia

The first parameter needs to be created before you change the Source step. From the Home ribbon tab, select Manage Parameters, New Parameter. Then you need to enter in a name for the parameter, a type of text and the current value. The type has to be text for the parameter to get included in the next step. Finally click OK to create the parameter.

Now we can start to change the source step in the query. Start by clicking on the cog wheel of the Source step which will open a dialog. Then click on Advanced to give the option to break the URL into parts. You can click on Add Part to add extra rows. The first drop down allows offers Text or Parameter. If you don’t get the drop down on the left it means you don’t have any text parameters.

Adding Extra Parameters

Earlier I stated that the url had three parts that could be put into parameters. We can create and add the new parameters inside the From Web dialog. Click Add Part to add a new row. From the new drop down select New Parameter and enter in the details of your new parameter, remember they need to be text.

When you have finished adding new parameter and breaking the url up, click OK to save your changes. Check you still get the correct data from your query.

Creating Function from Query

We now have a query that can fetch web data which we now need to use to build a function from it..

Right click on your query and select Create Function. Type in a name for the query.

The query and the parameters are all moved into a group and a function is created. If any changes are required in the function, the changes need to be made to the query. If changes are made to the function it will break the link with the query.

Fetch web data from Multiple pages

I have a list of races with year, number and name in an Excel file which I load into a new query. In order for the function to work the RaceYear and RaceNumber columns need to be text.

From the Add Column click Invoke Function. Type in a New column name and select the function in the drop. Then the parameters for the function will appear. Change the first drop down for each parameter to Column Name and select the column.

When you click the OK button the function will run for every row of the table. This will create a column that contains tables. Click on the icon in the top right of the new column. In the dialog box that appears it will show the columns in the table. I un-ticked the Use original column name as prefix to keep the column names short,

After OK is clicked the column of tables is replaced by the three columns from the table.

Conclusion

If your data is web based this provides an efficient method of fetching web data into one table only using a few queries and a function.

Resources

Power Query – Multi-step Function

Sun, 29 Sep 2019 17:28:40 +0000

This is the second post in my Writing Functions in M series. This post describes how to build a multi-step function in M to allow for a more complex function. We will create a parameter to base the calculations on and then build a function.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Introduction

In the first post of this series we built a different functions and all of them were a single step calculation. Some functions need to be made up of multiple steps. The example we will use in this example is to calculate the next working day based on Saturday and Sunday being non-working days.

In simple terms the calculation is simple, if today is a Friday add 3 days, Saturday add 2 days, otherwise just add one day. I realise this could be done with a nested if, I’m using this example as a nice easy one to walk through.

Creating a Parameter

In order to make writing functions easier we create parameters to store the values passed into our multi-step function. This function will only require a single parameter to store the date which we want to calculate the next working day.

From the Home ribbon tab, select Manage Parameters and the New Parameter. Then enter in a name for the parameter, e.g. ThisDate and select a Type, e.g. Date. Finally add in a Current Value and press OK.

Calculation Steps

Now we can start creating the function in with a blank query which I have renamed to NextWorkDay. This first step I am going to make equal to the parameter we created, ThisDate. (Remember the = , I have forgotten it too many times)

The first calculation step I am going to add is to calculate the day of the week of the given date. This can be done from the ribbon tab DateTime Tools. Select Date – Day – Day of Week and a new step gets added using the function Date.DayOfWeek.

Rename the step just added to something shorter, e.g. WeekDay.

The next step will be to calculate how many days to add to the date. You can add a new step by clicking on the fx next to the formula bar.

We use an If statement based on the previous step. There is no quick way to add this so you will need to type in formula. The editor does now have intellisense so you should get some assistance in typing in names.

Using the logic described in the second paragraph of the introduction the complete formula is

1
2
3


= if WeekDay=4 then 3 else 
 if WeekDay=5 then 2 
 else 1

And I rename the step to be DaysToAdd to make the next step easier and it should return 1,2 or 3. You can click on any step and see the result, for example clicking on Source gives you the date.

So the final step is to use Source and DaysToAdd to calculate the next working day. We use the function Date.AddDays which requires 2 parameters, a date which is Source and the number of days to add which is DaysToAdd.

We can change the date in the parameter ThisDate to test different dates and check the calculation logic. Now we are ready to convert this to function.

Creating the Multi-step Function

Power Query includes a quick method to convert a query that uses a parameter into a function.

Right click on the NextWorkDay function and from the menu select Create Function. Enter in a name for the function in the Create Function dialog and press OK.

This will move both NextWorkDay query and ThisDate parameter into a separate group with the new function. If you want to edit the function you need to edit NextWorkDay and it will update the function fnNextWorkDay. You can edit the function directly but it will break the connection with NextWorkDay so debugging will be harder and is not recommended.

Using your Multi-step Function

The function can now be used in any query in your report by clicking Invoke Custom Column from the Add Column ribbon.

Conclusion

The quick Create Function ability makes creating and editing the functions much easier. It makes building functions worth doing and not a huge development process.

Resources

Power Query – Custom Handwritten Function

Fri, 27 Sep 2019 17:50:43 +0000

Introduction

In this post I will write a simple custom handwritten function to perform a calculation based off a parameter. We will then expand that function to include data types and optional parameters.

For those who are new to programming a function is a piece of code that returns some data, which could be a single value, record or table of data. If you ever need to duplicate a calculation more that twice then it is worth considering writing a function.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

The Hello World function

Every programming 101 course includes some sort of Hello World function as the introduction. I would hate to break tradition.

In Power Query we add a Blank Query and we get a blank formula bar. Enter in the following code.

1

= () => "Hello World"

When you press return the blank query will be converted into a custom function. It gains an fx in the Queries list. It can be renamed in the Query Settings just like a query.

The function can then be used to create a column in a table. From the Add Column ribbon tab, click Invoke Custom Function. Enter in a name for the new column and select the new function from the list.

This will add a column to your data. This example is a pretty useless example of just Hello World but tradition has been done and we are now ready to move to the next stage.

Adding a parameter to our Handwritten Function

The Hello World function just returned one value, which has a very limited use. So now we will add a parameter to our function and use that value to change our result to include that message.

Parameter definitions go between the brackets in the definition. I’ve updated the function to be this code and on pressing return will now have details of the parameter required.

1

= (GuestName) => "Hello " & GuestName

If the function is being used anywhere you will need to modify the function. You can either remove and re-add the function or click the cog wheel on the relevant step. There are two possible windows you will get on clicking the cog wheel.

Editing a working custom function column

Editing a custom function column that returns an error

Your table will now show the result of your updated handwritten function.

Data Types

I can feel my degree lectures turning in their graves as the above function has no data types specified. You can find many online references as to why adding data types into code is good practice, for this post please accept it is good practice and good practice usually leads to cleaner and faster code.

The fnHelloWorld has a parameter that is text and it returns some text. So we can adjust the code with “as text” after the GuestName parameter and also after the ). It won’t change the function much but it will in the long term be a great habit to adopt.

Quick Look at If

One of the most used constructs in programming in any language must be the if statement. In order to help you build quick easy functions it is worth noting the syntax.

1

if [TEST] then [TRUE RESULT] else [FALSE RESULT]

So here is a very simple example of a function that calculates a delivery cost based on if the value is over 100 the delivery is free.

Optional Parameters

Last topic in this post is to make parameter optional and handling when it is not provided. Optional parameters must be the last parameters defined in your function.

So in my example I’m going to add a column that calculates tax based on the value and the optional tax rate. If no tax rate is given 20% is assumed.

The parameters have to be withoutTax value followed by taxRate as the taxRate is optional. We can test for no taxRate being given by testing for taxRate = null. So our function is

1
2
3
4
5


= (withoutTax as number, optional taxRate as number) as number => 
 if taxRate = null then 
 withoutTax * 0.1 
 else 
 withoutTax * taxRate

Adding a tax column to example data shows how Alex’s tax rate is null so the tax is calculated at 10% and Jo’s tax is calculated at the given rate of 20%.

Conclusion on Handwritten Functions

These are very simple examples of functions but even these can save hours of repeating calculations. When your data has the same cleanup calculations being done to your data over many queries this allows for one function to be used repeatably.

Resources

Power Query in Flow – Summarising the Data

Mon, 23 Sep 2019 08:33:01 +0000

This is the third post in my series regarding Power Query in Flow. In this post I will look at the different options we have for summarising the data using grouping and pivoting with Power Query.

Here is a list of all the posts in the series so far.

Introduction

Again this post might be redundant if you’ve followed my advice and found out your dba’s favourite biscuits and got them to write you some awesome views in the database.

For those that haven’t got access to the dba lets look at how we can summarise the data. We will look at two different types of summarising the data, firstly grouping and then pivoting. I have assumed you’ve got your data into a simple table using merges etc.

Using Grouping

In this example we will group by a column and include some summary values. We will group by Product and then summary values of count of orders, total quantity and total value.

This is how the data starts.

On the toolbar click Transform Table and select Group By. We will start with a basic grouping of just one column to group by and one summary column of Order Count.

I selected ProdName for the group by , entered a name for the summary column to give Order Count by counting the rows.

To add more columns to the summary I click on the cog wheel on the Grouped rows step to re-open the group by dialog.

I click on Advanced to open up options to add extra grouping and aggregation options. I click on Add aggregation to add extra columns and change Operation to Sum and Column to total up Qty and Value. Clicking OK updates the table.

Using Pivoting

Pivoting is a form of grouping that splits the data across a number of columns, for example total sales for each product across hours of the day. The first step is to reduce your table into just the data you need. So include the data for each row title, one set of data to be the column headings and one to be the data summarised.

So I do Select Columns to reduce down to just Product, Hour of Sale and Qty of items sold.

Next, select the column of data that will make the column headings in your pivot. In my example I select the SaleHour column. Then from Transform Column, select Pivot Column.

In the Pivot column dialog, click on Advanced to reveal more options. In the Value column select the value you want to summarise and select the function you to use. Click OK to pivot the data.

Conclusion

Power Query provides a powerful way for summarising the data ready to process within the flow. As I stated earlier a view written by a dba would be more useful and quicker.

AI Builder Object Detect Model with Power Automate and Power Apps

Mon, 09 Sep 2019 09:38:06 +0000

This series of posts is to introduce using an AI Builder Object detect model with Power Automate to get and save predictions and then display these predictions over the images in a Power App.

Build the Model
Save Predictions
Use Predictions in a PowerApp

Introduction

In this post I will document building an AI Builder object detect model. We will create the list of items to detect, upload images, identify each image and then train and publish the model. I will be using 2 different types of Lego bricks, 1×2 block and 2×2 block.

This post will be split into multiple parts

Create Entity to store object names
Create Model
Upload and Tag images
Train Model

Create Entity to store object names

Before we can start building model we need to create a list of object names stored in the Common Data Store (CDS) in an entity. We need to create the entity and load object names.

We start in Flow.Microsoft.com and then on the left hand menu click Data and then Entities.

From the top ribbon select New entity which will open a pane on the right hand side. Enter a Display Name for the entity. It will complete the other fields for you.

Click Create to create the entity.

The next stage is to add data to the entity. The easiest way to do this is via Excel. Click on the Edit Data in Excel. A file will be downloaded which you need to open in Excel.

You might need to enable the feature to get the pane to the feature in Excel. Enter in the Object Names and then click Publish to save the values to your entity.

Once the values have been published you can check back in your entity by back on the website and clicking on Data.

If Excel won’t play ball, and for one client it just won’t, then you can use a very simple flow to add the entries.

Create the AI Builder Object Detect Model

On the left hand menu select AI Builder and then Build. The AI Builder currently offers 4 types of model. For this post we are going to build an Object detection model, so click on Object Detection. Then enter a name for your model and click Create.

Select Object Names

The next stage in building the model, is to select the names to be used in the model. Click on select object names. From the panel select the entity you created and tick the Name column and click Select Fields.

On the next screen select the values and click Next.

Add Images

After the object names have been selected you then need to add images for the model to be trained from. Click on the Add Images and this will open a file selection window, select files and click Open.

On the next window it will show all the selected files. Click on Upload to start the upload. When the upload has completed click Close to move onto the next stage.

The next window will show you a summary of your uploads. You can click on +Add Images to add more images to improve your model. You need at least 15 images but more is recommended. Click Next to start tagging your images.

Tagging Images

The next stage is tagging the images. I recommend a cup of tea, some music because this takes time.

Click Next to start the tagging. Thumbnails of the images will be shown. Click on an image to enlarge that image and start the tagging. You can drag a box of an object or it will suggest objects with a white dotted box. Select the tag to apply.

On the right is a summary for this image and on the top ribbon you can choose to not use an image. When you click on the arrow it will move you onto the next image.

When you have tagged all the images, click on Done Tagging in the top right hand corner.

Train Object Detect Model

When you click Next you get shown the details of your model so far. Click on the Train button to start the training.

Training takes a while, the above one took 5 minutes to train. I assume its longer when there are more images and more tags. Eventually your list of models will show a time of being trained.

The last step is to publish your object detect model. Click on the model name in the model list and it will show you a score for the training and give you a Publish button. Click the button to publish your model ready to use.

Conclusion to Object Detect Model

This is a new feature still in preview, so it has its clunky parts, but it brings machine learning firmly into the hands of the citizen developer without any Azure bills.

I really hope in 6 months time I need to write this post again because they’ve added new features and made it slightly less clunky.

Joining Tables of Data in Flow’s Power Query

Fri, 06 Sep 2019 20:00:01 +0000

This is the second post in my series regarding Power Query in Flow. In this post I will introduce one of the most powerful parts of Power Query, joining tables. Its the vlookup part of Power Query.

Here is a list of all the posts in the series so far.

Introduction

If you have made best buddy friends with your dba (database administrator) then this post is possibly not required because if your dba will write a view to do the join for you, that is always better. BTW in my experience, dbas like biscuits.

For those who don’t have access to their dba or haven’t bought enough biscuits this what we are going to do. My database contains a table called productsales.

In the Product Sales table is a ProductID column. I am going to use this value to merge in the values from the Product table (i.e. joining tables) so I can get the name and price.

Multiple Tables

We start by adding both tables to our query. If multiple tables are in a query, Power Query only returns one table. You can tell which table will be returned by looking at the list of tables. The tables with italic names will not be returned, therefore the un-italicised table will be returned.

Right click on a table name to open the menu. Then select Enable Load to change the table that will be loaded. In this example I will select that productsales will be loaded.

Merging tables

We want to add the product name and product price to product sales tables. We will do this by merging the two tables.

Select the productsales table. Then from the toolbar, select Combine Tables and Merge Queries.

In the Merge dialog that appears select table you wish to merge with in the drop down, in my example I select products table. Then click on the columns that match, in my example I select product id in both tables.

Leave the join kind as Left Outer as this will keep all the sales records and find matching products. Further details of join types can be found on Radacad’s site. https://radacad.com/choose-the-right-merge-join-type-in-power-bi

Expanding the table column

Clicking OK will update the table to include a column which contains Tables.

Click on the icon in top right of the new column to display the columns available in the tables. Unselect any un-required columns. Unselect the Use original column name as prefix to prevent the new columns being named long names for example products.ProdName. Then click OK to add the new columns.

Conclusion to Joining Tables

Merging is a powerful addition to transforming data. It is better if a view can added to the database by your dba as that will always be more efficient. Power Query merges do offer a great alternative.

Power Query in Microsoft Flow – An Introduction

Mon, 26 Aug 2019 21:34:42 +0000

This is the first post in a new series regarding Power Query in Microsoft Flow. In this post I will cover the basics of connecting to SQL Server and using Power Query to filter and sort the data and finally add calculated columns.

This series assumes that you are not a user of Power Query desktop from Excel or Power BI.

Here is a list of all the posts in the series so far.

Introduction to Power Query in Microsoft Flow

In September 2018 Microsoft announced the integration of Power Query Online for Microsoft Flow. The reasons were to offer an alternative to OData and SQL for getting data from SQL Server connector. They included a hint that other data sources will come in the future but no details when or what.

For those coming from Power Query for Excel or Power BI you need to understand you are currently limited to data stored in SQL Server online or server via a gateway. Also the Power Query Online environment is not as powerful as the desktop app version. It is still written in M, so you can do many of the transformations you can in Excel and Power BI’s Power Query.

Connecting to the Database

Our first action is to connect Power Query to the data. Power Query action is from the SQL Server connector.

If this is the first time you’ve connected to the database you will be prompted for database details and the user id to use with a password.

Then the action then changes to just show a button. This means you have connected to the database successfully.

Selecting Data to Transform

When you click the Edit Query button for the first time, you will be presented with the tables and views in the database. You need to tick the tables and views that you need. For this post I’m just going select a single table Expenses and then click Next.

The next window is the main Online Power Query window, where we will add steps to transform your data.

List of parameters and tables
View of the data at the current step
List of steps
Toolbar used to add new steps

Filtering Data

One of the most common requests on data is to filter the values. Clicking on the arrow next to column header will show you a list of values with tick boxes. Un-tick the unrequired values to filter your data. When you do this it will add a new step to the applied steps.

When you want to change the filtered value, you need to click the cog wheel next to the step. You can click on X to remove any step.

Sorting Data

When you click the drop down arrow next the column heading, you can select Sort ascending or Sort descending. The first time you select sort, the data gets sorted and a new step is added to the applied steps.

If you then sort another column, the existing step is just extended to add the next column to the sort. This is similar to adding a new sort level in Excel.

Adding Columns

One of the most powerful parts of Power Query is to add columns using calculations. The interface can be really clever and also limited and drives me nuts which is perhaps a blog post all of its own.

If you select a column and then click Add Column from the top ribbon you will get offered choices based on the column type. For example, if you have selected a number column you get offered Standard, Scientific and Trigonometry functions and rounding plus others.

The result of the calculation is placed in a new column.

When you select multiple columns before selecting Add column, calculations to combine the columns are offered, such as difference between dates.

Renaming and Selecting Columns

You can rename any column by double clicking on the column header and typing in a new name. The first column you rename will add a new step to the query. If you rename another column straight after the Renamed Columns step is extended.

It is always a good idea to only return the columns that are required by the Flow. Under Manage Columns you can select Choose columns. When the dialog is selected, you can tick or un-tick any column.

The choose columns step has a cog wheel so can be modified at any point. I recommend you use this step rather than delete columns as that cannot be tweaked using the cog wheel.

Finishing

When you have finished transforming your data you can click Create on a new query or Update if you are editing an existing query. You can now use the data returned just as you would for getting items from any source.

Resources for Power Query in Microsoft Flow

As always there are a selection of resources out on the web.

•Microsoft https://flow.microsoft.com/en-us/blog/powerquery-flow/

Chris Webbhttps://blog.crossjoin.co.uk/2018/09/26/using-power-query-and-microsoft-flow-to-automate-the-creation-of-csv-files/

Erik Svensenhttps://eriksvensen.wordpress.com/2018/09/25/powerquery-everywhere-now-in-microsoftflow-as-well/

Conclusion

Power Query is a great addition to the Flow tool set. I am really hoping for more data sources to be allowed and for the online Power Query to include more features from the desktop version.

This is just an introduction so I will include best practices and further features.

SVG in Power BI Part 7 – Using a Theme File to add SVG Icons

Tue, 13 Aug 2019 13:01:24 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

Introduction to SVG Basics
KPI Shapes in Power BI
Filling up with colour using SVG in Power BI
Using Text in SVG
Using SVG Rotate to create a dial in Power BI
SVG Icons in Conditional Formatting
Using a Theme to add SVG Icons
Feb 2023 Update - 5 SVG Stars

As part of the July 2019 update there were changes to conditional formatting and I covered the use of SVG based measures to add icons in part 6 of this series. In this post I am going to cover adding to or swapping the built in icons using a theme file and some SVG.

Overview of Theme Files

Theme files are a JSON file with instructions for formatting your visuals. The formatting can include colours, fonts and now icons. Details of how to construct and use a theme file can be found in the links below.

https://docs.microsoft.com/en-us/power-bi/desktop-report-themes

https://powerbi.tips/tools/report-theme-generator-v3/

Adding New SVG Icons

Icons is a new section for the theme file. It is at the same level as the name. You need a name, url and description for the icon. I can see no evidence of the name or description being used. Please correct me if I am wrong.

The url can be SVG code in a similar format to the measures from previous posts. So in this example my first icon is a red quarter of a circle. Below is the SVG code for the shape.

1

data:image/svg+xml;utf8,

A simple theme file to create one extra icon would look like

1
2
3
4
5
6
7
8
9


{
 "name": "New Icons",
 "icons": {
 "complete025": {
 "url": "data:image/svg+xml;utf8, ",
 "description": "25% Complete"
 }
	}
}

This adds a single icon to the icons available in conditional formatting. It will be listed at the end of the icon list.

I then expanded the list of icons to cope with 25%, 50%, 75% and 100% going from red through amber to green.After applying the the theme to my report I created conditional formatting rules to a table to use my new icons.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{
 "name": "Percent Icons",
 "icons": {
 "complete025": {
 "url": "data:image/svg+xml;utf8, ",
 "description": "25% Complete"
 },
 "complete050": {
 "url": "data:image/svg+xml;utf8, ",
 "description": "50% Complete"
 },
		 "complete075": {
				"url":"data:image/svg+xml;utf8, ",
				"description":"75% Complete"
		 },
 "complete100": {
 "url": "data:image/svg+xml;utf8, ",
 "description": "100% Complete"
 }
 }
}

Conclusion

Adding a few extra icons to report is great. I am concerned that adding lots of complex icons will just make the report slow.

In Microsoft’s announcement they say you can replace existing icons using a theme file. For this you need the resourceKey for the icons. I have not been able to find a list of icons and the example they give didn’t work for me.

Power Automate – Creating Adaptive Cards

Wed, 07 Aug 2019 07:09:31 +0000

This post is to support a video posted on YouTube regarding creating adaptive cards for Microsoft Teams using Power Automate.

Designing an Adaptive Card

Before you can post an adaptive card to a team you need to design the card. Cards are designed online on an interactive website that will build the JSON definition for you. In this post we will create a very simple card.

::: info Instructions

Visit https://adaptivecards.io/
Click on Designer on the top navigation
Click on New Card
Drag TextBlock elements from the options on the left into the Card. Double click on the text to change the text.
Make changes to the layout and style as required. The JSON defining the card will be updated.
Click Copy card JSON to copy the code to the clipboard.

:::

Other elements could be added to the card and formatted as required.

Writing the Flow

For this post the flow is going to be very simple. I start by adding a button trigger with a single string input called Message.

Under the Teams connector, I find the Post your own adaptive card as the Flow bot to a channel. I select the Team and Channel that I want the message to go to.

In the message I paste the Adaptive cards code copied earlier. I double click on the MESSAGE and then click in the Dynamic Content pane on Message to replace the text.

Running the Flow

The flow can be run from your phone and the Flow app. Typing in a message will add the card to the channel in Teams.

Conclusion

This is a very simple example of an adaptive card being posted in Teams. The possibilities are endless and are a great next step up from sending email notifications to multiple people.

SVG in Power BI Part 6 – New Icon Conditional Formatting

Tue, 06 Aug 2019 10:37:05 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

Introduction to SVG Basics
KPI Shapes in Power BI
Filling up with colour using SVG in Power BI
Using Text in SVG
Using SVG Rotate to create a dial in Power BI
SVG Icons in Conditional Formatting
Using a Theme to add SVG Icons
Feb 2023 Update - 5 SVG Stars

In July 2019 Power BI Desktop update they included new conditional formatting on table and matrix to include icons. These icons can be defined using SVG which is pretty cool. So this is a new post in my Power BI SVG series looking how we can use SVG icons in conditional formatting.

Quick Overview of Icon Conditional Formatting

Once you have added a table or matrix visual to your report you can in formatting in the Conditional Formatting Section turn on Icons. This will immediately add icons to the selected column. In the example below icons were added to Turnover column.

Clicking on Advanced controls link opens options behind the selection of icons. The default rules are for 3 icons at

In this post though I’m looking at using SVG to define and select the icon. So I will not be using rules but will be using a measure. The example is going to be based on a measure written in a previous post, KPI Shapes in Power BI The measure decides on an up or down arrow based on another measure called Growth.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


Arrow = 
// Variables to store arrow paths
var uparrow = ""
var downarrow = ""

// Select arrow to use
var arrow = IF([Growth]>=0,uparrow,downarrow)

// Insert arrow into SVG
var svg ="data:image/svg+xml;utf8," &
 ""
 & arrow & 
 ""

RETURN svg

The arrow is in its own column and doesn’t resize with font sizes. With the Icon update we can add the icon to any column.

Select your table visual
In visual format, expand Conditional Formatting
Select the column and turn on Icons
Click on Advanced Controls
Change Format to Field Value
Change Based on Field to the measure, eg Arrow
Click OK

Conclusion

Icons update is an exciting one and much needed. It cuts out the need for SVG for many, which is great. Now with a little SVG we can add some conditional formatting without having to add the extra column.

Power BI – DAX CrossFilter Function

Sat, 27 Jul 2019 07:59:23 +0000

In this post I am continuing my campaign against bi-directional relationships. This is an introduction of the CROSSFILTER function, which can be used to simulate the bi-directional relationship in a calculation.

The Problem

Yesterday I posted regarding Cascading Filters and removing the need for bi-directional joins to filter slicers. Another reason I’ve seen users using the bi-directional filter is to do a calculation.

Using the same example as yesterday, see the link above, I created a measure to count the number of products.

Then added a card to count the number of products. With no filters applied it shows 7 products which matches the table and the slicer.

Selecting Daisy Delights from the Shop slicer, the table reduces to 5 rows, and after yesterday’s post, the Product slicer also reduces to 5 rows, but the Product Count card still shows 7.

The big bad bi-directional join would be one method to solve this, but that is not good practice and I’m not suggesting that.

CROSSFILTER Function

The CROSSFILTER function can be used within a CALCULATE function to specify a direction for relationship to be used between 2 tables. The options for the directions are None, One or Both. So I use this in the measure

Now when I select Daisy Delights on the report, the measure shown in the card changes to 5. The card value matches the other filtered elements.

Conclusion

Keeping data models as simple is vital to allow for reports to be kept maintained in the future. CROSSFILTER function is a great addition to the options for calculating measures.

Microsoft’s documentation can be found at https://docs.microsoft.com/en-us/dax/crossfilter-function

Power BI – Cascading Slicers

Fri, 26 Jul 2019 08:00:23 +0000

In this post I will look at filtering one slicer based a selection in another slicer a.k.a. cascading slicers. June 2019 Power BI Update included an update to allow a slicer to be filtered based on a measure, which means that you no longer need to use bi-directional relationships.

Slicer Series

This series will cover different topics regarding slicers.

Slicers Introduction
Resetting Slicers with a Bookmark Button
Cascading Slicers
Hierarchy Slicer
Sync Slicers
Clear all Slicers Button
Relative Date Slicer

Problem Overview

My simple example is a report showing sales of a product in different shops. The example report contains three tables, Products, Shops and Sales and I’ve related the tables as shown below.

Note the relationships show using arrows that Products can filter Sales and Shops can also filter Sales, and Sales cannot filter either Shops or Products. So I create a report with 2 slicers, a table showing product sales and a card showing total sales.

If I use the Shop Name slicer to filter to Daisy Delights the card and table filter to only show the sales of the 5 products that Daisy Delights sell. BUT the slicer of products does not get filtered.

Creating the Measure

So we want to create a measure to indicate if there are sales records. My initial reaction was to create a boolean (True / False) measure using ISEMPTY on the Sales table something similar to

Puzzled I went looking, and of course the best site SQLBI gave me the answer to use. The links I used were:

https://www.sqlbi.com/articles/syncing-slicers-in-power-bi/

https://www.sqlbi.com/articles/check-empty-table-condition-with-dax/

That measure would be True if Sales records exist and False if no Sales records exist. Sadly this measure doesn’t work as a slicer filter so we need to convert the True and False to numbers 1 and 0, the fastest way to do this INT function. So the above measure becomes

Filtering the Slicer

We then need to add the measure to the slicer filters. I select the slicer and drag the measure into the filter pane. I then set the operator to is and the value to 1. Then I click Apply Filter and the list of products is now filtered to only 5 products. The Product slicer is now a cascading slicer and will filter based on shop selected.

Conclusion to Cascading Slicers

Even though I would prefer to use the boolean filter as that makes logical sense to me, this is a cool update and it reduces those nasty bi-directional filters.

Power BI – Using Inactive Relationships in a Measure

Thu, 25 Jul 2019 09:54:02 +0000

In this post I will show a simple example of how to use an inactive relationship between two tables in a measure. The example I will use is a list of tasks with start and finish dates. The request is for how many tasks start per week and how many finish per week to be plotted on a column chart.

YouTube Version

Data Setup

The data for this report is a list of tasks in an Excel spreadsheet with an Task Id, Task Name, Start Date, and Finish Date columns. I have also added a calendar table with a Date column and a Start of Week column.

In the relationships view I drag Date from Calendar onto Start Date in Tasks and a 1 to many relationship appears as expected. That will allow me to count the tasks starting.

I also want to be able to count the finishing tasks, so I need a relationship to Finish Date from the calendar as well. So I drag the Date from Calendar onto Finish Date in Tasks. A relationship appears but this time it is dotted, which shows it is not the active relationship.

If you look in Manage relationships it will list both relationships but only one will be ticked as Active.

Creating the Measures

Now the relationships have been setup we can create the two measures to do the 2 different counts.

The first measure is the simplest and will use the active relationship between Calendar[Date] and Tasks[Start Date]. It is just a count of the Tasks Id column.

For the second measure to count the number of tasks finishing we need to use the other other relationship. We can do this by using the Calculate function and the UseRelationship function. The UseRelationship function allows you to use an inactive relationship by specifying the two related columns. It must be an existing relationship in the model.

1
2
3
4
5


Tasks Finishing = 
 CALCULATE(
 COUNT( Tasks[Task ID] ),
 USERELATIONSHIP( 'Calendar'[Date], Tasks[Finish Date] )
 )

We can now add these measures to a column chart to get a plot to compare starting to finishing tasks.

Conclusion

The USERELATIONSHIP function is a powerful addition to your creating measures tool-set. It does have some restrictions and these are detailed in the official documentation at https://docs.microsoft.com/en-us/dax/userelationship-function-dax

Power BI – Data Refresh Date

Tue, 09 Jul 2019 07:55:29 +0000

In this post I will walk through adding a data refresh date on a report page. This is a vital part of the report as it will show the report consumers when the dataset was last refreshed and so how up-to date the report is.

We need a date that will only get updated on a refresh so it needs to be defined in Power Query. So under Get Data select Blank Query and Power Query editor will open.

Into the formula bar enter the following and press return.

1

=DateTimeZone.UtcNow()

The = and () are vital, I spent a good 30 minutes one late evening wasting time due to missing out the brackets. You will get a result of the time in UTC.

The next step is to convert the answer into Table, so click on To Table on the DateTime Tools – Transform ribbon.

Rename your query and the column and change the type of the column to Date/Time/Timezone and you now have a table with a single row of a refresh date in UTC.

After you select Close & Apply you will be able to add the Refresh UTCDate to a card to show the data refresh date.

Conclusion to Data Refresh Date

This date will only show the date of the last refresh, this does not validate your data as being accurate if you have some other processes happening to prepare the data outside of the Power BI environment.

PowerApps – Using JSON Function to Save Pictures

Mon, 08 Jul 2019 10:21:38 +0000

In this post I will be explaining how to use the JSON Function to simplify the saving of multiple pictures from a PowerApp to a SharePoint library using Flow. There have been many wonderful workarounds to do this, finally we now have a slightly less Heath-Robinson method.

Set up the PowerApp

I start with a blank app, canvas or phone layout. Because my main client uses Windows 10 tablet / laptops to take the pictures I add a camera control. Then I change the OnSelect property to store the picture into a collection. I include an index column so Flow can calculate a unique filename later.

I then add a blank gallery and change the Items to be the collection populated by the camera, e.g. ccPictures and to make a grid I change the wrap to be 3. I add an Image control to the gallery so as I take pictures it will populate the grid.

Then I take some pictures to populate my collection. Obviously everyone needs photos of their Lego character collection.

In order for the pictures to be passed to Flow to be saved to SharePoint I need to convert the collection into JSON. So I add a button with the following code. Please note the second parameter of the JSON function is IncludeBinaryData as we want to include the pictures which are binary data.

Building the Flow

In the previous section, I created the app to make the JSON string to be processed by a Flow. So I now I will create a new flow with a PowerApp trigger.

The first step to add is a Parse JSON action to convert the JSON into an array. In order for the parameter requested by Flow to be named nicely I rename the step to just “JSON” and then in the Content box I click on Ask in PowerApps. Now the parameter is called JSON_Content. Afterwards you can rename the action back to Parse JSON and it won’t rename the parameter.

For the Schema you need to understand JSON a little. The JSON you return will be multiple rows each with 2 columns, Picture which is a string and Index which is a number. So in a text editor I create small example JSON string with multiple rows of data and copy it.

Then in the Parse JSON action I click on Use sample payload to generate schema and I paste my example in, clicking Done will create the schema.

The next step is for every item in the array to save a picture. So we start with an Apply to each action which will use the Body from the Parse JSON step.

Then we add a SharePoint action to Create file. Site address and path you will need to get from the SharePoint destination. The File Name I create using the Index value.

The File Content uses an Expression. So click on Expression and find the function decodeDataUri. The parameter for the function is the Picture value, which doesn’t appear in the offered Dynamic content, so we need to use the Item() function and then add ?[‘Picture’].

Connecting PowerApp to the Flow

We now have a Flow ready to receive JSON data and save pictures. So we now just need to connect them. In my experience adding a flow to a button that already has code does not end well, I usually loose the existing code. So I add a second button and from the Action ribbon click Flows and connect up my Flow and pass in as the parameter the variable containing the JSON, e.g. vvJSON. Then I copy and paste this code into the button that creates the JSON string.

The app is now ready to test.

Conclusion

The above solution is a very simple example of how the JSON function could be used. Before this went into production I would do better file naming. I have tested the above app to take 30 photos and save them to SharePoint and it worked fine. I assume there is a limit to the size of the JSON.

The JSON function does allow a whole collection to be passed to Flow easily. I will be looking for other possible uses.

Power BI – Conditional Formatting Update

Thu, 13 Jun 2019 19:26:40 +0000

The June 2019 update for Power BI included a few updates for conditional formatting, for visual backgrounds, titles, cards and gauges. This post is my take on the updates and on how I prefer to do the logic behind choosing a colour.

Full details of the June 2019 update can be found at https://powerbi.microsoft.com/en-us/blog/power-bi-desktop-june-2019-feature-summary/

Applying Conditional Colour Formatting

This update added to the places conditional colour formatting can be applied. In every place, the method of applying it is the same, the trick is working out which visuals will let you do what.

When you look at a visual’s formatting properties, right click on a colour selector. If Conditional Formatting appears then you can add rules if nothing appears then you can’t.

When it is applied, the colour is shown as Fx symbol.

List of Locations

Visual	Section	Property
Most Visuals	Title	Font color
Most Visuals	Title	Background color
Most Visuals	Background	Color
Stacked Bar Chart	Data colors	Default color*
Clustered Bar Chart	Data colors	Default color*
Stacked Column Chart	Data colors	Default color*
Clustered Column Chart	Data colors	Default color*
Line and Stacked Column Chart	Data colors	Default color*
Line and Clustered Column Chart	Data colors	Default color*
Scatter Chart	Data colors	Default color*
Treemap	Data colors	Advanced Controls
Funnel	Data colors	Default color
Gauge	Data colors	Fill
Card	Data label	Color
Card	Category label	Color
Table	Conditional Formatting	Background Color
Table	Conditional formatting	Font color
Matrix	Conditional formatting	Background Color
Matrix	Conditional formatting	Font color
Matrix	Conditional formatting	Data bars

Methods of Applying Conditional Formatting

There are three methods of applying conditional formatting: color scale, rules, and field value.

Colour Scale

Rules

Field Value

Colour Measures

One of my issues with Color scale or Rules is you cannot use multiple column values or measures in the colour choice. Field Value gives a work around for this by allowing you to use a Measure.

I have a very simple example of a list of Stores which have a Target and Sales value. So I create a measure that calculates a colour string based on Sales and Target using an IF function.

1
2
3
4
5


SalesColour = IF(
 SUM(Sales[Sales]) >= SUM(Sales[Target]), 
 "#006400", // Green
 "#BF0000" // Red
)

I then used Field Value conditional formatting to format a gauge fill.

Conclusion

Conditional formatting is getting there but it is still limited, and we have been spoiled by Excel. I expect more updates will come and I will need to update the table above.

SVG in Power BI – Part 5 – Using SVG Rotate

Sat, 08 Jun 2019 21:36:23 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

Introduction to SVG Basics
KPI Shapes in Power BI
Filling up with colour using SVG in Power BI
Using Text in SVG
Using SVG Rotate to create a dial in Power BI
SVG Icons in Conditional Formatting
Using a Theme to add SVG Icons
Feb 2023 Update - 5 SVG Stars

In this post I will introduce the possibilities that SVG rotate gives. This post will walk through creating a dial. This post will be done in 2 stages, first a simple dial and then add colours.

Simple Dial

We will start with a simple dial, made up of an arch with a line to indicate a percentage.

The arch is made using a path which combines 2 arcs to make an arch. The arch is then nested in a group element that applies a fill.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Simple Dial = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""
// arch shape
 var svg_arch = ""
// light blue arch
 var svg_blue_arch = "" & svg_arch & ""
return
 svg_start & svg_blue_arch & svg_end

We now need to add a line that will move based on a measure called Score using SVG rotate. So I start by calculating the rotation angle and drawing a line that matches the bottom left of the arch and then rotate it centered on the center point of the arch.

The rotation will be 0 to 180 degrees and the measure Score is a percentage so the calculation is [Score] * 180.

The line is drawn from 0,50 to 20,50 and then a transform is applied of a rotate which has 3 parameters, angle to rotate, x and y for the centre of the rotation. So the bottom part of my measure becomes

1
2
3
4
5
6
7


// light blue arch
 var svg_blue_arch = "" & svg_arch & ""
// draw line and rotate
 var svg_rotate = [Score] * 180
 var svg_line = "- "
return
 svg_start & svg_blue_arch & svg_line & svg_end

Show the Value

To make it very obvious what value is shown I’m going to add the value of the Score measure. The centre of the arch is 50,50 so I’ll put the middle of the text at 50,40. I used the FORMAT function to apply a percentage, “0%”, format to the score. (See the previous post in this series to look at SVG text)

The bottom part of the measure now becomes

1
2
3
4
5


var svg_line = ""
// Show Score value
 var svg_score = ""&FORMAT([Score],"0%")&""
return
 svg_start & svg_blue_arch & svg_line & svg_score & svg_end

Adding Colours

For this I researched being able to draw parts of an arch. In order to do that I would need to calculate the end points of each arch, which was possible but involved way more mathematics than I was willing to re-learn. My days of using Cos, Sin and Tan have long gone.

So I the trick I used was to draw the arch upside down out of sight and rotate the arch around into view. Different coloured arches can be rotated into view to show different colours.

To draw the arch up the other way I just changed 2 values in svg_arch variable. When drawing an arc in SVG the 5th number after the A specifies which way round the to reach the final point. So the svg_arch variable becomes.

1
2


// arch shape
 var svg_arch = " "

Then we add a coloured arch rotated for each part of the gauge. In my example I am going to show Green upto 100%, Amber starts at 75% and Red at 50%. So the complete measure code now is

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Coloured Dial = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""
// arch shape
 var svg_arch = " "
// coloured arches
 var svg_green_arch = "" & svg_arch & ""
 var svg_amber_arch = "" & svg_arch & ""
 var svg_red_arch = "" & svg_arch & ""
// draw line and rotate
 var svg_rotate = [Score] * 180
 var svg_line = ""
// Show Score value
 var svg_score = ""&FORMAT([Score],"0%")&""
return
 svg_start & svg_green_arch & svg_amber_arch & svg_red_arch & svg_line & svg_score & svg_end

Conclusion to SVG Rotate

This post ended up being longer than I expected so my series will expand to make use of a table to store the colour values in another post and drawing a clock into another post.

PowerApps – Transparency Update

Fri, 07 Jun 2019 17:53:58 +0000

A small update was sneaked in this week to help handling transparency with colours in PowerApps.

ColorValue Function Update

ColorValue function takes a colour value as a hex string and converts it into a colour values. In the example below #FF0000 fills a rectangle in Red.

The update now allows another 2 characters to the string to specify transparency. Like the red,green and blue values the transparency ranges from 0 to 255 or in hex 00 to FF

In the example below the red is made 50% transparent, 80 in HEX is 128 in decimal which is 255 / 2, hence 50%.

Transparent Colour

In the list of available preset colours such as AliceBlue, Aqua etc now includes a colour Transparent.

Conclusion

Small update but one thats very welcome! Now I just want to be able to use ColorFade or similar function to take an existing fill and give a more or less transparent version.

SVG in Power BI – Part 4 – Adding SVG Text

Tue, 21 May 2019 23:13:44 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

This is the fourth post in my series on using SVG in Power BI introducing adding SVG text and actually solves one of the niggles I had with doing KPI figures in Power BI. I didn’t get to do this part as part of my session which started this series. Here is the list of posts.

My niggle was the KPI visual. It is a fast way to have a red or green measure displayed. BUT, it requires a trend axis even if you don’t want one and it includes a tiny red exclamation mark or green tick. So I wanted the red / green and the goal shown with no trend or ticks and exclamation marks.

Simple Text Element

SVG Text elements need an x and y attributes to position the text and a style attribute that uses CSS notation to choose font properties. The example below has the word Hello where the bottom lower left corner is at 20,60.

1
2
3
4
5
6
7
8


Text Measure = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""
// text element including a style tag
 var svg_text ="Hello"
return
 svg_start & svg_text & svg_end

Aligning SVG Text

If I am going to replace the KPI visual I need to place a number centred in the SVG graphic. The default alignment is the x and y are the bottom left. This can be changed using the text-anchor attribute for the text element; it can be set to start, middle or end.

Below is an example of the string ABC being aligned start, middle and end.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Text Measure = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""
// text element including a style tag
 var svg_text1 ="ABC"
 var svg_text2 ="ABC"
 var svg_text3 ="ABC"
return
 svg_start & svg_text1 & svg_text2 & svg_text3 & svg_end

My KPI Visual

To replace the KPI shown at the start of this post I want a number in the middle of the square showing the measure, bottom left the goal and bottom right how close to the goal we were.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


My KPI = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""
// 3 text elements, actual, goal and % close
 var svg_Indicator =""
 & FORMAT([Total Sales],"#,##0") & ""
 var svg_goal = "Goal"
 & ""
 & FORMAT([Target Sales],"#,##0") & ""
 var svg_close =""
 & FORMAT([KPI Close],"0%") & ""
return
 svg_start & svg_Indicator & svg_goal & svg_close & svg_end

Format function is used to format the number into comma, #,##0 and percent, 0% formats.

The last part is to colour the number based on the Indicator reaching the target. This was covered in Part 2 of this series so I’ll just give the final code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


My KPI = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""
 
// calculate colour for element
 var ind_colour = IF([Total Sales]"
 & FORMAT([Total Sales],"#,##0") & ""
 var svg_goal = "Goal"
 & ""
 & FORMAT([Target Sales],"#,##0") & ""
 var svg_close =""
 & FORMAT([KPI Close],"0%") & ""
return
 svg_start & svg_Indicator & svg_goal & svg_close & svg_end

My KPI in a Table

One advantage of the SVG text KPI over the KPI visual is it can be embedded within a table. The total line will work as long as the number measures work.

Conclusion

Adding SVG text to a visual does make the visual more informative and the above example makes conditional formatting and data layout possible in a table.

SVG in Power BI – Part 3 – Fill SVG Icon with Colour

Mon, 29 Apr 2019 22:30:35 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

Introduction to SVG Basics
KPI Shapes in Power BI
Filling up with colour using SVG in Power BI
Using Text in SVG
Using SVG Rotate to create a dial in Power BI
SVG Icons in Conditional Formatting
Using a Theme to add SVG Icons
Feb 2023 Update - 5 SVG Stars

In this post we will introduce using icons and filling the icon with colour to show a percentage value.

Getting Icon Code

You can download icons from lots of web sites. Search for SVG icons or ask your marketing teams for company graphics as SVG. If you are using Office 2016 or above the icons in PowerPoint can be saved as SVG files. Icons can be found on the Insert ribbon and once inserted on a slide, right click and save as a picture.

Once you have the SVG icon take a look at the SVG code. There will be an open tag with lots of attributes then one or more elements and then a closing tag . You need to do a search replace the double quotes into single quotes, e.g. and its long so I save it into a measure.

Draw 2 Elephants

We are going to draw 2 elephants, one grey and one coloured one on top. The coloured one we will clip to size to show it being filled up.

Next task is to create a measure in our report. Inside the measure we create 2 variables, one to store the svg start and one for the svg end. As a quick test we return the concatenation of the variables and the previous created measure to draw our icon.

1
2
3
4
5
6
7


Elephant Measure = 
// svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""

return
 svg_start & [EleSVG] & svg_end

1

If you use the custom visual Image from Cloud Scope you can add the measure and the icon will be shown in Power BI.

In order to draw a grey and red elephant we will nest the elephant in a group tag and give the element properties that the nested elements will inherit.

I add 2 more variables one for the grey elephant and one for the red one. and tweak the return to use the new elephant variables. It will draw both elephants but they are exactly over each other so you only see the red one.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


Elephant Measure = 
 // svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""

 // Coloured Elephants
 var grey_ele = "" & [EleSVG] & ""
 var red_ele = "" & [EleSVG] & ""

return
 svg_start & grey_ele & red_ele & svg_end

Clip the Red Elephant

We now want to only show a percentage of the red elephant based off a percentage measure, e.g. work complete. For this we need to create a clip-path. A clip-path defines which part of the shape will be visible. So in this case it will be a rectangle starting at 0,0 and will be height = 100 and width = percentage measure * 100.

So we create a variable to calculate the width and then a variable to store the defs tag that contains the clip-path. The red_ele variable is then tweaked to include a reference to the clip-path and the return is changed to include the defs variable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


Elephant Measure = 
 // svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""

 // defs
 var clip_width = [Work Complete] * 100 
 var defs = "
 
 "

 // Coloured Elephants
 var grey_ele = "" & [EleSVG] & ""
 var red_ele = "" & [EleSVG] & ""

return
 svg_start & defs & grey_ele & red_ele & svg_end

Now we have 2 measures, one that calculates a percentage and one that shows that percentage in colour.

Filling from the bottom up

The above example fills from the left. It is the easiest to calculate but if someone was asked to draw this idea they would probably fill in the colour from the bottom up.

Our icon is in a square which goes from 0,0 and is 100 wide and 100 tall. If we wanted to show 25% full the clip path would need to be 0,75 and 100 wide and 25 tall. So 2 values change the height is 25 = 100 * 25% and the starting y is 75 = 100 – height.

So to fill from the bottom I need to change the defs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


Elephant Measure = 
 // svg essentials
 var svg_start = "data:image/svg+xml;utf8,"
 var svg_end = ""

 // defs
 var clip_height = [Work Complete] * 100 
 var clip_y = 100 - clip_height
 var defs = "
 
 "

 // Coloured Elephants
 var grey_ele = "" & [EleSVG] & ""
 var red_ele = "" & [EleSVG] & ""

return
 svg_start & defs & grey_ele & red_ele & svg_end

Conclusion

This gives us a quick visual measure to show a percentage using an SVG icon.

SVG in Power BI Part 2 – KPI Shapes

Sun, 28 Apr 2019 21:59:41 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

Introduction to SVG Basics
KPI Shapes in Power BI
Filling up with colour using SVG in Power BI
Using Text in SVG
Using SVG Rotate to create a dial in Power BI
SVG Icons in Conditional Formatting
Using a Theme to add SVG Icons
Feb 2023 Update - 5 SVG Stars

In this post I will show to use SVG to create simple KPI Indicators. In this example I have very simple data of a list of 5 shops with Turnover and Costs. I have added a measure for Profit Margin as a percentage.

Create SVG Measure

We shall start with a simple SVG image of a dot and show how to create an SVG measure to use in a table.

Create a measure to store the SVG in a string.

1
2
3


RedDot = "<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'> 
 <circle cx='50' cy='50' r='40' fill='red' /> 
 </svg>"

Prefix the SVG with code to declare that it is an SVG image.

1
2
3
4


RedDot = "data:image/svg+xml;utf8," &
 "<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'> 
 <circle cx='50' cy='50' r='40' fill='red' /> 
 </svg>"

On the Modelling ribbon tab, change the measure’s data category to Image URL.

Insert measure into a table. You will need to turn off the totals to make the total line not show svg code. I will fix this in a later post.

Change to be Reactive

The above example puts a red dot on every line. We will now adapt the measure to change the colour depending upon the measure Profit Margin measure to make this into a KPI shape.

We use a variable to store the colour and its value comes from an IF statement.
We insert the colour into the SVG statement
We return the SVG

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


KPI Dot = 
// Calculate the colour
var colour = IF([Profit Margin]>0.25,"Green","Red")
// Insert colour into SVG
var svg = "data:image/svg+xml;utf8," &
 "<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'> 
 <circle cx='50' cy='50' r='40' fill='" & colour & "' /> 
 </svg>"

RETURN svg

Up and Down Arrows

The above example changed a colour which is okay but it would be better if we could also change the shape of the icon. For this example I use another measure of Growth which compares Turnover to previous turnover. If the Growth is positive I want to display a green up arrow and if negative a red down arrow.

Set up variables for an up and down arrow
Select which arrow to use
Insert the arrow into the svg
Return the svg.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


Arrow = 
// Variables to store arrow paths
var uparrow = "<path d='M10 90 L50 10 L90 90 Z' fill='green' />"
var downarrow = "<path d='M10 10 L50 90 L90 10 Z' fill='red' />"

// Select arrow to use
var arrow = IF([Growth]>=0,uparrow,downarrow)

// Insert arrow into SVG
var svg ="data:image/svg+xml;utf8," &
 "<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'>"
 & arrow & 
 "</svg>"

RETURN svg

Conclusion

This gives an introduction of using SVG within Power BI to create KPI shapes. There is lots more that could be done with SVG and this series will continue to add more features using svg.

SVG in Power BI - Part 1 SVG Basics

Mon, 22 Apr 2019 17:23:04 +0000

Series

This is part of a series on using SVG in Power BI. The series supports my session at Power BI Days in Mechelen, Belgium in April 2019.

Introduction to SVG Basics
KPI Shapes in Power BI
Filling up with colour using SVG in Power BI
Using Text in SVG
Using SVG Rotate to create a dial in Power BI
SVG Icons in Conditional Formatting
Using a Theme to add SVG Icons
Feb 2023 Update - 5 SVG Stars

This post introduces the very basics of SVG. If you already know SVG, move along to my second post in this series.

SVG Basics

Before I get into the technical details of what SVG is, let me explain why SVG images are important. If you enlarge many image formats enough, they become fuzzy and pixelated. Because an SVG image is made up of lines and shapes defined by points, sizes, and colours, you can stretch them as big as you like and they stay sharp.

SVG stands for Scalar Vector Graphics, which is an XML-based markup format that can contain two-dimensional vectors.

Elements can be defined with an opening tag and a closing tag, for example <svg>...</svg>, or in a single self-closing tag, for example <rect ... />.

Example

1
2
3
4


<svg xmlns='http://www.w3.org/2000/svg'
 viewBox='0 0 100 100'>
 <circle cx='50' cy='50' r='40' fill='red' />
</svg>

The above example defines an area where the top-left is 0,0 and the bottom-right is 100,100. It contains a red circle with center at 50,50 and radius 40.

If an SVG statement contains more than one element, they are layered with the first item at the back and the last item at the front.

SVG Paths

The previous example drew a circle, and you can draw rectangles with <rect ... /> as well. The next step from SVG basics is drawing more complex shapes. For this, you need to define points, curves, and arcs. One method is the path element.

In a path element you can move the pen, draw lines, arcs, and curves. Below is a simple example that draws a green triangle.

1
2
3
4


<svg xmlns='http://www.w3.org/2000/svg'
 viewBox='0 0 100 100'>
 <path d='M10 90 L50 10 L90 90 Z' fill='green' />
</svg>

SVG Icons

Lots of icons available on the web are written in SVG and many can be downloaded. Be aware of copyright and always give credit where credit is due.

If you have PowerPoint you can insert icons onto a slide and save as a picture, which gives you SVG code.

SVG Resources

This has been a very short introduction to SVG. There are plenty of resources to assist in composing SVG creations.

W3Schools

W3Schools is a great resource to get started in SVG. The reason I return again and again is the Try It Yourself button, which gives a simple place to test your SVG.

Jenkov’s SVG Tutorial

This is a great site for learning different parts of SVG and how it works, including videos.

Conclusion

This was a brief introduction to SVG. It is a language worth understanding the basics of, and hopefully through this series you will find a new way to visualize your data.

Hat Full of Data

Humbled and Honoured

Using Variable Library in a Dataflow

Variable Library Series

Scenario

Getting the Value

Using the Variable Value

References for Variable Library in a Dataflow

Conclusion on using variable library in a dataflow

Using a Variable Library in Lakehouse Shortcuts

Variable Library Series

Setting Up the Shortcuts

Viewing the Options

Applying the Variables

Updating Variable Values

Conclusion on Variable Library in Lakehouse Shortcuts

Resources

Using a Variable Library in a Data Pipeline

Variable Library Series

Adding the Variables to the Data Pipeline

Using Variable Values

Using Variables for Data Connections

Connection IDs

Workspace, Warehouse and Lakehouse IDs

Connection String

Variable Library

Using the Variables in the Activities

Conclusion on Using a Variable Library in a Data Pipeline

Resources

Accessing a Variable Library in a Notebook

Variable Library Series

Connect to Library

Get Variable Value

Get a variable value by reference

Naming Conventions

Conclusion on using Variable Library in a Notebook

References

Happy New Year and 2025 Review

Welcome to 2026!

Blog Activity in 2025

YouTube Activity in 2025

Speaking

Looking Forward

Microsoft Fabric – Variable Library

Variable Library Series

Create a Variable Library

Add a Variable

Adding Notes

Saving

Conclusion

References

Microsoft Fabric – Using Copy Job with Dataverse

Table of Contents

Scenario

First Copy Job – Dataverse Connector

Create Copy Job

Choose Data Source

Choose Data

Choose Data Destination

Choose copy job mode

Map to destination

Review + save

Copy Job Run

Copy Job using SQL Server connection

New Copy Job using SQL Server

Choose Data

Incremental and Merge setup

Save, Review and Schedule

Incremental and Merge Running

Trouble Shooting

Alternative Options to using Copy Job with Dataverse

Link to Microsoft Fabric

Conclusion

Microsoft Fabric – Direct Lake and User Permissions

IMPORTANT

Scenario

Viewing Reports as Dan

Viewing Workspace App as Sam

Permissions

Fix Dan’s Lakehouse Access