Power Query on Hat Full of Data

Paginated Reports – Power Query as a Source

Tue, 08 Jul 2025 16:01:05 +0000

In March 2024 Microsoft added Power Query as a source for Paginated Reports. This means a paginated report can connect to any data source that Power Query can connect to. This post walks through the process with a few gotchas to be aware of.

Paginated Reports have had some Love

This post is part of the series covering my session titled Paginated Reports have had some Love. The resources for this series can be found at https://github.com/HatFullOfData/Sessions/tree/main/Paginated-Reports-have-had-some-love

Power Query as a Source
Create a report online
Dynamic Subscribing

Get Data

As with most reporting the first task is connect to the data using Power Query as a source. Inside a report in Report Builder, click on the data ribbon. Click on Get data to launch the Choose data source dialog. From here you can search and find lots of different data sources. For this post we are going to use an Excel workbook, so click on that.

When the connection settings dialog opens enter the File path or URL for the Excel file. There is a button to browse OneDrive, bad idea in my opinion but that is a different post! Please note you need to select the correct Authentication type. I’ve used Anonymous as my file is public on GitHub, you will need Organization Account for OneDrive or SharePoint.

Then click Next.

The next dialog, Choose Data, asks you to select the tables or sheets. Once you have ticked at least one you can then click Transform data to move to the Power Query window.

Power Query Editor

When the Power Query window opens you can add steps as per normal such as choosing columns etc. After you add all the transformations you need, click Create to add it to the report.

Power Query as a source in Explorer

When the data loads into the report it comes in 2 parts. Under data sources is Power Query and in datasets are the individual queries. Unlike other data sources such as a database each dataset will be a brand new query.

Editing the Query

When you need to edit the query, right click on the query and select Edit Mashup. It will always request you configure the connection. No Idea why but clicking the Configure Connection fixes the issue.

A few Gotchas

Multiple Queries

Each query can only return a single table. So when you have multiple queries, which one does it return? It returns the last one in the list of queries. It has nothing to do with marking a query for loading or not loading.

Refresh or not?

When you publish the report there is no semantic model. There is just a report. So every time the report is opened the Power Query executes. So lets keep those queries tidy and efficient.

Parameters

Parameters make Paginated Reports reusable and a powerful tool for the analyst. Power Query parameters though do not connect to the parameters in Report Builder. The Dataset properties show the M code, kind of. Its not editable so we can’t embed the Report Builder parameters. The only way to perform a filter is using the Filters in dataset properties which has the disadvantage that it filters after pulling all the data in.

Power Query as a Source Conclusion

I love Power Query, second to SQL, every data handler in the Microsoft stack should know it. But I don’t believe it brings any benefit putting it in report builder when I can do more from a semantic model. The gotchas aren’t a real problem on their own but they add up. So my recommendation would be to build the semantic model in Power BI desktop or in Microsoft Fabric and connect to that. That is a separate blog post, link up above.

Power Query – Handling Null Values Properly

Mon, 13 Jan 2025 10:53:43 +0000

This post is about handling null values in Power Query calculations. If you do a calculation in Power Query that involves a null value the answer returned is null. In the example below we get nulls for the first three rows because either Credit or Debit values are null. On the fourth row we get the answer 460 because Credit and Debit contain values.

1

= Table.AddColumn(#"Changed Type", "Total", each [Credit] - [Debit] )

This post gives you 2 methods on how handle them. I personally prefer method 2, but each to their own.

Method 1 – Replace Null Values

The most common method recommended is to replace null values with a value that works, so in the above example a 0. So before I add the above calculation I highlight both the Credit and Debit columns. Then I click replace values on the Transform ribbon.

The above works and is simple to explain and document. It does have the side effect that the average Credit and average Debit are now altered and filters to find just credits or debits will have to be different. It also takes 2 steps. The averages issue is the one that most people dislike.

Method 2 – Using the Coalesce operator with Null Values

Coalesce functions, DAX has one, return the first non-null value. In Power Query there is a coalesce operator ??. So Credit ?? 0 will return the credit value unless it is null and then will return 0.

So we can change our calculation to put ?? 0 after the column values and make it not return nulls. The Credit and Debit columns still have their original values for averages etc. It also is done within one step.

1

= Table.AddColumn(#"Changed Type", "Total", each [Credit] ?? 0 - [Debit] ?? 0 )

Conclusion

Power Query M is one of my favourite languages but its weird. So these posts are for purely selfish reasons to remind me how to do things. ?? is a really simple operator that solves so many problems and Microsoft give it 2 lines! https://learn.microsoft.com/en-us/powerquery-m/m-spec-operators. It could of course be chained to Column1 ?? Column2 ?? Column3 will give the the first non-null value from the three columns, and return null if they are all null.

Power Query – Fixing Missing Columns Dynamically

Wed, 08 Jan 2025 11:49:26 +0000

One of the challenges of making sure your query works in Power Query is data sources that have a changing schema. Some APIs miss out fields if the data is null and sometimes we need one report able to handle data files that are almost the same but not quite. In this post we will look at fixing 2 scenarios, a single missing column and multiple missing columns.

You know you have this problem when you get the an error similar to this:

Single Missing Column

This scenario is fixing one column missing. In my example the query was written based on a file that had a Debit column. The query is now pointed to file that is missing that column. We need the query to work if the Debit column is in the original or not.

Start in the broken version and find the first step where there is not an error. The fix is to add the Debit column if the table does not have the column.

M is not friendly to write so lets get the interface to write part of it. Click on Custom Column on the Add Column ribbon. Click Insert when prompted about inserting a step. Then enter in the name of the missing column for the column name and null for the value. When you click OK the column gets inserted and your query goes back to working.

But the query will fail when the file does include the Debit column. So we need to add some logic to the new step we added. We need to test to see if the table already has the column, if it does just return the table, if not add the column. This can be done using Table.HasColumns function and an “if” statement.

The line of M created by the inserting a custom column is

1

= Table.AddColumn(#"Promoted Headers", "Debit", each null)

This needs to become

1
2
3


= if Table.HasColumns(#"Promoted Headers","Debit") 
 then #"Promoted Headers" 
 else Table.AddColumn(#"Promoted Headers", "Debit", each null)

Please note the #”Promoted Headers” comes from the name of the previous step.

Reference for HasColumns can be found at https://learn.microsoft.com/en-us/powerquery-m/table-hascolumns

Multiple Missing Columns

The previous method works great for a single column and maybe even 2 but if you have more columns than that you really need another method. In this example we have a set of data with columns for months Jan – Dec. At the start of the year there is only a Jan column but we need the query to return all the columns.

The error will only complain about the first column it finds missing but if we fixed Feb it would then complain that Mar was missing etc. When you append 2 tables it creates a table with all the columns of both tables even if one table does not contain any data. So if we can create an empty table with all the columns we want and then append the tables we will get all the columns added.

Create a new blank query and enter in the formula to create a list of all the column names you want to make sure exist. Use {} brackets containing the column name strings separated by commas. This will display the list of values. Rename the query to something meaningful, for example Months.

1

= {"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"}

Now we need to add a step that converts this into a table. Right click on the Source step and select Insert Step After. Next we use #table function that takes 2 parameters, list of columns and values. We want an empty table so will use {} for the values. Change the formula to the following

1

= #table( Source , {} )

Reference for #table can be found at https://learn.microsoft.com/en-us/powerquery-m/sharptable

Return to the broken query and find the last step that the query works. On the Home ribbon select Append Queries and select Insert when prompted about inserting a step. In the Append dialog select the query that has the empty table just created. Then click OK. Your query now has all the columns required and should work if the columns are missing columns or not

Advanced Notes on Missing Columns

When appending 2 tables the first table columns will come first and in the order of the first table. So if you wish to order the columns of the query as well make sure your blank table query contains all the columns and the swap the order of the tables named in the Table.Combine step.

The list of column names and creating the blank table could be put as steps within the original query. That requires being comfortable with the Advanced editor and writing M. For those interested here is an example query.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


let
 Source = Excel.Workbook(Web.Contents(ExcelPath), null, true),
 Sheet2_Sheet = Source{[Item="Sheet2",Kind="Sheet"]}[Data],
 #"Promoted Headers" = Table.PromoteHeaders(Sheet2_Sheet, [PromoteAllScalars=true]),
 AllColumns = {"Project","Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"},
 BlankTable = #table( AllColumns , {} ),
 #"Appended Query" = Table.Combine({BlankTable, #"Promoted Headers"}),
 #"Changed Type" = Table.TransformColumnTypes(#"Appended Query",{{"Project", type text}, {"Jan", Int64.Type}, {"Feb", Int64.Type}, {"Mar", Int64.Type}, {"Apr", Int64.Type}, {"May", Int64.Type}, {"Jun", Int64.Type}, {"Jul", Int64.Type}, {"Aug", Int64.Type}, {"Sep", Int64.Type}, {"Oct", Int64.Type}, {"Nov", Int64.Type}, {"Dec", Int64.Type}})
in
 #"Changed Type"

Conclusion

Queries should be dynamic when required and missing columns is a common issue. Writing M is not friendly, writing dynamic M takes practice but it is worth it. Power Query that copes with your unclean data is the reality of handling data. Bast of luck I hope this helped.

Power Query – Creating New Parameters

Mon, 06 Jan 2025 11:52:37 +0000

When fetching data into Power Query you need to use values to point you to the write the data, for example name of a database server or path to a csv file. Good practice is to use parameters for these values so your query is reusable etc. And its always our intention to go and create the new parameter later.

Missing New Parameter Option

So lets take the simplest option of connecting to a csv file on the web. I have my path so I click get data and select Web. For those that want to play along here is the path to the file I’m using.

1

https://raw.githubusercontent.com/HatFullOfData/Demo/refs/heads/main/Products.csv

The dialog that appears has a box for the URL but is missing the drop down on the left that will allow me to select New Parameter. If I had already created a parameter it gives me the option but not if I have no parameters. Bizarre functionality.

Always Allow Parameters

The fix for this is to allow parameterization in data source dialogs. This can be done in the Power BI desktop options or on Power Query View ribbon for Power BI and Excel.

Once this is ticked when you now go to Get Data we get a drop down next to the text box. Now we can select New Parameter and enter in the details of the new parameter. When we click OK it returns to the first dialog with the new parameter selected.

As a side note I recommend that you always give parameters a Type rather than leaving it on Any. There are a few features such as Power BI deployment pipelines that will not work with parameters of type Any.

Conclusion

This is a really small feature, possibly doesn’t need a whole blog post. But that little drop down making the creating a new parameter that few clicks easier means I am more likely to follow my own advice and use parameters from the start.

Power Query – Get Previous Row Data

Tue, 08 Jun 2021 08:55:16 +0000

Occasionally I get asked a question via Twitter or LinkedIn that prompts me to write a post. The question asked was how to get previous row data. In this post I will describe how to get a value from the previous row that matches a criteria. The example data is peoples scores on different dates. I want to know if they are improving. For this I need their previous score.

Starting Data

Getting Previous Row Data

The first step is to sort the data. You need the previous score to be in the previous row, so we sort by name and then date.

Then from the Add Column ribbon we add an Index column. And then a custom column, “Prev Index” which is Index-1

Now we can merge a table to itself (yes a touch head screwy but its fine it works) by linking Prev Index to Index. This though will give Bob Alice’s last score as his previous score so we add the Name column to the merge as well. Holding down the Ctrl key will allow you to select multiple columns and make sure at the bottom of the dialog you have some matching rows..

This creates a column that contains tables. Press the expand button and only select the Score column. Click OK to give a column of the previous scores ready to rename.

I then rename Score.1 to Previous Score and use Choose Columns to remove Index and Previous Index to end up with a table of name, date, score and previous score.

Power Query – Dynamic Data Source and Web.Contents()

Tue, 04 May 2021 13:01:31 +0000

This is the first post in a series to create a Star Wars report. This post handles the issue of using a parameter as the url in a web Power Query. This type of data source is referred to as a Dynamic Data Source. It will not refresh in the Power BI service unless you do some tricks.

Star Wars Series

Starting on May 4th a series using Star Wars data had to be done. Its based off an api that has some great data that is paged in 10 row blocks. I’m looking to create complete report from queries to modelling to visuals and theme etc.

The data comes from https://swapi.dev/api/

Handling Dynamic Data sources
Fetching number of pages
Using next page url

Fetching Data

I started by looking at the people table. In a new Power BI report I select Web from Get Data and use the url https://swapi.dev/api/people/ and click OK. This throws you straight into Power Query where is has created a few steps to get you a table of data.

It only gives 10 rows of data, which does include the url to the next 10 rows and also how many rows there are in total. So I use a technique I’ve blogged about before and create a URL parameter, change the source step to use the URL and then convert the query into a function.

Look here for more details https://hatfullofdata.blog/power-query-fetch-web-data/

Test Refresh

Then as a test I invoke the function to fetch the first 10 rows into a table, close and apply Power Query and then publish my report to the Power BI service.

Then I do the important test, does it refresh? When I click on refresh it fails, and when I click on the small red triangle it gives an error stating the dataset uses a dynamic data source and these can’t be refreshed.

Part of the error includes a URL https://aka.ms/dynamic-data-sources , so I go and look at the web page. And it explains how you can identify if you have a dynamic data source that won’t refresh by looking in the Data source settings in Power Query. Sure enough, when I look, it gives a message which I now know means a dynamic data source so refresh won’t work. I assume hand-authored queries means dynamic data sources.

This doesn’t tell me how to to fix it though. So off onto the internet I go and search around my favourite Power Query bloggers and sure enough Chris Webb comes up trumps with a few posts.

The second post was the one I used the most in modifying my query. It also includes searching for cow data, which I think is superb!

Removing the Dynamic Data Source

So the Web.Contents used in the source of my function needs the base url to stay the same and then any dynamic parts put into the options. The base url must be a valid web address you can authenticate to as anonymous and the shorter the better as it makes the function more flexible. I delete the invoked function query and the function from my report.

So for the Star Wars data the base url is – https://swapi.dev/api

The people part and the page=1 can be added in as part of the options. So the new solution has 2 parameters, TableName and PageNum and they are both text. Then we have to edit the source step by hand to write the second parameter of Web.Con (the whole hand-crafted fix to get rid of hand-crafted error is just amusing). The options are a record so put inside square brackets [ .. ]. The query is another record so is a nested [ .. ].

Web.Contents options allow for two things to be set, a relative path, eg People and query parameters eg Page=1. APIs are fussy so make sure you get the case right in the query part. This is my final code for the Source step. (This is the reason I blog, so in 6 months time I know I have a working example piece of code.)

1
2
3
4
5
6


Source = Json.Document(Web.Contents("https://swapi.dev/api/",
 [
 RelativePath=TableName,
 Query=[page=PageNum]
 ]
 )),

Then I right click on the query and create a function. Invoking this function creates a table ready now to close and apply back into Power BI desktop. I publish and try a refresh and YES! it works. I now have a function that will take a table name and page number and give 10 rows of data.

In the next post we will use this function to get all the data for a table.

Power Query – VBA to Edit a Parameter Value

Wed, 21 Apr 2021 17:04:16 +0000

This post is to document how to use Excel VBA to edit a parameter value without using the a cell reference in Power Query. My query was using Web.Contents which works with parameters in Excel Power Query1 but doesn’t like a function as part of the path.

I started with Chris Webb’s great post on using an Excel named range and Web.Contents gave me problems. I would try Chris Webb’s method first which is found here https://blog.crossjoin.co.uk/2014/07/22/working-with-excel-named-ranges-in-power-query/

So I want a button to take the value from a cell and update the value in a parameter. I searched the web and found one solution buried in a MrExcel forum so I’m documenting the solution here mostly so I can find it again later.

VBA Code to Edit a Parameter Value

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


Sub ChangeParameterValue(ParameterName As String, ParameterValue As String)

 Dim qry As WorkbookQuery
 Dim formula As Variant
 
 '=== Get the query
 Set qry = ThisWorkbook.Queries(ParameterName)
 
 '=== Split the formula into 3 parts and update the second one
 formula = Split(qry.formula, Chr(34), 3)
 formula(1) = ParameterValue
 
 '=== Update the parameter value
 qry.formula = Join(formula, Chr(34))
 
End Sub

Description

The formula property of a query is really weird and hard to construct so the above function splits on Chr(34), which is a “. It then updates the middle value to the new value and then sticks the three values back together again.

The original forum post I got the solution from can be found at https://www.mrexcel.com/board/threads/vba-code-to-edit-power-query-data-source-settings.1146964/

Footnotes

1 – Web.Contents varies in different versions of Power Query so test carefully and do not assume you can copy and paste queries.

Power Query – AI Insights Error

Fri, 15 Jan 2021 22:09:52 +0000

AI Insights in Power Query is a great Premium feature in Power BI. An AI Insights Error can occur if you are not fully aware of the settings you need to check. This foxed me for 2+ hours today. So I hope this post helps someone not waste the 2 hours like me.

From the Add Columns ribbon I selected Text Analytics and then in the Text Analytics I selected Extract key phrases, but this is true for any of the AI functions.

It thought for a while, and I assumed it was processing. And then it gave this error. I tried every combination of what I could do to get rid of the error.

The issue is when you do an AI function it creates a data source with a Public privacy level. So if your data that you are analysing has a different privacy level you get the error above.

Fixing Privacy Levels

This issue has a really quick fix once you realise what it is caused by. On the home ribbon in Power Query, click on Data source setting. This opens the Data Source setting dialog box.

Click on each data source and click on Edit Permissions. Change the privacy level to be the same, I use Organizational usually.

When you have checked all your data sources you can click close in the Data Source Settings dialog and refresh your data to get rid of the error.

Conclusion on AI Insights Errors

This is a classic case of when you know the cause it’s easy to fix. I hope it helped.

Power BI – London Reactor – Tea & Biscuits Session

Sun, 15 Nov 2020 23:07:36 +0000

This post is to support my virtual introduction to Power BI session for London Reactor on 16th November 2020.

Resources

Tea Production – https://teahow.com/where-is-tea-grown/Favourite Biscuit – https://hatfullofdata.blog/wp-content/uploads/2020/11/Biscuit-Data.xlsx

The biscuit data is fictitious using randomly generated names and random selection of biscuits.

Session Recording

London Reactor Posts

Power BI – Import Text Using Examples

Sat, 22 Aug 2020 22:22:45 +0000

Import text using examples was introduced in the 2020 August update for Power BI. It writes the Power Query steps to transform a non-tabular text file into a table.

YouTube Version

https://youtu.be/6h7g-CwkeVg

Enabling the Feature

At the point of writing this blog post this feature is still in preview. The feature is enabled in File – Options in Preview features. It will like most features move to being enabled for everyone at some point.

Example Import text using Examples

For this example I have a very simple text file as shown in first picture in this post. It contains the details of journeys on different days. So when I select Get Data – Text/CSV and select the file, the dialog that shows a preview of the data in the file has a new button.

When the button is clicked a new dialog opens a dialog box with a data preview showing and an empty table. This means we need to enter in some answers into the table so a pattern can be calculated how to transform the data. When you start to type in a value it will suggest possible values.

A column is renamed by double clicking on the column title and entering a new name.

With only one value it cannot calculate the pattern required. When I have added more values it can determine the pattern. It often needs less values in future columns. In this example 2 dates were needed for the first column and then only one value for the remaining columns.

In the above example the Distance column was only part of the row of data. It calculates the ways to split the data. After this and all the columns have been added, we click Transform Data to see the Power Query steps that the import feature have been written.

Looking at the Power Query

So the magic behind this new feature is it writes a 10+ step query for you with a pretty good pattern for doing this. So the 5 steps after the source are working out how many rows make a block of data and grouping that data into one block. The next chunk of steps are splitting that block into separate columns.

How does import text using example decide the block length?

This for me is the clever part. Step three where it adds the conditional column is the crucial step. So I looked at three slightly different versions of the text file to see how it changes.

Example 1

In this example each of the blocks start with a date, which means the line starts with a number. Step 3 uses a test to see if in position 0 there is a number. So for this example, we could have as many rows as we liked in a block as long as the first row is the only row starting with a number.

Example 2

In this example the blocks start with the word “Date”. Step 3 looks for the line starting with “Date” as the first line. This is pattern is a great one as it works fine for different length blocks as long as the only lines starting with “Date”.

Example 3

In this example there is no easy way to identify the top row of the data. This means the only pattern it can determine is by length of the block so it guesses 4. This works if the data is in same length blocks.

Conclusion

I like the new import text using examples feature and I can see places where it will get used. I hope it gets progressed to recognise more patterns and be more flexible. When it works it stops the report writer needing to put together the Power Query steps to do the transform.

Power BI Series

Power Query – Replace Values for Whole Table

Wed, 15 Apr 2020 15:40:57 +0000

Today’s challenge was to replace the values on all the columns in the query when I know the columns will change so I don’t want to name them.

Quick Answer

For those who don’t want the long explanation. Do replace values on at least one column to get the replace step. In the example below the previous step is #’Changed Type’ and the columns are Jan 20 and Feb 20.

1

= Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,{"Jan 20", "Feb 20"})

Remove the {…} section and replace it with Table.ColumnNames(Previous Step)

1

= Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,Table.ColumnNames(#"Changed Type"))

Longer Answer

Replace values is a powerful tool for coping with null values and errors. When you select multiple columns and select Replaces values a step is written in that includes a list of column names, eg {“column1″,”column2″,”column3”}. If your data structure changes that step will either break because a column is missing or will not include a new column.

The function Table.ColumnNames(StepName) returns a list of column names from the named step. To see this result and filter the list right click on a step and select Insert Step After. Edit the new step to be Table.ColumnNames(StepName)

The list of column names can be filtered, even though there is no drop down arrow. Right click on the value to remove and select Remove Item.

If I then rename this step to ColumnNames I can then use that step name in a replace step.

Note

Yes the Replaced Value step is refering to a calculation that comes after it (yup its wacky). If you reorder the steps by dragging them Power Query will decide to change the code so steps refer to the new previous step.

References

Microsoft Docs Table.ColumnNames

Power Query Functions – SQL Procedures

Sun, 06 Oct 2019 17:25:02 +0000

This is the fourth post in my series regarding writing custom functions in Power Query. This post describes how to create a function that will execute an SQL procedure, passing in a parameter.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Stored Procedure

My database contains a very simple procedure that given a cost center returns all the expense records.

It can be run with the following SQL

1

EXECUTE dbo.GetExpenses @CostCenter="B01"

Create Initial Query

The initial query will excute a procedure with a fixed parameter being passed to the procedure using the SQL in the previous section.

Add a new source of SQL Server. Fill in the Server and Database and then click on Advanced options to reveal the SQL statement box where you enter the SQL statement to execute the procedure.

Click OK to and then Transform Data to edit the query. When the query opens it has only one step called Source.

Create Parameter and Function

After creating the query, we need to create a parameter to store the value to pass into the procedure. So from the Home ribbon tab select Manage Parameters – New Parameter. You need to give the parameter a name and the type Text and lastly a value.

Then we need to edit the Source step of the query to use the above parameter. Remember to get the number of ” correct, you need 2 ” if the ” is inside a string.

When you alter the query you might get a prompt asking for permission to run the query. For now let it run the query, further on in this post I will discuss this further.

After we have included the parameter into the query it is ready to be converted into a function, just as we have done in previous posts. Right click on the query and select Create Function, then enter in a name for the function. As before the query and parameter get moved into a group with the new function.

This function can now be invoked on a table of cost centers to give all the related expenses.

Changing Security

When you invoke the query you will get a prompt asking for permission to run a native database query. This is a caution to warn you that some SQL is being run which could makes changes to the database.

When you click on Edit Permission you will be shown the SQL that is going to be executed, so you can check it carefully.

This request for permission can be turned off. Click on File and then Options, and then Security. The first option in the dialog is regarding Native Database Queries, by default the require approval is ticked. Be aware this changes it for all queries.

Conclusion

SQL procedures are a great way to make use of the dba to build the queries you need to fetch the data using your parameters. Obviously this will require many biscuits for the dba. I think there is plenty of scope for functions that execute an SQL procedure.

Resources

I am not the first, and hopefully not the last to write blog posts on writing functions in M for Power Query. Here are a list of the resources I found useful. (If you know of any good ones I’ve missed please let me know!)

Power Query Functions – Fetch web data

Sat, 05 Oct 2019 17:33:54 +0000

This is the third post in my Custom functions in Power Query series. Often web based data is split across multiple pages using the url to select the data. This post describes how to build a function that populates the parts of the url so multiple pages can be used to fetch web data can be combined.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Initial Query to Fetch Web Data

We start the whole process by fetching data from one page of the web site. for my example in this post I am going to use data from Formula 1 racing. The page I am using as my initial page if for the first race in 2019 in Australia. The url is

1

https://www.formula1.com/en/results.html/2019/races/1000/australia/race-result.html

I’ve used table by example to extract the data I need. (There is a blog post coming cover table by example.) So I end up with three step query.

Adding First Parameter to Web Query

The next step is to replace parts of the url with parameters in the Source step. If we look at the url it has three parts that could be made into parameters. The Race Year, Race Number and Race Name. For the the first race in 2019 the values would be :

RaceYear – 2019
RaceNumber – 1000
RaceName – Australia

The first parameter needs to be created before you change the Source step. From the Home ribbon tab, select Manage Parameters, New Parameter. Then you need to enter in a name for the parameter, a type of text and the current value. The type has to be text for the parameter to get included in the next step. Finally click OK to create the parameter.

Now we can start to change the source step in the query. Start by clicking on the cog wheel of the Source step which will open a dialog. Then click on Advanced to give the option to break the URL into parts. You can click on Add Part to add extra rows. The first drop down allows offers Text or Parameter. If you don’t get the drop down on the left it means you don’t have any text parameters.

Adding Extra Parameters

Earlier I stated that the url had three parts that could be put into parameters. We can create and add the new parameters inside the From Web dialog. Click Add Part to add a new row. From the new drop down select New Parameter and enter in the details of your new parameter, remember they need to be text.

When you have finished adding new parameter and breaking the url up, click OK to save your changes. Check you still get the correct data from your query.

Creating Function from Query

We now have a query that can fetch web data which we now need to use to build a function from it..

Right click on your query and select Create Function. Type in a name for the query.

The query and the parameters are all moved into a group and a function is created. If any changes are required in the function, the changes need to be made to the query. If changes are made to the function it will break the link with the query.

Fetch web data from Multiple pages

I have a list of races with year, number and name in an Excel file which I load into a new query. In order for the function to work the RaceYear and RaceNumber columns need to be text.

From the Add Column click Invoke Function. Type in a New column name and select the function in the drop. Then the parameters for the function will appear. Change the first drop down for each parameter to Column Name and select the column.

When you click the OK button the function will run for every row of the table. This will create a column that contains tables. Click on the icon in the top right of the new column. In the dialog box that appears it will show the columns in the table. I un-ticked the Use original column name as prefix to keep the column names short,

After OK is clicked the column of tables is replaced by the three columns from the table.

Conclusion

If your data is web based this provides an efficient method of fetching web data into one table only using a few queries and a function.

Resources

Power Query – Multi-step Function

Sun, 29 Sep 2019 17:28:40 +0000

This is the second post in my Writing Functions in M series. This post describes how to build a multi-step function in M to allow for a more complex function. We will create a parameter to base the calculations on and then build a function.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Introduction

In the first post of this series we built a different functions and all of them were a single step calculation. Some functions need to be made up of multiple steps. The example we will use in this example is to calculate the next working day based on Saturday and Sunday being non-working days.

In simple terms the calculation is simple, if today is a Friday add 3 days, Saturday add 2 days, otherwise just add one day. I realise this could be done with a nested if, I’m using this example as a nice easy one to walk through.

Creating a Parameter

In order to make writing functions easier we create parameters to store the values passed into our multi-step function. This function will only require a single parameter to store the date which we want to calculate the next working day.

From the Home ribbon tab, select Manage Parameters and the New Parameter. Then enter in a name for the parameter, e.g. ThisDate and select a Type, e.g. Date. Finally add in a Current Value and press OK.

Calculation Steps

Now we can start creating the function in with a blank query which I have renamed to NextWorkDay. This first step I am going to make equal to the parameter we created, ThisDate. (Remember the = , I have forgotten it too many times)

The first calculation step I am going to add is to calculate the day of the week of the given date. This can be done from the ribbon tab DateTime Tools. Select Date – Day – Day of Week and a new step gets added using the function Date.DayOfWeek.

Rename the step just added to something shorter, e.g. WeekDay.

The next step will be to calculate how many days to add to the date. You can add a new step by clicking on the fx next to the formula bar.

We use an If statement based on the previous step. There is no quick way to add this so you will need to type in formula. The editor does now have intellisense so you should get some assistance in typing in names.

Using the logic described in the second paragraph of the introduction the complete formula is

1
2
3


= if WeekDay=4 then 3 else 
 if WeekDay=5 then 2 
 else 1

And I rename the step to be DaysToAdd to make the next step easier and it should return 1,2 or 3. You can click on any step and see the result, for example clicking on Source gives you the date.

So the final step is to use Source and DaysToAdd to calculate the next working day. We use the function Date.AddDays which requires 2 parameters, a date which is Source and the number of days to add which is DaysToAdd.

We can change the date in the parameter ThisDate to test different dates and check the calculation logic. Now we are ready to convert this to function.

Creating the Multi-step Function

Power Query includes a quick method to convert a query that uses a parameter into a function.

Right click on the NextWorkDay function and from the menu select Create Function. Enter in a name for the function in the Create Function dialog and press OK.

This will move both NextWorkDay query and ThisDate parameter into a separate group with the new function. If you want to edit the function you need to edit NextWorkDay and it will update the function fnNextWorkDay. You can edit the function directly but it will break the connection with NextWorkDay so debugging will be harder and is not recommended.

Using your Multi-step Function

The function can now be used in any query in your report by clicking Invoke Custom Column from the Add Column ribbon.

Conclusion

The quick Create Function ability makes creating and editing the functions much easier. It makes building functions worth doing and not a huge development process.

Resources

Power Query – Custom Handwritten Function

Fri, 27 Sep 2019 17:50:43 +0000

Introduction

In this post I will write a simple custom handwritten function to perform a calculation based off a parameter. We will then expand that function to include data types and optional parameters.

For those who are new to programming a function is a piece of code that returns some data, which could be a single value, record or table of data. If you ever need to duplicate a calculation more that twice then it is worth considering writing a function.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

The Hello World function

Every programming 101 course includes some sort of Hello World function as the introduction. I would hate to break tradition.

In Power Query we add a Blank Query and we get a blank formula bar. Enter in the following code.

1

= () => "Hello World"

When you press return the blank query will be converted into a custom function. It gains an fx in the Queries list. It can be renamed in the Query Settings just like a query.

The function can then be used to create a column in a table. From the Add Column ribbon tab, click Invoke Custom Function. Enter in a name for the new column and select the new function from the list.

This will add a column to your data. This example is a pretty useless example of just Hello World but tradition has been done and we are now ready to move to the next stage.

Adding a parameter to our Handwritten Function

The Hello World function just returned one value, which has a very limited use. So now we will add a parameter to our function and use that value to change our result to include that message.

Parameter definitions go between the brackets in the definition. I’ve updated the function to be this code and on pressing return will now have details of the parameter required.

1

= (GuestName) => "Hello " & GuestName

If the function is being used anywhere you will need to modify the function. You can either remove and re-add the function or click the cog wheel on the relevant step. There are two possible windows you will get on clicking the cog wheel.

Editing a working custom function column

Editing a custom function column that returns an error

Your table will now show the result of your updated handwritten function.

Data Types

I can feel my degree lectures turning in their graves as the above function has no data types specified. You can find many online references as to why adding data types into code is good practice, for this post please accept it is good practice and good practice usually leads to cleaner and faster code.

The fnHelloWorld has a parameter that is text and it returns some text. So we can adjust the code with “as text” after the GuestName parameter and also after the ). It won’t change the function much but it will in the long term be a great habit to adopt.

Quick Look at If

One of the most used constructs in programming in any language must be the if statement. In order to help you build quick easy functions it is worth noting the syntax.

1

if [TEST] then [TRUE RESULT] else [FALSE RESULT]

So here is a very simple example of a function that calculates a delivery cost based on if the value is over 100 the delivery is free.

Optional Parameters

Last topic in this post is to make parameter optional and handling when it is not provided. Optional parameters must be the last parameters defined in your function.

So in my example I’m going to add a column that calculates tax based on the value and the optional tax rate. If no tax rate is given 20% is assumed.

The parameters have to be withoutTax value followed by taxRate as the taxRate is optional. We can test for no taxRate being given by testing for taxRate = null. So our function is

1
2
3
4
5


= (withoutTax as number, optional taxRate as number) as number => 
 if taxRate = null then 
 withoutTax * 0.1 
 else 
 withoutTax * taxRate

Adding a tax column to example data shows how Alex’s tax rate is null so the tax is calculated at 10% and Jo’s tax is calculated at the given rate of 20%.

Conclusion on Handwritten Functions

These are very simple examples of functions but even these can save hours of repeating calculations. When your data has the same cleanup calculations being done to your data over many queries this allows for one function to be used repeatably.

Resources

Power Query in Flow – Summarising the Data

Mon, 23 Sep 2019 08:33:01 +0000

This is the third post in my series regarding Power Query in Flow. In this post I will look at the different options we have for summarising the data using grouping and pivoting with Power Query.

Here is a list of all the posts in the series so far.

Introduction

Again this post might be redundant if you’ve followed my advice and found out your dba’s favourite biscuits and got them to write you some awesome views in the database.

For those that haven’t got access to the dba lets look at how we can summarise the data. We will look at two different types of summarising the data, firstly grouping and then pivoting. I have assumed you’ve got your data into a simple table using merges etc.

Using Grouping

In this example we will group by a column and include some summary values. We will group by Product and then summary values of count of orders, total quantity and total value.

This is how the data starts.

On the toolbar click Transform Table and select Group By. We will start with a basic grouping of just one column to group by and one summary column of Order Count.

I selected ProdName for the group by , entered a name for the summary column to give Order Count by counting the rows.

To add more columns to the summary I click on the cog wheel on the Grouped rows step to re-open the group by dialog.

I click on Advanced to open up options to add extra grouping and aggregation options. I click on Add aggregation to add extra columns and change Operation to Sum and Column to total up Qty and Value. Clicking OK updates the table.

Using Pivoting

Pivoting is a form of grouping that splits the data across a number of columns, for example total sales for each product across hours of the day. The first step is to reduce your table into just the data you need. So include the data for each row title, one set of data to be the column headings and one to be the data summarised.

So I do Select Columns to reduce down to just Product, Hour of Sale and Qty of items sold.

Next, select the column of data that will make the column headings in your pivot. In my example I select the SaleHour column. Then from Transform Column, select Pivot Column.

In the Pivot column dialog, click on Advanced to reveal more options. In the Value column select the value you want to summarise and select the function you to use. Click OK to pivot the data.

Conclusion

Power Query provides a powerful way for summarising the data ready to process within the flow. As I stated earlier a view written by a dba would be more useful and quicker.

Joining Tables of Data in Flow’s Power Query

Fri, 06 Sep 2019 20:00:01 +0000

This is the second post in my series regarding Power Query in Flow. In this post I will introduce one of the most powerful parts of Power Query, joining tables. Its the vlookup part of Power Query.

Here is a list of all the posts in the series so far.

Introduction

If you have made best buddy friends with your dba (database administrator) then this post is possibly not required because if your dba will write a view to do the join for you, that is always better. BTW in my experience, dbas like biscuits.

For those who don’t have access to their dba or haven’t bought enough biscuits this what we are going to do. My database contains a table called productsales.

In the Product Sales table is a ProductID column. I am going to use this value to merge in the values from the Product table (i.e. joining tables) so I can get the name and price.

Multiple Tables

We start by adding both tables to our query. If multiple tables are in a query, Power Query only returns one table. You can tell which table will be returned by looking at the list of tables. The tables with italic names will not be returned, therefore the un-italicised table will be returned.

Right click on a table name to open the menu. Then select Enable Load to change the table that will be loaded. In this example I will select that productsales will be loaded.

Merging tables

We want to add the product name and product price to product sales tables. We will do this by merging the two tables.

Select the productsales table. Then from the toolbar, select Combine Tables and Merge Queries.

In the Merge dialog that appears select table you wish to merge with in the drop down, in my example I select products table. Then click on the columns that match, in my example I select product id in both tables.

Leave the join kind as Left Outer as this will keep all the sales records and find matching products. Further details of join types can be found on Radacad’s site. https://radacad.com/choose-the-right-merge-join-type-in-power-bi

Expanding the table column

Clicking OK will update the table to include a column which contains Tables.

Click on the icon in top right of the new column to display the columns available in the tables. Unselect any un-required columns. Unselect the Use original column name as prefix to prevent the new columns being named long names for example products.ProdName. Then click OK to add the new columns.

Conclusion to Joining Tables

Merging is a powerful addition to transforming data. It is better if a view can added to the database by your dba as that will always be more efficient. Power Query merges do offer a great alternative.

Power Query in Microsoft Flow – An Introduction

Mon, 26 Aug 2019 21:34:42 +0000

This is the first post in a new series regarding Power Query in Microsoft Flow. In this post I will cover the basics of connecting to SQL Server and using Power Query to filter and sort the data and finally add calculated columns.

This series assumes that you are not a user of Power Query desktop from Excel or Power BI.

Here is a list of all the posts in the series so far.

Introduction to Power Query in Microsoft Flow

In September 2018 Microsoft announced the integration of Power Query Online for Microsoft Flow. The reasons were to offer an alternative to OData and SQL for getting data from SQL Server connector. They included a hint that other data sources will come in the future but no details when or what.

For those coming from Power Query for Excel or Power BI you need to understand you are currently limited to data stored in SQL Server online or server via a gateway. Also the Power Query Online environment is not as powerful as the desktop app version. It is still written in M, so you can do many of the transformations you can in Excel and Power BI’s Power Query.

Connecting to the Database

Our first action is to connect Power Query to the data. Power Query action is from the SQL Server connector.

If this is the first time you’ve connected to the database you will be prompted for database details and the user id to use with a password.

Then the action then changes to just show a button. This means you have connected to the database successfully.

Selecting Data to Transform

When you click the Edit Query button for the first time, you will be presented with the tables and views in the database. You need to tick the tables and views that you need. For this post I’m just going select a single table Expenses and then click Next.

The next window is the main Online Power Query window, where we will add steps to transform your data.

List of parameters and tables
View of the data at the current step
List of steps
Toolbar used to add new steps

Filtering Data

One of the most common requests on data is to filter the values. Clicking on the arrow next to column header will show you a list of values with tick boxes. Un-tick the unrequired values to filter your data. When you do this it will add a new step to the applied steps.

When you want to change the filtered value, you need to click the cog wheel next to the step. You can click on X to remove any step.

Sorting Data

When you click the drop down arrow next the column heading, you can select Sort ascending or Sort descending. The first time you select sort, the data gets sorted and a new step is added to the applied steps.

If you then sort another column, the existing step is just extended to add the next column to the sort. This is similar to adding a new sort level in Excel.

Adding Columns

One of the most powerful parts of Power Query is to add columns using calculations. The interface can be really clever and also limited and drives me nuts which is perhaps a blog post all of its own.

If you select a column and then click Add Column from the top ribbon you will get offered choices based on the column type. For example, if you have selected a number column you get offered Standard, Scientific and Trigonometry functions and rounding plus others.

The result of the calculation is placed in a new column.

When you select multiple columns before selecting Add column, calculations to combine the columns are offered, such as difference between dates.

Renaming and Selecting Columns

You can rename any column by double clicking on the column header and typing in a new name. The first column you rename will add a new step to the query. If you rename another column straight after the Renamed Columns step is extended.

It is always a good idea to only return the columns that are required by the Flow. Under Manage Columns you can select Choose columns. When the dialog is selected, you can tick or un-tick any column.

The choose columns step has a cog wheel so can be modified at any point. I recommend you use this step rather than delete columns as that cannot be tweaked using the cog wheel.

Finishing

When you have finished transforming your data you can click Create on a new query or Update if you are editing an existing query. You can now use the data returned just as you would for getting items from any source.

Resources for Power Query in Microsoft Flow

As always there are a selection of resources out on the web.

•Microsoft https://flow.microsoft.com/en-us/blog/powerquery-flow/

Chris Webbhttps://blog.crossjoin.co.uk/2018/09/26/using-power-query-and-microsoft-flow-to-automate-the-creation-of-csv-files/

Erik Svensenhttps://eriksvensen.wordpress.com/2018/09/25/powerquery-everywhere-now-in-microsoftflow-as-well/

Conclusion

Power Query is a great addition to the Flow tool set. I am really hoping for more data sources to be allowed and for the online Power Query to include more features from the desktop version.

This is just an introduction so I will include best practices and further features.

Power Query on Hat Full of Data

Paginated Reports – Power Query as a Source

Paginated Reports have had some Love

Get Data

Power Query Editor

Power Query as a source in Explorer

Editing the Query

A few Gotchas

Multiple Queries

Refresh or not?

Parameters

Power Query as a Source Conclusion

Power Query – Handling Null Values Properly

Method 1 – Replace Null Values

Method 2 – Using the Coalesce operator with Null Values

Conclusion

More Power Query Posts

Power Query – Fixing Missing Columns Dynamically

Single Missing Column

Multiple Missing Columns

Advanced Notes on Missing Columns

Conclusion

More Power Query Posts

Power Query – Creating New Parameters

Missing New Parameter Option

Always Allow Parameters

Conclusion

Power Query – Get Previous Row Data

Starting Data

Getting Previous Row Data

More Power Query Posts

Power Query – Dynamic Data Source and Web.Contents()

Star Wars Series

Fetching Data

Test Refresh

Removing the Dynamic Data Source

More Power Query Posts

Power Query – VBA to Edit a Parameter Value

VBA Code to Edit a Parameter Value

Description

More Power Query Posts

Power Query – AI Insights Error

Fixing Privacy Levels

Conclusion on AI Insights Errors

More Power Query Posts

Power BI – London Reactor – Tea & Biscuits Session

Resources

Session Recording

London Reactor Posts

Power BI – Import Text Using Examples

YouTube Version

Enabling the Feature

Example Import text using Examples

Looking at the Power Query

How does import text using example decide the block length?

Example 1

Example 2

Example 3

Conclusion

More Power BI Posts

Power BI Series

Power Query – Replace Values for Whole Table

Quick Answer

Longer Answer

Note

References

More Power Query Posts

Power Query Functions – SQL Procedures

Stored Procedure

Create Initial Query

Create Parameter and Function

Changing Security

Conclusion

Resources

Power Query Functions – Fetch web data

Initial Query to Fetch Web Data

Adding First Parameter to Web Query

Adding Extra Parameters

Creating Function from Query

Fetch web data from Multiple pages