M on Hat Full of Data

First refresh it broke!

Mon, 12 May 2025 15:57:47 +0000

I set up my dataflow and it was working great. Then it broke on the refresh. Here is how I fixed it.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

Previously

In the previous post we ingested a SharePoint folder into a table in a Lakehouse. That is a well known process and nothing new except for one thing. I split the process into two stages, firstly the FullFileList which contained the list of files and the binary content of the files and then a separate query to combine data from those files. Power Query created Helper queries to perform the combining. Then it broke.

It Broke

It all appears to work fine. Then I try a refresh and it fails. On going back to look at the dataflow I see that there are 2 extra steps in FullFileList that have been added by Microsoft Fabric.

1
2
3
4
5
6
7


let
 Source = SharePoint.Files("https://lgb123.sharepoint.com/sites/FabricDemo/", [ApiVersion = 15]),
 #"Filtered rows" = Table.SelectRows(Source, each [Folder Path] = "https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/"),
 #"From Value" = Table.FromValue(#"Filtered rows"),
 #"Remove Columns" = Table.RemoveColumns(#"From Value", Table.ColumnsOfType(#"From Value", {type table, type record, type list, type nullable binary, type binary, type function}))
in
 #"Remove Columns"

The last step Remove Columns, removes all columns of type table, record, list, binary or function. So this removes the Content column that is binary and therefore the queries doing the combine of the Content column all fail. Hence it broke.

It Broke in the Staging

To improve performance and reliability, Dataflow Gen2 uses staging items to store intermediate data during data transformation

The above quote is from https://learn.microsoft.com/en-us/fabric/data-factory/data-in-staging-artifacts It stores the intermediate queries, so FullFileList would have been stored as a table in the staginglakehouse or stagingwarehouse. So Fabric is adding the step to remove the binary column so staging can work.

Turn Off Staging

When I right click on the FullFileList query in the query list the menu includes Enable Staging, by default it is turned on, indicated with a tick. So I click on Enable staging to remove the tick and turn it off. I publish the dataflow and wait. Yes! the refresh works, I test it multiple times.

Conclusion

I am curious as to how turning off staging will impact the dataflow. So I will be keeping an eye on this one. But this appears to be the fix we need to get it working again for now. I will do my enquiries and retests to see if this is still needed.

SharePoint Folder into Microsoft Fabric

Fri, 09 May 2025 17:08:15 +0000

SharePoint folder has been a connector into Power Query for Excel, Power BI Desktop and dataflows for quite a while. This post is to cover the basics before I write another post to explain a more complex scenario. This post will use GEN2 Dataflows in Microsoft Fabric. Power Query can take files on a SharePoint site and from a selected sheet, table or named range append the contents to one big table. This post explains what you should be aware of when doing this.

SharePoint and Microsoft Fabric

SharePoint is no database but we keep putting data there so here are my findings and notes from working with SharePoint libraries and lists in Microsoft Fabric.

SharePoint Library

I have a SharePoint library on a site called Budgets. It contains 3 files which are all the same format, 2 tabs, 1 table and 2 named regions.

The URL to the library is https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/Forms/AllItems.aspx . From this I can work out the URL to the SharePoint site is https://lgb123.sharepoint.com/sites/FabricDemo/Budgets/ You can always test it by making sure the site url takes you to the front page.

Connecting to SharePoint Folder

In a GEN2 Dataflow, from the Home ribbon select Get Data and then more. If SharePoint folder isn’t showing in the dialog that appears, search for SharePoint. Then select SharePoint folder. Then enter in your site URL. Please note the full library url will not work.

If this is the first time you’ve connected to this SharePoint site you will need to set up the connection. All you need to do is change the Authentication kind to Organizational account and make sure you are signed in. Click Next to move onto the Preview folder data. If you have more than one library or folders it will show too many files, don’t worry we will fix this.

Click Create, rather than Combine so we can control which files get combined.

Filtering to the right SharePoint Folder

In my example I have multiple SharePoint libraries in the same site, so the initial list returned contains too many files. There is a column called Folder Path. I find a row that contains the value I want, right click on the value. Then select Text filters – Equals. This will add a filter a step to the query to filter to only files in that path.

Even if your current site only has one document library so currently this step is not required, I would add it anyway. Then when anyone adds files a new library it will not break your query.

Combining the Files

In order to make future additions to these queries easier and to reduce repeated work happening I would rename the query so far to FullFileList or similar and the right click on the query and select Reference. Rename the new query to something suitable, in my case it was Budget. The diagram will now show the two queries connected.

The next stage is to do the combining. Make sure the Content column has a data type of Binary. Then click the button in the top corner of the Content column to start the file combination. After a few seconds the Combine Files will appear requesting you to select which sheet, table or named range you are combining.

Once you click Combine it will create the Helper queries and related queries and will combine the files. You then will have a query with the first column Source.Name and the rest the data from the selected item.

Adding a Destination

There might be other transformation steps you need to add before your data is ready. I’ve added an unpivot to my data, but that would a complete post on its own. Finally we get to add a destination to save the data to. If you are going to a Lakehouse, Warehouse or Database you should be aware of option available to you.

The first option is if you are saving to an existing table or creating a new table.

When you click next the second option is available. Turning off Use automatic settings will give you the options. Is the update an append or a replace? Is the schema dynamic or fixed, ie if your query adds a new column what happens? I’ve picked replace, the default option but as the SharePoint folder grows I might look to more clever file filters so that only new data is selected and therefore can be appended.

Conclusion

Uploading multiple Excel files will always be a requirement. So its worth knowing what Power Query can do for us in a dataflow to do that. There are probably more capacity efficient methods which need exploring.

Power Query – Fixing Missing Columns Dynamically

Wed, 08 Jan 2025 11:49:26 +0000

One of the challenges of making sure your query works in Power Query is data sources that have a changing schema. Some APIs miss out fields if the data is null and sometimes we need one report able to handle data files that are almost the same but not quite. In this post we will look at fixing 2 scenarios, a single missing column and multiple missing columns.

You know you have this problem when you get the an error similar to this:

Single Missing Column

This scenario is fixing one column missing. In my example the query was written based on a file that had a Debit column. The query is now pointed to file that is missing that column. We need the query to work if the Debit column is in the original or not.

Start in the broken version and find the first step where there is not an error. The fix is to add the Debit column if the table does not have the column.

M is not friendly to write so lets get the interface to write part of it. Click on Custom Column on the Add Column ribbon. Click Insert when prompted about inserting a step. Then enter in the name of the missing column for the column name and null for the value. When you click OK the column gets inserted and your query goes back to working.

But the query will fail when the file does include the Debit column. So we need to add some logic to the new step we added. We need to test to see if the table already has the column, if it does just return the table, if not add the column. This can be done using Table.HasColumns function and an “if” statement.

The line of M created by the inserting a custom column is

1

= Table.AddColumn(#"Promoted Headers", "Debit", each null)

This needs to become

1
2
3


= if Table.HasColumns(#"Promoted Headers","Debit") 
 then #"Promoted Headers" 
 else Table.AddColumn(#"Promoted Headers", "Debit", each null)

Please note the #”Promoted Headers” comes from the name of the previous step.

Reference for HasColumns can be found at https://learn.microsoft.com/en-us/powerquery-m/table-hascolumns

Multiple Missing Columns

The previous method works great for a single column and maybe even 2 but if you have more columns than that you really need another method. In this example we have a set of data with columns for months Jan – Dec. At the start of the year there is only a Jan column but we need the query to return all the columns.

The error will only complain about the first column it finds missing but if we fixed Feb it would then complain that Mar was missing etc. When you append 2 tables it creates a table with all the columns of both tables even if one table does not contain any data. So if we can create an empty table with all the columns we want and then append the tables we will get all the columns added.

Create a new blank query and enter in the formula to create a list of all the column names you want to make sure exist. Use {} brackets containing the column name strings separated by commas. This will display the list of values. Rename the query to something meaningful, for example Months.

1

= {"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"}

Now we need to add a step that converts this into a table. Right click on the Source step and select Insert Step After. Next we use #table function that takes 2 parameters, list of columns and values. We want an empty table so will use {} for the values. Change the formula to the following

1

= #table( Source , {} )

Reference for #table can be found at https://learn.microsoft.com/en-us/powerquery-m/sharptable

Return to the broken query and find the last step that the query works. On the Home ribbon select Append Queries and select Insert when prompted about inserting a step. In the Append dialog select the query that has the empty table just created. Then click OK. Your query now has all the columns required and should work if the columns are missing columns or not

Advanced Notes on Missing Columns

When appending 2 tables the first table columns will come first and in the order of the first table. So if you wish to order the columns of the query as well make sure your blank table query contains all the columns and the swap the order of the tables named in the Table.Combine step.

The list of column names and creating the blank table could be put as steps within the original query. That requires being comfortable with the Advanced editor and writing M. For those interested here is an example query.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


let
 Source = Excel.Workbook(Web.Contents(ExcelPath), null, true),
 Sheet2_Sheet = Source{[Item="Sheet2",Kind="Sheet"]}[Data],
 #"Promoted Headers" = Table.PromoteHeaders(Sheet2_Sheet, [PromoteAllScalars=true]),
 AllColumns = {"Project","Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"},
 BlankTable = #table( AllColumns , {} ),
 #"Appended Query" = Table.Combine({BlankTable, #"Promoted Headers"}),
 #"Changed Type" = Table.TransformColumnTypes(#"Appended Query",{{"Project", type text}, {"Jan", Int64.Type}, {"Feb", Int64.Type}, {"Mar", Int64.Type}, {"Apr", Int64.Type}, {"May", Int64.Type}, {"Jun", Int64.Type}, {"Jul", Int64.Type}, {"Aug", Int64.Type}, {"Sep", Int64.Type}, {"Oct", Int64.Type}, {"Nov", Int64.Type}, {"Dec", Int64.Type}})
in
 #"Changed Type"

Conclusion

Queries should be dynamic when required and missing columns is a common issue. Writing M is not friendly, writing dynamic M takes practice but it is worth it. Power Query that copes with your unclean data is the reality of handling data. Bast of luck I hope this helped.

Power Query – Get Previous Row Data

Tue, 08 Jun 2021 08:55:16 +0000

Occasionally I get asked a question via Twitter or LinkedIn that prompts me to write a post. The question asked was how to get previous row data. In this post I will describe how to get a value from the previous row that matches a criteria. The example data is peoples scores on different dates. I want to know if they are improving. For this I need their previous score.

Starting Data

Getting Previous Row Data

The first step is to sort the data. You need the previous score to be in the previous row, so we sort by name and then date.

Then from the Add Column ribbon we add an Index column. And then a custom column, “Prev Index” which is Index-1

Now we can merge a table to itself (yes a touch head screwy but its fine it works) by linking Prev Index to Index. This though will give Bob Alice’s last score as his previous score so we add the Name column to the merge as well. Holding down the Ctrl key will allow you to select multiple columns and make sure at the bottom of the dialog you have some matching rows..

This creates a column that contains tables. Press the expand button and only select the Score column. Click OK to give a column of the previous scores ready to rename.

I then rename Score.1 to Previous Score and use Choose Columns to remove Index and Previous Index to end up with a table of name, date, score and previous score.

Power Query – Dynamic Data Source and Web.Contents()

Tue, 04 May 2021 13:01:31 +0000

This is the first post in a series to create a Star Wars report. This post handles the issue of using a parameter as the url in a web Power Query. This type of data source is referred to as a Dynamic Data Source. It will not refresh in the Power BI service unless you do some tricks.

Star Wars Series

Starting on May 4th a series using Star Wars data had to be done. Its based off an api that has some great data that is paged in 10 row blocks. I’m looking to create complete report from queries to modelling to visuals and theme etc.

The data comes from https://swapi.dev/api/

Handling Dynamic Data sources
Fetching number of pages
Using next page url

Fetching Data

I started by looking at the people table. In a new Power BI report I select Web from Get Data and use the url https://swapi.dev/api/people/ and click OK. This throws you straight into Power Query where is has created a few steps to get you a table of data.

It only gives 10 rows of data, which does include the url to the next 10 rows and also how many rows there are in total. So I use a technique I’ve blogged about before and create a URL parameter, change the source step to use the URL and then convert the query into a function.

Look here for more details https://hatfullofdata.blog/power-query-fetch-web-data/

Test Refresh

Then as a test I invoke the function to fetch the first 10 rows into a table, close and apply Power Query and then publish my report to the Power BI service.

Then I do the important test, does it refresh? When I click on refresh it fails, and when I click on the small red triangle it gives an error stating the dataset uses a dynamic data source and these can’t be refreshed.

Part of the error includes a URL https://aka.ms/dynamic-data-sources , so I go and look at the web page. And it explains how you can identify if you have a dynamic data source that won’t refresh by looking in the Data source settings in Power Query. Sure enough, when I look, it gives a message which I now know means a dynamic data source so refresh won’t work. I assume hand-authored queries means dynamic data sources.

This doesn’t tell me how to to fix it though. So off onto the internet I go and search around my favourite Power Query bloggers and sure enough Chris Webb comes up trumps with a few posts.

The second post was the one I used the most in modifying my query. It also includes searching for cow data, which I think is superb!

Removing the Dynamic Data Source

So the Web.Contents used in the source of my function needs the base url to stay the same and then any dynamic parts put into the options. The base url must be a valid web address you can authenticate to as anonymous and the shorter the better as it makes the function more flexible. I delete the invoked function query and the function from my report.

So for the Star Wars data the base url is – https://swapi.dev/api

The people part and the page=1 can be added in as part of the options. So the new solution has 2 parameters, TableName and PageNum and they are both text. Then we have to edit the source step by hand to write the second parameter of Web.Con (the whole hand-crafted fix to get rid of hand-crafted error is just amusing). The options are a record so put inside square brackets [ .. ]. The query is another record so is a nested [ .. ].

Web.Contents options allow for two things to be set, a relative path, eg People and query parameters eg Page=1. APIs are fussy so make sure you get the case right in the query part. This is my final code for the Source step. (This is the reason I blog, so in 6 months time I know I have a working example piece of code.)

1
2
3
4
5
6


Source = Json.Document(Web.Contents("https://swapi.dev/api/",
 [
 RelativePath=TableName,
 Query=[page=PageNum]
 ]
 )),

Then I right click on the query and create a function. Invoking this function creates a table ready now to close and apply back into Power BI desktop. I publish and try a refresh and YES! it works. I now have a function that will take a table name and page number and give 10 rows of data.

In the next post we will use this function to get all the data for a table.

Power Query – Replace Values for Whole Table

Wed, 15 Apr 2020 15:40:57 +0000

Today’s challenge was to replace the values on all the columns in the query when I know the columns will change so I don’t want to name them.

Quick Answer

For those who don’t want the long explanation. Do replace values on at least one column to get the replace step. In the example below the previous step is #’Changed Type’ and the columns are Jan 20 and Feb 20.

1

= Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,{"Jan 20", "Feb 20"})

Remove the {…} section and replace it with Table.ColumnNames(Previous Step)

1

= Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,Table.ColumnNames(#"Changed Type"))

Longer Answer

Replace values is a powerful tool for coping with null values and errors. When you select multiple columns and select Replaces values a step is written in that includes a list of column names, eg {“column1″,”column2″,”column3”}. If your data structure changes that step will either break because a column is missing or will not include a new column.

The function Table.ColumnNames(StepName) returns a list of column names from the named step. To see this result and filter the list right click on a step and select Insert Step After. Edit the new step to be Table.ColumnNames(StepName)

The list of column names can be filtered, even though there is no drop down arrow. Right click on the value to remove and select Remove Item.

If I then rename this step to ColumnNames I can then use that step name in a replace step.

Note

Yes the Replaced Value step is refering to a calculation that comes after it (yup its wacky). If you reorder the steps by dragging them Power Query will decide to change the code so steps refer to the new previous step.

References

Microsoft Docs Table.ColumnNames

Power Query Functions – SQL Procedures

Sun, 06 Oct 2019 17:25:02 +0000

This is the fourth post in my series regarding writing custom functions in Power Query. This post describes how to create a function that will execute an SQL procedure, passing in a parameter.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Stored Procedure

My database contains a very simple procedure that given a cost center returns all the expense records.

It can be run with the following SQL

1

EXECUTE dbo.GetExpenses @CostCenter="B01"

Create Initial Query

The initial query will excute a procedure with a fixed parameter being passed to the procedure using the SQL in the previous section.

Add a new source of SQL Server. Fill in the Server and Database and then click on Advanced options to reveal the SQL statement box where you enter the SQL statement to execute the procedure.

Click OK to and then Transform Data to edit the query. When the query opens it has only one step called Source.

Create Parameter and Function

After creating the query, we need to create a parameter to store the value to pass into the procedure. So from the Home ribbon tab select Manage Parameters – New Parameter. You need to give the parameter a name and the type Text and lastly a value.

Then we need to edit the Source step of the query to use the above parameter. Remember to get the number of ” correct, you need 2 ” if the ” is inside a string.

When you alter the query you might get a prompt asking for permission to run the query. For now let it run the query, further on in this post I will discuss this further.

After we have included the parameter into the query it is ready to be converted into a function, just as we have done in previous posts. Right click on the query and select Create Function, then enter in a name for the function. As before the query and parameter get moved into a group with the new function.

This function can now be invoked on a table of cost centers to give all the related expenses.

Changing Security

When you invoke the query you will get a prompt asking for permission to run a native database query. This is a caution to warn you that some SQL is being run which could makes changes to the database.

When you click on Edit Permission you will be shown the SQL that is going to be executed, so you can check it carefully.

This request for permission can be turned off. Click on File and then Options, and then Security. The first option in the dialog is regarding Native Database Queries, by default the require approval is ticked. Be aware this changes it for all queries.

Conclusion

SQL procedures are a great way to make use of the dba to build the queries you need to fetch the data using your parameters. Obviously this will require many biscuits for the dba. I think there is plenty of scope for functions that execute an SQL procedure.

Resources

I am not the first, and hopefully not the last to write blog posts on writing functions in M for Power Query. Here are a list of the resources I found useful. (If you know of any good ones I’ve missed please let me know!)

Power Query Functions – Fetch web data

Sat, 05 Oct 2019 17:33:54 +0000

This is the third post in my Custom functions in Power Query series. Often web based data is split across multiple pages using the url to select the data. This post describes how to build a function that populates the parts of the url so multiple pages can be used to fetch web data can be combined.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Initial Query to Fetch Web Data

We start the whole process by fetching data from one page of the web site. for my example in this post I am going to use data from Formula 1 racing. The page I am using as my initial page if for the first race in 2019 in Australia. The url is

1

https://www.formula1.com/en/results.html/2019/races/1000/australia/race-result.html

I’ve used table by example to extract the data I need. (There is a blog post coming cover table by example.) So I end up with three step query.

Adding First Parameter to Web Query

The next step is to replace parts of the url with parameters in the Source step. If we look at the url it has three parts that could be made into parameters. The Race Year, Race Number and Race Name. For the the first race in 2019 the values would be :

RaceYear – 2019
RaceNumber – 1000
RaceName – Australia

The first parameter needs to be created before you change the Source step. From the Home ribbon tab, select Manage Parameters, New Parameter. Then you need to enter in a name for the parameter, a type of text and the current value. The type has to be text for the parameter to get included in the next step. Finally click OK to create the parameter.

Now we can start to change the source step in the query. Start by clicking on the cog wheel of the Source step which will open a dialog. Then click on Advanced to give the option to break the URL into parts. You can click on Add Part to add extra rows. The first drop down allows offers Text or Parameter. If you don’t get the drop down on the left it means you don’t have any text parameters.

Adding Extra Parameters

Earlier I stated that the url had three parts that could be put into parameters. We can create and add the new parameters inside the From Web dialog. Click Add Part to add a new row. From the new drop down select New Parameter and enter in the details of your new parameter, remember they need to be text.

When you have finished adding new parameter and breaking the url up, click OK to save your changes. Check you still get the correct data from your query.

Creating Function from Query

We now have a query that can fetch web data which we now need to use to build a function from it..

Right click on your query and select Create Function. Type in a name for the query.

The query and the parameters are all moved into a group and a function is created. If any changes are required in the function, the changes need to be made to the query. If changes are made to the function it will break the link with the query.

Fetch web data from Multiple pages

I have a list of races with year, number and name in an Excel file which I load into a new query. In order for the function to work the RaceYear and RaceNumber columns need to be text.

From the Add Column click Invoke Function. Type in a New column name and select the function in the drop. Then the parameters for the function will appear. Change the first drop down for each parameter to Column Name and select the column.

When you click the OK button the function will run for every row of the table. This will create a column that contains tables. Click on the icon in the top right of the new column. In the dialog box that appears it will show the columns in the table. I un-ticked the Use original column name as prefix to keep the column names short,

After OK is clicked the column of tables is replaced by the three columns from the table.

Conclusion

If your data is web based this provides an efficient method of fetching web data into one table only using a few queries and a function.

Resources

Power Query – Multi-step Function

Sun, 29 Sep 2019 17:28:40 +0000

This is the second post in my Writing Functions in M series. This post describes how to build a multi-step function in M to allow for a more complex function. We will create a parameter to base the calculations on and then build a function.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

Introduction

In the first post of this series we built a different functions and all of them were a single step calculation. Some functions need to be made up of multiple steps. The example we will use in this example is to calculate the next working day based on Saturday and Sunday being non-working days.

In simple terms the calculation is simple, if today is a Friday add 3 days, Saturday add 2 days, otherwise just add one day. I realise this could be done with a nested if, I’m using this example as a nice easy one to walk through.

Creating a Parameter

In order to make writing functions easier we create parameters to store the values passed into our multi-step function. This function will only require a single parameter to store the date which we want to calculate the next working day.

From the Home ribbon tab, select Manage Parameters and the New Parameter. Then enter in a name for the parameter, e.g. ThisDate and select a Type, e.g. Date. Finally add in a Current Value and press OK.

Calculation Steps

Now we can start creating the function in with a blank query which I have renamed to NextWorkDay. This first step I am going to make equal to the parameter we created, ThisDate. (Remember the = , I have forgotten it too many times)

The first calculation step I am going to add is to calculate the day of the week of the given date. This can be done from the ribbon tab DateTime Tools. Select Date – Day – Day of Week and a new step gets added using the function Date.DayOfWeek.

Rename the step just added to something shorter, e.g. WeekDay.

The next step will be to calculate how many days to add to the date. You can add a new step by clicking on the fx next to the formula bar.

We use an If statement based on the previous step. There is no quick way to add this so you will need to type in formula. The editor does now have intellisense so you should get some assistance in typing in names.

Using the logic described in the second paragraph of the introduction the complete formula is

1
2
3


= if WeekDay=4 then 3 else 
 if WeekDay=5 then 2 
 else 1

And I rename the step to be DaysToAdd to make the next step easier and it should return 1,2 or 3. You can click on any step and see the result, for example clicking on Source gives you the date.

So the final step is to use Source and DaysToAdd to calculate the next working day. We use the function Date.AddDays which requires 2 parameters, a date which is Source and the number of days to add which is DaysToAdd.

We can change the date in the parameter ThisDate to test different dates and check the calculation logic. Now we are ready to convert this to function.

Creating the Multi-step Function

Power Query includes a quick method to convert a query that uses a parameter into a function.

Right click on the NextWorkDay function and from the menu select Create Function. Enter in a name for the function in the Create Function dialog and press OK.

This will move both NextWorkDay query and ThisDate parameter into a separate group with the new function. If you want to edit the function you need to edit NextWorkDay and it will update the function fnNextWorkDay. You can edit the function directly but it will break the connection with NextWorkDay so debugging will be harder and is not recommended.

Using your Multi-step Function

The function can now be used in any query in your report by clicking Invoke Custom Column from the Add Column ribbon.

Conclusion

The quick Create Function ability makes creating and editing the functions much easier. It makes building functions worth doing and not a huge development process.

Resources

Power Query – Custom Handwritten Function

Fri, 27 Sep 2019 17:50:43 +0000

Introduction

In this post I will write a simple custom handwritten function to perform a calculation based off a parameter. We will then expand that function to include data types and optional parameters.

For those who are new to programming a function is a piece of code that returns some data, which could be a single value, record or table of data. If you ever need to duplicate a calculation more that twice then it is worth considering writing a function.

This series is to support my sessions at Data Relay 2019 and will cover the topics in the session.

The Hello World function

Every programming 101 course includes some sort of Hello World function as the introduction. I would hate to break tradition.

In Power Query we add a Blank Query and we get a blank formula bar. Enter in the following code.

1

= () => "Hello World"

When you press return the blank query will be converted into a custom function. It gains an fx in the Queries list. It can be renamed in the Query Settings just like a query.

The function can then be used to create a column in a table. From the Add Column ribbon tab, click Invoke Custom Function. Enter in a name for the new column and select the new function from the list.

This will add a column to your data. This example is a pretty useless example of just Hello World but tradition has been done and we are now ready to move to the next stage.

Adding a parameter to our Handwritten Function

The Hello World function just returned one value, which has a very limited use. So now we will add a parameter to our function and use that value to change our result to include that message.

Parameter definitions go between the brackets in the definition. I’ve updated the function to be this code and on pressing return will now have details of the parameter required.

1

= (GuestName) => "Hello " & GuestName

If the function is being used anywhere you will need to modify the function. You can either remove and re-add the function or click the cog wheel on the relevant step. There are two possible windows you will get on clicking the cog wheel.

Editing a working custom function column

Editing a custom function column that returns an error

Your table will now show the result of your updated handwritten function.

Data Types

I can feel my degree lectures turning in their graves as the above function has no data types specified. You can find many online references as to why adding data types into code is good practice, for this post please accept it is good practice and good practice usually leads to cleaner and faster code.

The fnHelloWorld has a parameter that is text and it returns some text. So we can adjust the code with “as text” after the GuestName parameter and also after the ). It won’t change the function much but it will in the long term be a great habit to adopt.

Quick Look at If

One of the most used constructs in programming in any language must be the if statement. In order to help you build quick easy functions it is worth noting the syntax.

1

if [TEST] then [TRUE RESULT] else [FALSE RESULT]

So here is a very simple example of a function that calculates a delivery cost based on if the value is over 100 the delivery is free.

Optional Parameters

Last topic in this post is to make parameter optional and handling when it is not provided. Optional parameters must be the last parameters defined in your function.

So in my example I’m going to add a column that calculates tax based on the value and the optional tax rate. If no tax rate is given 20% is assumed.

The parameters have to be withoutTax value followed by taxRate as the taxRate is optional. We can test for no taxRate being given by testing for taxRate = null. So our function is

1
2
3
4
5


= (withoutTax as number, optional taxRate as number) as number => 
 if taxRate = null then 
 withoutTax * 0.1 
 else 
 withoutTax * taxRate

Adding a tax column to example data shows how Alex’s tax rate is null so the tax is calculated at 10% and Jo’s tax is calculated at the given rate of 20%.

Conclusion on Handwritten Functions

These are very simple examples of functions but even these can save hours of repeating calculations. When your data has the same cleanup calculations being done to your data over many queries this allows for one function to be used repeatably.

M on Hat Full of Data

First refresh it broke!

SharePoint and Microsoft Fabric

Previously

It Broke

It Broke in the Staging

Turn Off Staging

Conclusion

SharePoint Folder into Microsoft Fabric

SharePoint and Microsoft Fabric

SharePoint Library

Connecting to SharePoint Folder

Filtering to the right SharePoint Folder

Combining the Files

Adding a Destination

Conclusion

Power Query – Fixing Missing Columns Dynamically

Single Missing Column

Multiple Missing Columns

Advanced Notes on Missing Columns

Conclusion

More Power Query Posts

Power Query – Get Previous Row Data

Starting Data

Getting Previous Row Data

More Power Query Posts

Power Query – Dynamic Data Source and Web.Contents()

Star Wars Series

Fetching Data

Test Refresh

Removing the Dynamic Data Source

More Power Query Posts

Power Query – Replace Values for Whole Table

Quick Answer

Longer Answer

Note

References

More Power Query Posts

Power Query Functions – SQL Procedures

Stored Procedure

Create Initial Query

Create Parameter and Function

Changing Security

Conclusion

Resources

Power Query Functions – Fetch web data

Initial Query to Fetch Web Data

Adding First Parameter to Web Query

Adding Extra Parameters

Creating Function from Query

Fetch web data from Multiple pages

Conclusion

Resources

Power Query – Multi-step Function

Introduction

Creating a Parameter

Calculation Steps

Creating the Multi-step Function

Using your Multi-step Function

Conclusion

Resources

Power Query – Custom Handwritten Function

Introduction

The Hello World function

Adding a parameter to our Handwritten Function

Data Types

Quick Look at If

Optional Parameters

Conclusion on Handwritten Functions

Resources