Der Beitrag Change axis and measure in a visual by clicking a slicer erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>As of today (May 16th, 2022) Field Parameters are a preview feature in Power BI Desktop. It allows users to change the measure or the dimensional attributs within a report, by clicking on a slicer. Because it’s still in preview, make sure you enable it under Options –> Preview features –> Field Parameters.
By clicking on a slicer you can determine which axis label your line chart has, which row or column header your matrix visual has, or which measure is displayed in your visualization. This way you can use the limited space on your report more than once and – if used cleverly – integrate more analysis options into your reports.
You can download the file here
Cheers from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag Change axis and measure in a visual by clicking a slicer erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag Loading multivalued fields from Microsoft Access – not possible with Power Query erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>‚Multivalued fields‘ are a special feature of MS Access (SQL Server does not have this feature). These allow to store not only one value but up to 100 values in one field of a single record. Click here for examples and to learn how to create them.
I don’t want to drag this post out unnecessarily. As you can see from the title, there is no way to import ‚multivalued fields‘ from MS Access using Power Query. These fields are simply ignored during import.
Because I didn’t know if I was simply missing something here – e.g. an unknown parameter of the Access.Database() function – I contacted Curt Hagenlocher of the Power Query team. He confirmed that there is ‚no way to import values of this type‚ using Power Query. Thank your Curt, for this statement.
Since I still see it relatively often in the self-service area that MS Access is used as a tool for data entry and processing, my advice at this point is: If you develop the Access database yourself (i.e. you can design it according to your own ideas) and can already foresee that it will be accessed via Power Query, avoid the use of ‚multivalued fields‘. If you want to know how to replace ‚multivalued fields‘ with an adequate modeling, this article will show you how.
Since we are already in the middle of December: Have a great Christmastime and a happy new year,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag Loading multivalued fields from Microsoft Access – not possible with Power Query erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag How to speed up metadata translations in Power BI erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>
To clear up any misunderstandings right at the beginning: No, my tool does not translate your data model by itself. You have to translate the names of tables, columns and measures yourself. But in this post I provide you with a practical solution that reduces your manual effort in Tabular Editor to a minimum after you have done the translation of the objects. You don’t have to create the individual languages (aka „cultures“) manually and then right click and rename each individual table, column and measure with a mouse click. My tool does that for you. There is already a well-documented solution to this problem by Kasper de Jonge, but it boils down to editing JSON files. However, if you – like me – prefer editing in tabular form in an Excel file, then my current post should help you. In addition, there is an excellent article from the Tabular Editor team, from whom I learned a lot about how the process works manually.
In my attempt to find a solution for Excel-based translations, I asked Daniel Otykier – the creator of Tabular Editor – how it is possible to programmatically translate the objects of the data model via C#. Daniel was then kind enough to point me to the macro recorder in Tabular Editor 3 to see how to create languages and do translations for the individual objects via C# script. TE3 is an amazing tool. If you can afford it, buy it! Anyway: Because TE2 is free (download it here), I’ll show you how to do it with TE2. To follow my solution path two requirements must be met:
Before we look at how the whole thing works, a few words about the available languages.
When you save a Power BI Desktop file for the first time, the file remembers the initial model language (or default language), which you can determine as follows:
This list contains 42 different languages.
If you go into the Power BI service, you can change the language of the UI there as follows:
This list contains 44 languages, 2 more than the list in Power BI Desktop. What is the difference? The Power BI service additionally contains the two languages Arabic and Hebrew. The question why these two languages are missing in Power BI Desktop can be answered quickly by selecting one of the two languages in the Power BI service (here using Hebrew as an example):
Here, everything suddenly looks upside down. In Arabic it is the same. While this is implementable in a web application like the Power BI service, I guess it would have meant significantly more effort for Power BI Desktop as a desktop application. But that is just a guess.
At the end of this article you will find a video in which I show you how to use the Excel file. Nevertheless, I summarize here briefly the steps that can be implemented with the Excel file.
In order to translate the individual objects within Excel, I first have to get them in the initial model language in my Excel file. So that I don’t have to do this manually, I have integrated a Power Query solution that, based on the current process ID of the open Power BI Desktop file, reads these into the sheets ‚Translation – Tables‘, ‚Translation – Columns‘ and ‚Translation – Measures‘.
Microsoft has a detailed documentation about which languages are supported in Power BI. Unfortunately, I only noticed this after I started a discussion on Twitter to help me identify the languages (thanks to everyone who participated ). What is unfortunately missing in this documentation is the corresponding ‚Culture Code‘ of each language, because I need this when I want to create a translation in the data model. An example: For the language ‚Chinese (Simplified)‘ there are – as far as I know – 4 different culture codes: „zh-Hans“, „zh“, „zh-CN“, „zh-SG“. „zh“ is recognized by the Power BI service, „zh-HK“ for example is not. Therefore, I searched for a working Culture Code for each of these 44 languages and checked it against the Power BI service.
You simply select the desired language via the dropdown (the dropdown offers the language in English and in the original language, which can be very helpful) and the Culture Code then results by itself. My tool is made for up to 5 translations, but you can extend that, if you will.
After loading the objects of the data model into the Excel file and defining the up to 5 languages you want to translate into, you can perform the translation in tabular form in the sheets ‚Translation – Tables‘, ‚Translation – Columns‘ and ‚Translation – Measures‘.
Now that the translations are in Excel, I still need to get them into the data model. For this, my tool generates C# code in 2 places, which I can simply paste into the Advanced Scripting window of Tabular Editor.
You can download my Excel file here.
If your data model changes – e.g. you add measures and now also want to translate these – then reloading the tables, columns and measures can lead to the fact that the translations already made are no longer in the correct row.
Cheers from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag How to speed up metadata translations in Power BI erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag Let your users know how up-to-date their data is erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>I think the information when the data model was last updated is probably the most important. I like to place this information directly on the report so that it is immediately visible and, in case the report is printed, it is also included on the print (…yes, many customers want to be able to print their reports ). Since this information is only needed in the second place and takes up more space in case of a data model with several fact tables, I provide this information via a visual header tooltip. This tooltip can be called up when this information is needed and thus does not constantly take up space on the report. Here I provide three pieces of information:
The animated GIF below shows how I provide this information:
If you are now wondering how this information gets into the report, read on.
There are several approaches to store and report the lastest update of the data model. I present 3 versions and all those versions are based on a Power Query query, which provides the interesting timestamp , which is then loaded into the data model as a disconnected and invisible single-column, single-row table.
You can find several solutions for this topic which use the M-function DateTime.LocalNow()
to load a table which stores the current time at every refresh of the data model. I blogged about it in 2017 (in German) and here you can find a similar version from Kasper de Jonge. The problem with this approach is the following: The datetime value returned by DateTime.LocalNow()
depends on the system of the computer on which the M script is executed. Which computer this is is sometimes not clear at first sight. Here are a few scenarios as an example:
DateTime.LocalNow()
returns the datetime value of the operating system on which Power BI Desktop runs.
In case you are working with cloud data sources, the M-script is executed on the server in the Microsoft data center, where your Power BI datasets are located. In my case this is a data center in Ireland. Ireland is not in the same time zone as Germany. Therefore DateTime.LocalNow()
returns a different value here than e.g. on my local laptop.
If you automate the data update to the Power BI service through an on-premises data gateway, all M-scripts are executed in the gateway before they pass through the network in compressed form to the Power BI service. Thus, the datetime value returned in this case depends on the system on which the gateway is installed.
There are certainly other scenarios. What these 3 points should show is that DateTime.LocalNow()
is probably not the safest method to get the desired result. Let’s look at alternatives.
An alternative approach would be to use the function DateTimeZone.FixedUtcNow()
in combination with DateTimeZone.SwitchZone()
like so: DateTimeZone.SwitchZone( DateTimeZone.FixedUtcNow(), 1)
. While DateTimeZone.FixedUtcNow()
displays the current time in ‚Coordinated Universal Time‘, the second parameter of the DateTimeZone.SwitchZone()
function lets me shift the UTC time by x hours (and also minutes if needed). This sounds like a great alternative to DateTime.LocalNow()
but the devil is in the details.
Since I live in Germany and we are still jumping diligently between summer time (aka Daylight savings time) and winter time here, in my case this difference to UTC time cannot be static, but has to move between 1 and 2 depending on the date. I have already blogged about this here in German.
So this approach leads definetely to a useful result, but instead of calculating the – sometimes changing – difference to UTC time, it would be nicer if you could simply specify the desired time zone and get the datetime value, wouldn’t it? In this case I use REST API calls.
Instead of worrying about how to calculate the difference between UTC and my own time zone, you could use a web service of your choice. The following script calls a REST API that returns the current date and time in the desired time zone (passed as parameter), which in my case is CET – Central European Time.
let fn = () => | |
let | |
Quelle = Json.Document(Web.Contents("http://worldclockapi.com/api/json/cet/now")), | |
#"In Tabelle konvertiert" = Record.ToTable(Quelle), | |
#"Gefilterte Zeilen" = Table.SelectRows(#"In Tabelle konvertiert", each ([Name] = "currentDateTime")), | |
#"Geänderter Typ" = Table.TransformColumnTypes(#"Gefilterte Zeilen",{{"Value", type datetimezone}}), | |
Output = Table.TransformColumnTypes(#"Geänderter Typ",{{"Value", type datetime}})[Value]{0} | |
in | |
Output, | |
fnType = type function () as list meta | |
[ | |
Documentation.Name = " fnGetTimeFromAPI ", | |
Documentation.Description = "This function gets the current time (CET), using the API >>http://worldclockapi.com<<", | |
Documentation.Parameters = "", | |
Documentation.Category = " ", | |
Documentation.Author = " Lars Schreiber: www.ssbi-blog.de " | |
] | |
in | |
Value.ReplaceType(fn, fnType) |
Whichever of the above methods you choose to save the time of the lastest data update: The result will be a single-column and single-row disconnected table in the data model, which you will probably hide there.
To put this info into a report, you should put the datetime value into a DAX-measure, so that you can control the formatting (including line breaks). The following DAX statement does the job:
LastRefresh =
„Latest model refresh: „ & UNICHAR ( 10 )
& FORMAT ( VALUES ( ‚LatestRefresh'[timestamp] ), „dd/mm/yyyy hh:mm:ss“ )
UNICHAR(10)
creates the line break and the FORMAT()
function makes sure I can format the datetime value as I need it. Put this measure in a card visual and you’re good to go. Now I make sure that the transaction data in the fact tables is up to date.
In case the data sources doesn’t already provide the data in the necessary structure, I am a strong advocate of doing all ETL tasks in Power Query, even if DAX could partially do the same tasks. Therefore, there are almost never columns or tables calculated with DAX in my data models. In the current case, this is different. My data model has 3 fact tables. In order to know if my data sources contain current data, I select the corresponding date columns from the fact tables, which I want to check for actuality. Normally these are date columns which are related to the calendar table. Doing this in Power Query could take muuuuch longer than doing it with DAX, so the decision is easy.
To create a table using DAX, which gives me the necessary information, I use the following DAX statement:
MaxDatesFactTables =
{
( „PnL_ACT (column: ‚Investment date‘ )“, CALCULATE ( MAX ( PnL_ACT[Invest date] ), ALL ( PnL_ACT ) ), 3 ),
( „PnL_PLAN (column: ‚date‘)“, CALCULATE ( MAX ( PnL_PLAN[date] ), ALL ( PnL_PLAN ) ), 2 ),
( „PnL_Forecast (column: ‚date‘)“, CALCULATE ( MAX ( PnL_FC[date] ), ALL ( PnL_FC ) ), 1 )
}
The result looks as follows:
The three columns contain the following information:
Now let’s take a look at how I prefer to present this info.
I think the info, when the dataset had it’s latest refresh is important almost everytime… that’s why I include it on every page… but what about the latest dates in each fact table? This can be important, but I hardly need this info every single time I take a look at a report page… This is where visual specific tool tips become handy… I create a tooltip page (see the official documentation to know how that works) on which I display the disconnected table „MaxDatesFactTables “ as a matrix. Since the table created in DAX has the headings value1, value2 and value3, I overwrite them in the column captions of the matrix visual: value1 becomes Table and value2 becomes Max Date.
To place this tooltip in the visual header of the card visual, I go to the format properties of the card visual in which the LatestRefresh is located.
I turn on the visual header (if not already done) and activate the ‚Visual header tooltip icon‘. After that a new menu item appears in the format properties: ‚Visual header tooltip‘. Here I select my tooltip page in the field ‚report page‘. Done!
In addition to the lastest refresh of the data model and the actuality of the data in the data source, the question sometimes arises as to when the data was last uploaded to the Power BI service. If you update your data via scheduled refresh, the time of the last model refresh and upload to the Power BI service will be (nearly) identical. However, if you manually publish your data to the Power BI service (via Power BI Desktop), then model refresh and publishing to the Power BI service are two processes that can vary greatly in time. This information is available to every user directly in the report.
And as mentioned at the beginning: Even if you automatically update your Power BI dataset (and thus the lastest model refresh and the upload to the Power BI service have identical times): If you want to print the report, you have to put the info on the report yourself
Cheers from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag Let your users know how up-to-date their data is erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag Fast table construction with Table.FromColumns and lists with metadata records erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Table.FromColumns()
, which takes a list of lists and creates a table from them. This way you can easily create a calendar table for example. In this article I will show you how you can use the metadata record of list items to flexibly assign both column names and datatypes to the created table.
Take a look at the following M script. At the beginning I define 4 lists: A list of…
The ListOfColumns
is basically a compilation of the lists defined above, which are to be included in the final table. Here I can flexibly determine 1) if they should be included and 2) in which order they should be placed.
The final Step GetTable
creates a table from the lists defined in ListOfColumns
.
let | |
//Defining the lists | |
Date = List.Buffer( | |
List.Transform( | |
{Number.From(#date(2019,1,1))..Number.From(#date(2020,12,31))}, | |
each Date.From(_)) | |
), | |
DayName = List.Transform(Date, each Date.DayOfWeek(_)), | |
MonthName = List.Transform(Date, each Date.MonthName(_)), | |
MonthNumber = List.Transform(Date, each Date.Month(_)), | |
//Preparing the lists of lists for the Table.FromColumns function | |
ListOfColumns = { | |
Date, | |
//Choose not to include this column in the final table | |
// DayName, | |
//change order of columns | |
MonthNumber, | |
MonthName | |
}, | |
//Finally create the table from lists | |
GetTable = Table.FromColumns( ListOfColumns ) | |
in | |
GetTable |
The problem with this solution is that the final table has neither data types nor column names.
A possible solution is to define in the ListOfColumns
list not only the lists to be included in the table, but also the column name that the list is to receive in the table and the later data type. I define this via the metadata record (To be seen in code lines 16 to 23):
let | |
//Defining the lists | |
Date = List.Buffer( | |
List.Transform( | |
{Number.From(#date(2019,1,1))..Number.From(#date(2020,12,31))}, | |
each Date.From(_) | |
) | |
), | |
DayName = List.Transform(Date, each Date.DayOfWeek(_)), | |
MonthName = List.Transform(Date, each Date.MonthName(_)), | |
MonthNumber = List.Transform(Date, each Date.Month(_)), | |
//Preparing the lists of lists for the Table.FromColumns function | |
ListOfColumnsAndTypes = { | |
Date meta [ColName = "Date", DType = Date.Type], | |
//Choose not to include this column in the final table | |
// DayName meta [ColName = "DayName", DType = Text.Type], | |
//change order of columns | |
MonthNumber meta [ColName = "MonthNumber", DType = Int64.Type], | |
MonthName meta [ColName = "MonthName", DType = Text.Type] | |
}, | |
//Get Column Names from Metadata record | |
ListOfColNames = List.Transform(ListOfColumnsAndTypes, each Value.Metadata(_)[ColName]), | |
//Finally create the table from lists and add Column names | |
GetTable = Table.FromColumns( ListOfColumnsAndTypes, ListOfColNames ), | |
//Ascribing the types to the different columns using the metadata record | |
TableNewDataTypes = Table.TransformColumnTypes( | |
GetTable, | |
List.Zip( | |
{ | |
ListOfColNames, | |
List.Transform(ListOfColumnsAndTypes, each Value.Metadata(_)[DType]) | |
} | |
) | |
) | |
in | |
TableNewDataTypes |
In line 28 I use the Value.Metadata()
function to retrieve the ColName information from the metadata record of each list item and get a list of column names.
In line 31 I not only create the table, but also use the second argument of the Table.FromColumn function to define the correct column names.
From row 34 on I assign the correct data types to the columns, which I had stored in the metadata record of the list items. The result is a table that can be created flexibly and that shows both column name and correct data types:
Cheers from Hamburg, Germany
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag Fast table construction with Table.FromColumns and lists with metadata records erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag 3 ways for sums over columns in Power Query erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>The goal is to create a row at the end of the table that represents the total for the month columns and contains the row label ‚Total‘ in the first column.
Imke and Bill have sent me their solutions, but the comments of the source code are from my pen. With this I wanted to make it easier for you as a reader and I hope that I didn’t make any mistakes. If you find any mistakes here, it’s on me.
let | |
/* | |
Creating a list of monthnames... | |
format and language of the list can easily by changes by the last two parameters | |
*/ | |
MonthsList = List.Transform( | |
{0..11}, | |
each Date.ToText(#date(1900, 1, 1) + Duration.From(_ * 31), "MMMM", "en-Us") | |
), | |
Source = Excel.CurrentWorkbook(){[Name = "Datenbasis"]}[Content], | |
//Get all column names | |
ColNames = List.Buffer(Table.ColumnNames(Source)), | |
//Get all Month columns | |
Headers = List.Buffer(List.Intersect({ColNames, MonthsList})), | |
//Create a row header 'Total' | |
FirstCol = Table.FromColumns({{"Total"}}, {ColNames{0}}), | |
/* | |
Use List.Accumulate to take the first column 'Cost centre' and | |
add all the month columns with the sum over the month column to it | |
*/ | |
AccTbl = List.Accumulate( | |
Headers, | |
FirstCol, | |
(st, cur) => | |
Table.AddColumn(st, cur, each List.Sum(Table.Column(Source, cur)), type number) | |
), | |
//Combine the source table and the sum table (with one row) | |
CombineTbls = Table.Combine({Source, AccTbl}) | |
in | |
CombineTbls |
let | |
/* | |
Creating a list of monthnames... format and language of the list can | |
easily be changed by the last two parameters | |
*/ | |
MonthsList = List.Transform({0..11}, each Date.ToText(#date(1900,1,1) + | |
Duration.From(_*31), "MMMM", "en-GB")), | |
Source = Excel.CurrentWorkbook(){[Name="Datenbasis"]}[Content], | |
//Get all column names | |
ColNames = List.Buffer(Table.ColumnNames(Source)), | |
//Get all Month columns | |
ColsToSum = List.Buffer(List.Intersect({ColNames, MonthsList})), | |
/* | |
Determining the headers for the sum-table... | |
- column1 is the column with the cost centres, where we want to see | |
the word 'TOTAL' in the last row | |
- all the other columns are month columns with the sum value in the | |
last row | |
- All other columns that may occur should not be summed... | |
this is why they are not included in this step | |
*/ | |
Headers = Table.Buffer(Table.FromColumns({{ColNames{0}} & ColsToSum})), | |
/* | |
Creating a table with the column names of the source table in column1 | |
and the values of the source table in lists in column 2 | |
*/ | |
Custom1 = Table.FromColumns({ColNames, Table.ToColumns(Source)}), | |
/* | |
The merge results in the column name for the first column 'Cost centre' | |
and all month columns. However, it returns a NULL value for all other | |
columns that may occur | |
*/ | |
#"Merged Queries" = | |
Table.NestedJoin( | |
Custom1, {"Column1"}, Headers, {"Column1"}, "Custom1", JoinKind.LeftOuter | |
), | |
// Expanding the columns leads to column names or NULL value | |
#"Expanded {0}" = | |
Table.ExpandTableColumn( | |
#"Merged Queries", "Custom1", {"Column1"}, {"Column1.1"} | |
), | |
/* | |
Depending on the result of the join, generate the following calculated column | |
*/ | |
#"Added Custom" = | |
Table.AddColumn( | |
#"Expanded {0}", | |
"ColumnsValue", each | |
// if the join returns 'Cost centre' | |
if [Column1.1] = ColNames{0} then | |
//then create a combined list of Costs centres and 'Total' | |
[Column2] & {"Total"} | |
else | |
//if the join doesn't return 'Cost centre', but is not NULL then | |
if [Column1.1] <> null then | |
/* | |
Create a combination the list of values and | |
the sum of values in the bottom line | |
*/ | |
[L = List.Sum([Column2]), N = [Column2] & {L}][N] | |
else | |
/* | |
if the value is NULL, then return only | |
the values, but create no sum | |
*/ | |
[Column2] | |
), | |
/* | |
Create a final table from the values column and | |
the column 1, which includes the column headers | |
*/ | |
TblFromCols = | |
Table.FromColumns( | |
#"Added Custom"[ColumnsValue], #"Added Custom"[Column1] | |
) | |
in | |
TblFromCols |
let | |
/* | |
This is Bill's version of getting a list of months... | |
as this is not a critical part for the over all solution, | |
I just copied it into this solution as well | |
*/ | |
List_AllMonths = List.Transform({0..11}, each Date.ToText(#date(1900, 1, 1) + | |
Duration.From(_ * 31), "MMMM", "en-Us")), | |
Source = Excel.CurrentWorkbook(){[Name="Datenbasis"]}[Content], | |
//Getting all column names of the source table | |
List_AllColumnNames = Table.ColumnNames(Source), | |
//Getting all month columns from the source table | |
List_MonthCol = List.Intersect({List_AllColumnNames, List_AllMonths}), | |
/* | |
This list is used, to make grouping over a dynamic number of columns possible, | |
which is used in the next step 'grouping' | |
*/ | |
List_ColsToAggregate = | |
List.Transform( | |
List_MonthCol, | |
each {_, (x)=> List.Sum(Record.Field(x, _)), type number} | |
), | |
/* | |
This step creates the sum over the columns and respects | |
that the number of columns can change | |
*/ | |
Grouping = | |
Table.AddColumn( | |
Table.Group( | |
Source, {}, List_ColsToAggregate, GroupKind.Local | |
), | |
"Cost centre", each "Total" | |
), | |
//Combine the source table and the sum table | |
TabCombine = Table.Combine({Source, Grouping}) | |
in | |
TabCombine |
There are many roads to the solution. These 3 here, seem very creative to me and show quite different ways to solve the problem. With the available (relatively small) amount of data, the 3 methods seem to be almost equally fast. Of course this would have to be tested with larger amounts of data, but that’s not what this post is about :-). I thank Imke and Bill a lot for their solutions. I have learned a lot. Which of them do you find the most exciting?
Cheers from Hamburg, Germany
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag 3 ways for sums over columns in Power Query erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag How to handle custom errors in M gracefully erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Of course some M-enthusiasts in the community have already dealt with the topic of error handling and I recommend you to have a look at these as well:
Miguel Escobar highlights in the first of the mentioned posts, that you can raise your own (custom) error messages, if you want to.
Power Query/ M returns its own error messages when you try to evaluate an expression that M cannot evaluate. For example, the expression A + 1
returns the following error message.
But what if you want an error message to be returned on an expression that could be evaluated by M but doesn’t fit into your business logic?! In the following example, I want an error message to be returned whenever the tax percentage is not 7% or 19%.
The purpose of custom errors is to return error messages when the expression is technically evaluable but contradicts your own business logic. In this case, it is good and right to provide the user with the right information so that he can correct the error as quickly as possible. Let’s see how this can be done.
You can throw a custom error by using the keyword ‚error‘ followed by the function Error.Record(). Within the function you can define up to 3 arguments, which are:
The following screenshot shows how to manually generate this error message in a calculated column:
But what if I want to check for this (logical) error in many places in my queries? Do I then have to write this Error.Record manually each time, or copy it from one of my previous solutions? To avoid this manual process, I have written my own function that fixes this problem!
Whenever it makes sense, I outsource tasks to custom functions that I can then reuse. This reduces the risk of errors, makes my code clearer and easier to maintain. My goal in calling my custom error therefore looks like this:
So instead of manually rewriting (or copying) the Error.Record at any necessary point in my code, I want to define it centrally at one point and then call it (based on its ID) via a function from any of my queries. I’ll show you how to do this now.
I have commented on the functionality of my function directly in the source code and hope that this is sufficient to understand it.
(IDCustErr as number)=> | |
let | |
//IDCustErr = 5, /* for debugging purposes */ | |
// ============ START: Define your list of custom error records here ==================== | |
ListCustomErrors = { | |
//Custom error #1: The only permissible tax rates are as follows 7% and 19% | |
[ | |
CustErrID = 1, | |
reason ="Illegal tax rate", | |
message = "The only allowed tax rates are 7% and 19%!", | |
detail = "Make sure in your data source, that only those 2 tax rates are included!", | |
InternalDescription = "When ever another tax rate comes up, throw this error record" | |
], | |
//Custom error #2: | |
[ | |
CustErrID = 2, | |
reason ="Reason Error 2", | |
message = "Message Error 2", | |
detail = "Details Error 2", | |
InternalDescription = "INternal Description Error 2" | |
] | |
/* | |
Define as many other custom error records, as you want/ need here | |
*/ | |
}, | |
// ============ END: Define your list of custom error records here ==================== | |
//Convert the list into a table | |
ListToTable = Table.FromList(ListCustomErrors, Splitter.SplitByNothing(), null, null, ExtraValues.Error), | |
//Expand the table | |
ExpandColumn = Table.ExpandRecordColumn(ListToTable, "Column1", {"CustErrID", "reason", "message", "detail", | |
"InternalDescription"}, {"CustErrID", "reason", "message", "detail", "InternalDescription"}), | |
//This step filters the table down to the record, which has the custom error ID that you passed to the function with 'IDCustErr' | |
FilterForCustErrID = | |
try | |
//Try to call the CustErrID | |
Table.SelectRows(ExpandColumn, each ([CustErrID] = IDCustErr)){0} | |
otherwise | |
//if the CustErrID doesn't exist, throw the following error | |
error Error.Record( "Error ID not existing!", "The error ID doesn't exist! Use an existing one!"), | |
//Throw the custom error | |
Result = error Error.Record( FilterForCustErrID[reason], FilterForCustErrID[message], FilterForCustErrID[detail] ) | |
in | |
Result |
I hope this was interesting for one or the other. I am sure that this method can be further developed. If you succeed, please let me know.
Greetings from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag How to handle custom errors in M gracefully erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag How does the filter context in DAX apply? erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Custom TI functions can be useful and necessary if the standard TI functions of DAX are no longer sufficient. The pattern from SQLBI looks as follows:
[SalesYTD] :=
CALCULATE (
[Sales],
FILTER (
ALL ( ‚Date‘ ),
‚Date'[Year] = MAX ( ‚Date'[Year] ) &&
‚Date'[Date] <= MAX ( ‚Date'[Date] )
)
)
https://www.daxpatterns.com/time-patterns/
Let’s try to follow the pattern by using the (green highlighted) example value 3rd of January 2018:
The pattern does the following:
Now please answer the following question: Why is MAX( 'Calendar'[Year] )
and MAX( 'Calendar'[Date] )
evaluated under the existing »filter context«? Take some time and see if you can find a satisfactory answer.
I have been using this pattern for a very long time and simply accepted that it works without really asking myself why it works. There is this already very old but wonderful article by Jeffrey Wang (read my interview with him here) which describes how cross filtering works in DAX. He describes that there are only 3 ‚targets of filter context‘ in the DAX language:
VALUES( Table[Column] )
andDISTINCT( Table[Column] )
.Nothing else gets effected by the filter context. But how does that relate to the TI pattern of our Italian friends, as neither MAX( 'Calendar'[Year] )
nor MAX( 'Calendar'[Date] )
contain one of the 3 targets from above?
Since DAX was developed for business users, „the original design goal was to make syntax as simple as possible in common usage patterns while maintaining a coherent and semantically sound language model“ (quote Jeffrey Wang, from my interview with him). That’s why there is so much ’syntax sugar‘ in DAX, which means that a more complex DAX expression is hidden in a simpler one.
I think that most DAX users know that
=
CALCULATE ( [Total Sales], Product[Color] = „Red“ )
is internally converted to
=
CALCULATE (
[Total Sales],
FILTER ( ALL ( Product[Color] ), Product[Color] = „Red“ ).
)
just to make the DAX entry point for beginners easier. The same applies to all aggregate functions that accept a column reference such as Table[Column] as the only argument, such as SUM( Table[Column] )
and also MAX( Table[Column] )
. Jeffrey describes it in the above mentioned blog article as follows:
Note that DAX function Sum(T[C]) is just a shorthand for SumX(T, [C]), the same is true for other aggregation functions which take a single column reference as argument. Therefore the table in those aggregation functions is filtered by filter context.
Armed with this knowledge, let’s have a look at the TI pattern without syntax sugar:
Since each MAX( Table[Column] )
expression is represented internally by MAXX( Table, Table[Column] )
, the circle closes here: The first parameter of MAXX() is a table reference and this belongs to the 3 targets of the filter context. All this can of course be transferred to all other DAX expressions.
And now to be honest, was that the answer you would have given me to my question?!
Cheers from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag How does the filter context in DAX apply? erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag Table.Profile and its unknown second parameter erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Table.Profile()
as a function to get meta information about tables. During my research I noticed that this function accepts a second optional parameter which is not documented anywhere. Also other posts about this function, which colleagues from the community had already written, did not mention this parameter (at least none I could find). This made me curious, so I reached out to the Power Query dev team.
The syntax of the function is as follows: Table.Profile(table as table, optional additionalAggregates as nullable list) as table
. The first mandatory parameter is the table whose profile is to be created. The second parameter is a puzzle. Let’s take a closer look at this one.
This argument is expected as a list of lists, like this: {{},{}}. Each of these inner lists represents a new calculated profile column and consists of three items:
Let’s take a detailed look.
This is just the name of the new profile column, which of course should reflect the content of the column.
The typecheck function is expected to be something like each Type.Is(_, type any)
. While the keyword „each“‚ refers to the current record in the profiling table, the underscore „_“ refers not – as you might expect – to the current record, but to the column of the source table, whose name can be found in the current record in the first column „Column“. So the underscore „_“ doesn’t return a record, but a list. This information will also be important for the third item of the parameter additionalAggregates:
the aggregation function.
If the second parameter – the typecheck function – returns true, then the aggregation function kicks in. Otherwise the output of the aggregation function will be null.
So how could examples of the aggregation function look like?
= Table.Profile(Table.FromRows({{1, 2}, {3, null}}, type table [A=any, B=number]), {{"New profile column", each Type.Is(_, type number), each List.Average(_)}})
What happens in the profile table?
Type.Is(_, type number)
), so the function List.Average()
is not used for column A. The value null is returned accordingly.Type.Is(_, type number)
), so the function List.Average()
can kick in. The value 2 is returned as the average of 2 and null is 2.= Table.Profile(Table.FromRows({{"ABC", 2}, {"C", 3}}, type table [A=text, B=any]), {{"AverageCharactersCount", each Type.Is(_, type text), each List.Average( List.Transform(_, each Text.Length(_))) }})
What happens in the profile table?
List.Average( List.Transform(_, each Text.Length(_)))
is used for column A. The returned value is 2 as the average text length of „ABC“ and „C“ is 2. List.Average( List.Transform(_, each Text.Length(_)))
doesn’t kick in, but returns null.When you change a data type using the UI, the M function Table.TransformColumnTypes()
is used behind the scenes to do that job. What you should always keep in mind when using this function is, that it always returns a nullable data type. This is something I forgot, even if it was a central message of this of my own recent posts. Why is this important in view of the last post? Take a look at the following examples:
In the first screenshot I use ascribing types and my custom column to Table.Profile()
just works.
In the second screenshot I use the UI/ Table.TransformColumnTypes()
to assert the type and my function returns null, where I expected the value 2. Why does that happen. As I said, in my example Table.TransformColumnTypes() returns a nullable text type and Type.Is(type nullable text, type text) = false
. This is why I get back null
.
To fix this behavior, I have to check for type nullable text
, instead of type text
:
The moment you do it right, it’s running.
Greetings from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag Table.Profile and its unknown second parameter erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>Der Beitrag Tables in Power Query – how, when and why erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>In M there are two kinds of values: primitive and structured values. Examples for primitive values are:
They are primitive in that way, that they are not contructed out of other values. In contrast to primitive values, we have so called structured values in M, which are composed of other values, primitive and structured ones. A table is one of those structured values (the others are lists and records) and it is described as „a set of values organized into columns (which are identified by name), and rows.“
Even if tables usually contain columns and rows, they can be empty, which looks like this: #table({},{})
.
An example of a very simple representative of a table is:
#table( {"column_1", "column_2"}, { {1, 2}, {3, 4} } )
Before taking a more detailed look at tables of what they are and what they are for, let’s discuss why to use tables at all.
Tables are often the final result of a query, which can then either be used as an intermediate query or loaded into the data model in Power BI and Power Pivot. In addition, there are functions that work with tables as input parameters and others that generate tables as return values. For these reasons it is necessary to know how to deal with them in order to use the M language safely.
There is no literal syntax to create a table, like for records ([]) and lists ({}), but there are several native M functions, that create tables.
This function (it is spoken „pound table“) is something special, as it is not (yet) returned by the intrinsic variable #shared
. Since the first parameter of the function can be used in different ways, there are several ways to use this function.
The second parameter is always a list of list, filled with values. Each inner list represents a row within the table. Let’s take a look how the first parameter can be used in different ways.
#table( {"A", "B"}, { {1, 2}, {3, 4} } )
→By specifying the column names as text values in a list, every column gets its specific column name.
#table(5, {{ 1,2,3,4,5}} )
→ If I need to create a known number of columns (e.g. 100), but they only need to have values and no specific column headings, I can use this syntax.
#table( type table [A = number, B = text], {{1,"one"}, {2,"two"}, {3,"three"}} )
→ With this version you can define not only the column headings and the corresponding values, but also the data types of the individual columns.
I will describe why using ascribed types for defining types of values in a column of a table is a BAD IDEA later in a separat section of this post. Now let’s take a look at functions that create tables from other values in M.
Conversion functions create a table from other values, like lists, records etc. Here a few prominent representatives:
In addition to the conversion functions, there is a ton of other functions returning tables in M. First of all I want to mention those functions that allow you to define tables yourself by giving them lists built in either row or column logic:
In addition to these functions, there are many more that return tables. Most of them are connectors to external data sources. Examples of this are: Csv.Document(), DataLake.Files(), or Github.Tables().
Dealing with types in M is really tiresome. In several places I stumbled across problems in M whose solution was to be found in M’s type system. I had written under Version #3 that it is not recommended to ascribe types to the values of a column in a table. I wrote a blog post about it, which describes the reasons in detail and I recommend you read it, before going further in this post. In short Power Query accepts your type definition via ascribed types as correct and does no validation, wether the declared type of the column is compatible with the values in that column. Only when you try to load these values into another engines (Excel data model, the Power BI data model, or pure Excel) will you get error messages because these systems recognize the error.
There are 3 operators that can be used in conjunction with tables: „=“ and „<>“ make it possible to compare tables, while „&“ combines tables. Here are some examples of how to use that:
#table({"A","B"},{{1,2}}) = #table({"B","A"},{{2,1}})
→ true. Tables are equal if all of the following are true:
#table({"A","B"},{{1,2}}) = #table({"B","A"},{{2,1}, {3,4}})
→ false, due to a different number of rows.
#table({"A","B"},{{1,2}}) <> #table({"B","A"},{{2,1}, {3,4}})
→ true
#table({"A","B"}, {{1,2}}) & #table({"B","C"}, {{3,4}})
→ #table({"A","B", "C"}, {{1,2, null}, {null, 3,4}})
. This can also be achieved using the function Table.Combine({ #table({"A","B"}, {{1,2}}), #table({"B","C"}, {{3,4}}) })
.
Once you have a table, it is often necessary to access specific items within the table directly.
TableName[ColumnName]
→ the result is a list. Therefore Value.Is(#table({"A","B"},{{1,2}})[A], type list)
returns true
. If you want to learn more about lists, read this post.
TableName{zero-based row index}
→ the result is a record. Therefore Value.Is(#table({"A","B"},{{1,2}, {3,4}}){1}
, type record) returns true
. If you want to learn more about records, read this post.
Accessing a cell in a table brings accessing a row and accessing a column together:
TableName[ColumnName]{zero-based row index}
which is equivalent to TableName{zero-based row index}[ColumnName]
, column and line references can therefore be swapped.
The result has no predictable type because it returns what is in the cell and that can be anything: a scalar value, a list, a record, another table, a function, etc.
If you don’t want to address only one column, one row, or a certain cell of a table, but need to select several columns or rows by condition, there are functions in M that help you. Here are a few examples:
Table.SelectRows() → Returns a table of rows from the table, that matches the selection condition.
Table.SelectColumns() → Returns a table with only specific columns that match the selection conditions. The third and optional parameter missingField of this function is very useful, which says what to do if the addressed field (column) does not exist.
Table.ColumnsOfType() → This function is very useful, if you want to select columns of your table, which match a specified type or a list of types. That way you can for example select all columns that are of type text. Although this function can be very helpful, it does not always return the expected columns. I’ve written a post about this that deals with pitfalls with Table.ColumnsOfType.
If you want to get meta information about the table you are using, the following two functions will help you:
Table.Schema() → This function returns information about the columns of the table, like:
Table.Profile() → Returns the following information for the columns in a table:
Table.Buffer() → This is a special function, which is used, among other things, when it comes to query performance optimization, as it keeps a snapshot of the table in memory. There are several good blog posts out there, that cover this topic. See Imkes post about Table.Buffer inside List.Generate to improve performance.
Even though there is much more to say about tables and their capabilities, I hope this was a helpful introduction to the topic
Greetings from Germany,
Lars
Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…
Der Beitrag Tables in Power Query – how, when and why erschien zuerst auf THE SELF-SERVICE-BI BLOG.
]]>