Als Abonnent meines Newsletters erhältst Du die Beispieldateien zu den Beiträgen dazu. Hier geht’s zum Abonnement des Newsletters!

Klicke auf die Abbildung, um zum Video auf Youtube zu gelangen:

Bis zum nächsten Mal und denk dran: Sharing is caring. Wenn Dir der Beitrag gefallen hat, dann teile ihn gerne. Falls Du Anmerkungen hast, schreibe gerne einen Kommentar, oder schicke mir eine Mail an lars@ssbi-blog.de

Viele Grüße aus Hamburg,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Video: Lists, records und tables in M für Power Query in Power BI und Excel erschien zuerst auf THE SELF-SERVICE-BI BLOG.

Hierbei handelt es sich um ein Funktion, die ich auf GitHub veröffentlicht und dann in meine Website eingebettet habe. Um diese Funktion zunächst kopieren zu können, klicke bitte auf „view raw“ in der unteren rechten Ecke.

Den Funktions-Code kopieren

Du wirst dann auf meine GitHub-Seite weitergeleitet, die Dir den gesamten M-Code im Browser darstellt. Kopiere diesen unter Verwendung der Tastenkombinationen Strg+a (um alles zu markieren) und anschließend Strg+c (um alles zu kopieren).

Den Funktions-Code in Power Query einfügen

Mit dem M-Code in der Zwischenablage, öffne eine leere Abfrage in Power Query für Power BI Desktop oder Excel. Der folgende Screenshot zeigt Dir, wie Du das in Power BI Desktop machen kannst:

Anschließend musst Du den M-Code in den Advanced-Editor einfügen und zuvor das automatisch hinzugefügte let-Statement entfernen:

Der Funktion einen sprechenden Namen geben

Nachdem Du die Funktion nun in eine leere Abfrage eingefügt hast, ist es sehr sinnvoll dieser Abfrage einen sprechenden Namen zu geben, denn der Name der Abfrage ist der Name der Funktion. Dies ist deshalb wichtig, weil Du die Funktion immer auf Basis ihres Namens aufrufen wirst. „Abfrage1()“ ist da sicherlich nicht so sinnvoll wie „fnCalendarTable“. Ändere den Namen im Eigenschaftenfenster der Abfrage:

Die Verwendung der Funktion

Wie Du die nun eingefügte Funktion in Power Query nutzen kannst, hängt ganz davon ab, was es für eine Funktion ist. Einige können in berechneten Spalten aufgerufen werden, andere werden einfach über einen Klick auf „Aufrufen“ (die Schaltfläche findest Du im vorangegangenen Screenshot) ausgeführt und geben dann meist (aber nicht ausschließlich) eine Tabelle zurück.

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag So verwendest Du geschriebene M-Funktionen in Power Query wieder erschien zuerst auf THE SELF-SERVICE-BI BLOG.

Recommended ressources about standard error handling in M

Of course some M-enthusiasts in the community have already dealt with the topic of error handling and I recommend you to have a look at these as well:

• https://www.poweredsolutions.co/2019/10/28/step-level-error-in-power-bi-power-query/
• https://www.poweredsolutions.co/2019/06/18/error-handling-iferror-in-power-bi-power-query/
• https://blog.crossjoin.co.uk/2014/12/22/viewing-error-messages-for-all-rows-in-power-query/

Miguel Escobar highlights in the first of the mentioned posts, that you can raise your own (custom) error messages, if you want to.

What are custom errors and when are they useful?

Power Query/ M returns its own error messages when you try to evaluate an expression that M cannot evaluate. For example, the expression A + 1 returns the following error message.

But what if you want an error message to be returned on an expression that could be evaluated by M but doesn’t fit into your business logic?! In the following example, I want an error message to be returned whenever the tax percentage is not 7% or 19%.

The purpose of custom errors is to return error messages when the expression is technically evaluable but contradicts your own business logic. In this case, it is good and right to provide the user with the right information so that he can correct the error as quickly as possible. Let’s see how this can be done.

The manual version

You can throw a custom error by using the keyword ‚error‘ followed by the function Error.Record(). Within the function you can define up to 3 arguments, which are:

• reason as text,
• optional message as nullable text and
• optional detail as any

The following screenshot shows how to manually generate this error message in a calculated column:

But what if I want to check for this (logical) error in many places in my queries? Do I then have to write this Error.Record manually each time, or copy it from one of my previous solutions? To avoid this manual process, I have written my own function that fixes this problem!

Using a custom error function for convenience

Whenever it makes sense, I outsource tasks to custom functions that I can then reuse. This reduces the risk of errors, makes my code clearer and easier to maintain. My goal in calling my custom error therefore looks like this:

So instead of manually rewriting (or copying) the Error.Record at any necessary point in my code, I want to define it centrally at one point and then call it (based on its ID) via a function from any of my queries. I’ll show you how to do this now.

The function fnReturnCustError()

I have commented on the functionality of my function directly in the source code and hope that this is sufficient to understand it.

I hope this was interesting for one or the other. I am sure that this method can be further developed. If you succeed, please let me know.

Greetings from Germany,
Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag How to handle custom errors in M gracefully erschien zuerst auf THE SELF-SERVICE-BI BLOG.

Syntax of the function

The syntax of the function is as follows: Table.Profile(table as table, optional additionalAggregates as nullable list) as table. The first mandatory parameter is the table whose profile is to be created. The second parameter is a puzzle. Let’s take a closer look at this one.

Second parameter «additionalAggregates»

This argument is expected as a list of lists, like this: {{},{}}. Each of these inner lists represents a new calculated profile column and consists of three items:

1. item: Output column name as text
2. item: typecheck function as function
3. item: aggregation function as function

Let’s take a detailed look.

Output column name

This is just the name of the new profile column, which of course should reflect the content of the column.

Typecheck function

The typecheck function is expected to be something like each Type.Is(_, type any). While the keyword „each“‚ refers to the current record in the profiling table, the underscore „_“ refers not – as you might expect – to the current record, but to the column of the source table, whose name can be found in the current record in the first column „Column“. So the underscore „_“ doesn’t return a record, but a list. This information will also be important for the third item of the parameter additionalAggregates: the aggregation function.

Aggregation function

If the second parameter – the typecheck function – returns true, then the aggregation function kicks in. Otherwise the output of the aggregation function will be null.

So how could examples of the aggregation function look like?

Aggregation function – example #1

= Table.Profile(Table.FromRows({{1, 2}, {3, null}}, type table [A=any, B=number]), {{"New profile column", each Type.Is(_, type number), each List.Average(_)}})

What happens in the profile table?

• Column A: This column is of type any. My function is explicitly looking for type number (Type.Is(_, type number)), so the function List.Average() is not used for column A. The value null is returned accordingly.
• Column B: This column is of type number. My function is explicitly looking for type number (Type.Is(_, type number)), so the function List.Average() can kick in. The value 2 is returned as the average of 2 and null is 2.

Aggregation function – example #2

= Table.Profile(Table.FromRows({{"ABC", 2}, {"C", 3}}, type table [A=text, B=any]), {{"AverageCharactersCount", each Type.Is(_, type text), each List.Average( List.Transform(_, each Text.Length(_))) }})

What happens in the profile table?

• Column A: This column is of type text. My function is explicitly looking for type text (Type.Is_, type text), so the function List.Average( List.Transform(_, each Text.Length(_))) is used for column A. The returned value is 2 as the average text length of „ABC“ and „C“ is 2. 
• Column B: This column is of type any. My function is explicitly looking for type text (Type.Is_, type text), so the function List.Average( List.Transform(_, each Text.Length(_))) doesn’t kick in, but returns null.

When using the UI to assert types

When you change a data type using the UI, the M function Table.TransformColumnTypes() is used behind the scenes to do that job. What you should always keep in mind when using this function is, that it always returns a nullable data type. This is something I forgot, even if it was a central message of this of my own recent posts. Why is this important in view of the last post? Take a look at the following examples:

In the first screenshot I use ascribing types and my custom column to Table.Profile() just works.

In the second screenshot I use the UI/ Table.TransformColumnTypes() to assert the type and my function returns null, where I expected the value 2. Why does that happen. As I said, in my example Table.TransformColumnTypes() returns a nullable text type and Type.Is(type nullable text, type text) = false. This is why I get back null.

To fix this behavior, I have to check for type nullable text, instead of type text:

The moment you do it right, it’s running.

Greetings from Germany,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Table.Profile and its unknown second parameter erschien zuerst auf THE SELF-SERVICE-BI BLOG.

Als Abonnent meines Newsletters erhältst Du die Beispieldateien zu den Beiträgen dazu. Hier geht’s zum Abonnement des Newsletters!

Anmerkung

Der nachfolgende Beitrag war ursprünglich deutlich länger und detaillierter. Manchmal – Gott sei Dank in diesem Fall zum ersten Mal – kommt es jedoch vor, dass sich ein Leser bei mir meldet und mir mitteilt, dass ich inhaltlichen Unsinn geschrieben habe. In diesem Fall war es der Kollege Frank Tonsen, der sich nicht nur die Zeit nahm, meinen Beitrag zu lesen, sondern auch die notwendige Aufmerksamkeit besaß, meinen Fehler zu entdecken. Mehr noch: Nachdem Frank und ich eine kurze E-Mailkonversation zu diesem Thema hatten, schickte er mir sogar seine Version der Lösung, die ich in leicht angepasster Form hier abdrucke. @Frank: Vielen Dank. Derartige Reaktionen sind es, die ich an dieser Community so mag

Die Zielstellung

Die Aufgabe bestand darin, Datums- und Uhrzeitangaben, die als koordinierte Weltzeit (UTC – Coordinated Universal Time) von einem Server geliefert wurden, in unsere lokale Zeit umzurechnen. Aufgrund der in Deutschland herrschenden Regelung der Sommer- und Winterzeit kann zur UTC-Zeit jedoch nicht pauschal eine Stunde addiert werden, da die Zeitdifferenz während der Winterzeit 2 Stunden beträgt. Ich möchte die Problemlösung als benutzerdefinierte Funktion zur Verfügung stellen.

Widmen wir uns der Problemlösung.

Problemlösung

Generell kann zur Zeitverschiebung zwischen UTC und unserer Zeit in Deutschland folgendes gesagt werden:

• Unsere Sommerzeit beginnt am letzten Sonntag im März. Um 2 Uhr lokale Zeit (d. h. 1 Uhr UTC), wird die Uhr auf 3 Uhr vorgestellt. Während der Sommerzeit beträgt die Zeitverschiebung daher 2 Stunden
• Unsere Winterzeit beginnt am letzten Sonntag im Oktober. Um 3 Uhr lokale Zeit (d. h. 1 Uhr UTC), wird die Uhr auf 2 Uhr zurückgestellt. Während der Winterzeit beträgt die Zeitverschiebung daher 1 Stunde.

Für das aktuelle Jahr 2019 sieht es also wie folgt aus:

Am 31.03. (letzter Sonntag im März) wurde um 1 Uhr UTC die Zeitverschiebung von 1 Stunde, auf 2 Stunde erhöht. Dadurch folgte auf die lokale Uhrzeit 01:59:59 Uhr nicht 2:00:00 Uhr, sondern 03:00:00 Uhr:

Am 27.10. (letzter Sonntag im Oktober) wurde um 1 Uhr UTC die Zeitverschiebung von 2 Stunden, auf 1 Stund reduziert. Dadurch folgte auf die lokale Uhrzeit 02:59:59 Uhr nicht 3:00:00 Uhr, sondern 02:00:00 Uhr:

Die benutzerdefinierte Funktion, die all das berücksichtigt, sieht wie folgt aus:

Falls Du wissen möchtest, wie Du meine M-Funktionen in Deinen Projekten wiederverwenden kannst, schau bitte hier nach.

Bis zum nächsten Mal und denk dran: Sharing is caring. Wenn Dir der Beitrag gefallen hat, dann teile ihn gerne. Falls Du Anmerkungen hast, schreibe gerne einen Kommentar, oder schicke mir eine Mail an lars@ssbi-blog.de

Viele Grüße aus Hamburg,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Leserfrage: Umrechnung UTC-Zeit in lokale Zeit mit Power Query erschien zuerst auf THE SELF-SERVICE-BI BLOG.

What is a table in M?

In M there are two kinds of values: primitive and structured values. Examples for primitive values are:

• „a“,
• 1,
• true.

They are primitive in that way, that they are not contructed out of other values. In contrast to primitive values, we have so called structured values in M, which are composed of other values, primitive and structured ones. A table is one of those structured values (the others are lists and records) and it is described as „a set of values organized into columns (which are identified by name), and rows.“

Even if tables usually contain columns and rows, they can be empty, which looks like this: #table({},{}).

An example of a very simple representative of a table is:

#table( {"column_1", "column_2"}, { {1, 2}, {3, 4} } )

Before taking a more detailed look at tables of what they are and what they are for, let’s discuss why to use tables at all.

Why use table at all?

Tables are often the final result of a query, which can then either be used as an intermediate query or loaded into the data model in Power BI and Power Pivot. In addition, there are functions that work with tables as input parameters and others that generate tables as return values. For these reasons it is necessary to know how to deal with them in order to use the M language safely.

How to create tables in M with native M functions

There is no literal syntax to create a table, like for records ([]) and lists ({}), but there are several native M functions, that create tables.

#table()

This function (it is spoken „pound table“) is something special, as it is not (yet) returned by the intrinsic variable #shared. Since the first parameter of the function can be used in different ways, there are several ways to use this function.

The second parameter is always a list of list, filled with values. Each inner list represents a row within the table. Let’s take a look how the first parameter can be used in different ways.

Version #1 – Specifying column names in a list

#table( {"A", "B"}, { {1, 2}, {3, 4} } ) →By specifying the column names as text values in a list, every column gets its specific column name.

Version #2 – Defining a number of columns

#table(5, {{ 1,2,3,4,5}} ) → If I need to create a known number of columns (e.g. 100), but they only need to have values and no specific column headings, I can use this syntax.

Version #3 – Using so called ascribed types to define column names and types

#table( type table [A = number, B = text], {{1,"one"}, {2,"two"}, {3,"three"}} ) → With this version you can define not only the column headings and the corresponding values, but also the data types of the individual columns.

I will describe why using ascribed types for defining types of values in a column of a table is a BAD IDEA later in a separat section of this post. Now let’s take a look at functions that create tables from other values in M.

Conversion functions: Table.From*()/ *.ToTable

Conversion functions create a table from other values, like lists, records etc. Here a few prominent representatives:

• Table.FromList()
• Table.FromRecords()
• Record.ToTable()

Other M functions that return tables

In addition to the conversion functions, there is a ton of other functions returning tables in M. First of all I want to mention those functions that allow you to define tables yourself by giving them lists built in either row or column logic:

• Table.FromColumns()
• Table.FromValue()
• Table.FromRows()

In addition to these functions, there are many more that return tables. Most of them are connectors to external data sources. Examples of this are: Csv.Document(), DataLake.Files(), or Github.Tables().

Don’t use ascribed types for type definitions

Dealing with types in M is really tiresome. In several places I stumbled across problems in M whose solution was to be found in M’s type system. I had written under Version #3 that it is not recommended to ascribe types to the values of a column in a table. I wrote a blog post about it, which describes the reasons in detail and I recommend you read it, before going further in this post. In short Power Query accepts your type definition via ascribed types as correct and does no validation, wether the declared type of the column is compatible with the values in that column. Only when you try to load these values into another engines (Excel data model, the Power BI data model, or pure Excel) will you get error messages because these systems recognize the error.

Operators for tables + equivalent functions

There are 3 operators that can be used in conjunction with tables: „=“ and „<>“ make it possible to compare tables, while „&“ combines tables. Here are some examples of how to use that:

#table({"A","B"},{{1,2}}) = #table({"B","A"},{{2,1}}) → true. Tables are equal if all of the following are true:

• The number of columns is the same.
• Each column name in one tables is also present in the other table (regardless of its position in the table).
• The number of rows is the same.
• Each row has equal values in corresponding cells.

#table({"A","B"},{{1,2}}) = #table({"B","A"},{{2,1}, {3,4}}) → false, due to a different number of rows.

#table({"A","B"},{{1,2}}) <> #table({"B","A"},{{2,1}, {3,4}}) → true

#table({"A","B"}, {{1,2}}) & #table({"B","C"}, {{3,4}}) → #table({"A","B", "C"}, {{1,2, null}, {null, 3,4}}). This can also be achieved using the function Table.Combine({ #table({"A","B"}, {{1,2}}), #table({"B","C"}, {{3,4}}) }).

Accessing table elements

Once you have a table, it is often necessary to access specific items within the table directly.

Accessing a single column

TableName[ColumnName] → the result is a list. Therefore Value.Is(#table({"A","B"},{{1,2}})[A], type list) returns true. If you want to learn more about lists, read this post.

Accessing a single row

TableName{zero-based row index} → the result is a record. Therefore Value.Is(#table({"A","B"},{{1,2}, {3,4}}){1}, type record) returns true. If you want to learn more about records, read this post.

Accessing a single cell

Accessing a cell in a table brings accessing a row and accessing a column together:

TableName[ColumnName]{zero-based row index} which is equivalent to TableName{zero-based row index}[ColumnName], column and line references can therefore be swapped.

The result has no predictable type because it returns what is in the cell and that can be anything: a scalar value, a list, a record, another table, a function, etc.

Special functions to select elements in a table

If you don’t want to address only one column, one row, or a certain cell of a table, but need to select several columns or rows by condition, there are functions in M that help you. Here are a few examples:

Table.SelectRows() → Returns a table of rows from the table, that matches the selection condition.

Table.SelectColumns() → Returns a table with only specific columns that match the selection conditions. The third and optional parameter missingField of this function is very useful, which says what to do if the addressed field (column) does not exist.

• MissingField.Error → Default: Return an error
• MissingField.Ignore → Don’t select the column
• MissingField.UseNull → Fill the column with null values

Table.ColumnsOfType() → This function is very useful, if you want to select columns of your table, which match a specified type or a list of types. That way you can for example select all columns that are of type text. Although this function can be very helpful, it does not always return the expected columns. I’ve written a post about this that deals with pitfalls with Table.ColumnsOfType.

Other special table functions

If you want to get meta information about the table you are using, the following two functions will help you:

Table.Schema() → This function returns information about the columns of the table, like:

• the 0-based position of the column in table,
• the name of the type of the column,
• nullability,
• maximal length,
• …

Table.Profile() → Returns the following information for the columns in a table:

• minimum,
• maximum
• average
• standard deviation,
• count
• null count
• distinct count.

Table.Buffer() → This is a special function, which is used, among other things, when it comes to query performance optimization, as it keeps a snapshot of the table in memory. There are several good blog posts out there, that cover this topic. See Imkes post about Table.Buffer inside List.Generate to improve performance.

Even though there is much more to say about tables and their capabilities, I hope this was a helpful introduction to the topic

Greetings from Germany,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Tables in Power Query – how, when and why erschien zuerst auf THE SELF-SERVICE-BI BLOG.

Preface

The function Table.ColumnsOfType() allows you to retrieve a list of columns that match specified types (second argument) from a given table (first argument). This can be very helpful, for example, if you want to select all text columns of a given table.

The syntax of the function is as follows: Table.ColumnsOfType(table as table, listOfTypes as list) as list.

So let’s see how it works with an example.

The starting point

My starting point was the following script, which…

1. Creates a table ‚Source‘ and ascribes types to each of the columns,
2. Returns a list of all column names from columns of type text,
3. Selects those columns from table ‚Source‘

The following screenshot shows the script and corresponding views in Power Query:

Everything works as expected. And now let’s try to reproduce this without ascribing the types, but using the UI…

In the example shown above, I use exactely the same table as in the previous example for step Source. Both tables are defined in M, so they are not imported from any external data source. After that I use the UI to change the data types of columns A and B, which leads to step #"Changed Type". This creates the corresponding M code and you can see that it assigns the type number to column „A“ and the type text to column „B“, as in the previous example. Step ColumnsToSelect and Output are exactly the same as in the previous example. But as you can see from the screenshots to the right of the M code, the list that filters for columns of type text(step ColumnsToSelect) returns no value. This means that no column can be selected in step Output. What’s going on here?

What happened here?

To bring a little more light into the darkness, I apply the function Table.Schema() to both examples after the types have been assigned. For the first working solution, Table Schema() returns the following:

For the variant that did not return a result for me, Table.Schema() returns this result:

I was using Table.Schema() immediately when I encountered this problem. However, I didn’t notice the difference with IsNullable. It was the Power Query Dev team who made me aware of this difference. Many thanks for that. But why is it that when using the UI for the type declaration all columns are nullable? Very simple: The used function Table.TransformColumnTypes() – although the M-Code does not contain anything about ’nullable‘ – automatically returns nullable datatypes. This is rarely a problem. But in my example, which uses the function Table.ColumnsOfType(), it becomes one, because this function needs an ‚exact match‘ for the type to be selected.

So what’s the solution?

Solution

The thing is that nullability matters for Table.ColumnsOfType(). And if the function Table.TransformColumnTypes() always returns a nullable data type, then the function Table.ColumnsOfType() must also search for such a data type. The solution is as follows:

A quite simple solution, that has been driving me crazy for a while.

Conclusion

1. Within the function Table.ColumnsOfType() it must be exactly defined, which type the columns of the table have, in order to come to a result. This also includes nullability.
2. From the pure M code it cannot be read whether the generated type is nullable or not. The Table.Schema() function is very helpful here.
3. Table.TransformColumnTypes() is not the only possible cause for ‚unwanted‘ nullability. If you took a closer look at my first example with the ascribed types in this post, then the column „C“ of type any was converted to nullable any as well. This was only made clear by Table.Schema().

Greetings from Germany,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Pitfalls with Table.ColumnsOfType erschien zuerst auf THE SELF-SERVICE-BI BLOG.

Why types in M are important

From the first moment you use Power Query, you are confronted with the M type system. Especially if you don’t come from a database environment, but from Excel, this is quite confusing in the beginning, because if the type is not right, Power Query doesn’t do what you want. Take the following example: Columns addend_1 and addend_2 contain numeric values. However, they differ in that column addend_1 has the type whole number, while addend_2 has the type text. When you try to sum both values, the result column only returns error messages, because M cannot sum numeric values and text values:

If I change the type of the column addend_2 from text to whole number, the result can be calculated correctly:

So much for the basics that probably every Power Query user has come across before.

Different ways to define types for table columns in M

Since types in M are important, there are several ways to define them: Some via the GUI, others via M code only.

Table.TransformColumnTypes()

There are two known ways to change the type of a table column via the GUI. In both cases the function Table.TransformColumnTypes() is used under the hood.

Ascribed Types in M

In M it is possible to define types yourself as well as tables, lists, records and other values. A defined table type for the table shown in the screenshot could therefore look as follows:

type table[Name = text, Age = number]

The type alone doesn’t help me much, but there are (at least) two ways to assign such a type to a table…

Using Value.ReplaceType

Changing the type of a value (a table is a value in M) is possible by using the function Value.ReplaceType(), as you can see in the following example:

Using a table function, that allowes types as parameter

There are a couple of table functions in M, that accept a table type as parameter. Here are two examples:

#table()

#table() accepts a table type as the first parameter. This makes it possible to pass the type directly inside the function:

Table.FromColumns()

Table.FromColumns() accepts a table type as a second parameter. This makes it possible to pass the type directly inside the function:

In both cases the result is as follows:

But if the result of using ascribed types looks completely correct, why not use them?

Why to avoid ascribed types for tables in M?

So far I have dealt with cases in which the type conversion of the respective column could take place without any problems. But have a look at the following example:

The screenshot above shows two examples for defining the type of table columns:

On the left you can see the version with ascribed types, using Value.ReplaceType(). On the right you see the same using Table.TransformColumnTypes(). What both examples have in common is that a value has crept into the Age column („ABC“) that is not consistent with the declared type. And this is where the difference between ascribed types and using a corresponding function like Table.TransformColumnTypes() becomes apparent:

The left version simply ascribes a type to a table. Because errors are not part of the type system, you won’t see any errors, even if a value of a column is not consistent with the declared type.

The right version uses the function Table.TransformColumnTypes() and this one does more than only asserting a type: It either transform the specific value of the column, or returns an error, if not possible.

So the ascribed type is always exactly what the user has defined and there is no validation whether the declared type of the column is compatible with the values in the column.

But there is an exception…

Loading tables with ascribed types into the data model or Excel

While the Mashup Engine does not validate the consistency of ascribed types and values in a column, such validation occurs when the results of a Power Query query are either loaded into Analysis Services Tabular (for example, Power BI Data Model/Excel Data Model) or loaded directly into an Excel sheet. I now show what happens when you want to load a table into the Power BI data model, where the ascribed type is not consistent with each of the column values. It looks the same when loading into an Excel sheet:

I load the table I just looked at into the Power BI data model, in which the value „ABC“ appears in column Age…

When loading I get the message that my query contains errors…

When clicking on „View errors“ I get an empty query which is anything but helpful…

Here the dog bites its tail, because the empty error message comes from the fact that the query behind it uses the function Table.SelectRowsWithErrors(#"Added Index", {"Name", "Age"}), which searches for „Error“ in the corresponding source table. However, these errors do not appear due to ascribed types… You understand the dilemma????…

Take a moment to digest this: Imagine, you have loooong tables and have worked with ascribed types. In the table there are no errors visible, although they should be visible. When loading into the data model you will get an error message and if you click on „View errors“ you will see an empty table. This is really not easy to understand, confusing an misleading.

The data loaded into the data model despite an import error then looks like this: The value „ABC“ is simply missing…

Okay, everything looks like using ascribed types isn’t a good idea in principle – but they can also be very useful…

When to use ascribed types in M?

Ascibed types should not be used to declare types as you have seen so far, but they should be used to attach metadata to type values. This is useful, for example, if you want to integrate the documentation of a custom function into the metadata of the function. I described this in great detail in my series ‚Writing documentation for custom M functions‚.

Conclusion

Ascribed types are useful if I want to change the metadata of a value. This can be the case if I want to attach documentation to a custom function.
For the declaration of types of one or more table columns I can only strongly advise against the use of ascribed types, because there is no check whether the contained values are type-compliant. I also get an error message when loading into the data model or into an Excel sheet, but I have no obvious way to identify them with Power Query.

Greetings from Germany,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Ascribed types in M: why to generally avoid them erschien zuerst auf THE SELF-SERVICE-BI BLOG.

What is a record in M?

In M there are two kinds of values: primitive and structured values. Examples for primitive values are:

• „a“,
• 1,
• true.

They are primitive in that way, that they are not contructed out of other values. In contrast to primitive values, we have so called structured values in M, which are composed of other values, primitive and structured ones. A record is one of those structured values (the others are lists and tables) and it is described as „a set of fields. A field is a name/value pair where the name is a text value that is unique within the field’s record.“

A defined record is introduced with an opening squared bracket „[“ and ended with a closing squared bracket „]“. Even if records usually contain fields, they can be empty, which looks like this: [].

An example of a very simple representative of a record is:

[A=1, B=2]

Before taking a more detailed look at records of what they are and what they are for, let’s discuss why to use records at all.

Why use records at all?

In the vast majority of cases, a record will be used as an intermediate product for other queries. In any case, it is extremely rare for a record to be loaded into the data model. If a record in Power BI is nevertheless loaded into the data model, it behaves like a table. Field names become column captions and field values become values of the table.

Reasons for using records in Power Query can include the following: You…

• … want to create a record to pass as a parameter to a native M function,
• … are using a function that creates a record as a return value and want to process it further,
• … want to access/ work with rows in tables,
• … would like to create documentation for custom functions, using the meta data record in M,
• … want to implement proper error handling within your m scripts,
• … and many more

Let’s take a look at how to create records.

How to create records in M

In my experience there are 3 ways in M to create a record:

1. Using the record initialization syntax with squared brackets

You can define a record by using an opening squared bracket „[„, and optional field-list and end it with a closing squared bracket „]“. The simplest record is an empty record.

Empty record

The empty record is the simplest of all records, in which the field-list is empty:

[] → Record.FieldCount([]) = 0

An empty record is of course more of a special case, because records normally contain fields with values.

Non-empty records

To create a non-empty record, I can add comma seperated set of fields. Fields are pairs of names of type text and values of any possible type:

In the screenshot above you can see two things:

• The field names can be written without quotation marks, even if they contain spaces.
• The values of records (as well as lists) can contain values of any type.

For field names it is important that no field name occurs more than once within the same record. These are used as unique identifiers. If this rule is violated, the following error message is displayed:

Defining records manually is one way. Let’s look at how to create records with corresponding functions.

2. Using native M functions, that create records

At the time of writing, we have 26 functions in M that return a record. Popular examples are:

• Record.FromList() 
• Record.FromTable()
• Date.ToRecord()
• DateTime.ToRecord()

Calling the intrinsic variable #shared returns a record, containing all names and values of all the queries (tables, constants, records, lists, native M functions, custom functions, etc.) in the current Excel/ Power BI Desktop file:

All those functions create records based on the input parameters. But there is another way to create a record.

3. Referring to a row in a table

The third way to get a record is to reference a row of a table using the following syntax: Table{row number}:

Note that the row reference in a table, like list-items, is based on a zero-based index. So the value 1 must always be subtracted from the desired row number to reference the desired row.

If you create a calculated column within a table, you can use the underscore „_“ to reference the current row of the table, which means the current record. This makes the following formulas possible:

The above formula works as follows:

1. The underscore „_“ represents the current row/ record of the table
2. Record.ToList(_) converts this record to a list
3. List.Sum() calculates the sum of all the fields of the given record

Now that we know what records are and how they are created, we are going to focus on dealing with records.

Operators for records+ equivalent functions

There are 3 operators that can be used in conjunction with records: „=“ and „<>“ make it possible to compare records, while „&“ combines/ merges records. Here are some examples of how to use that:

Record comparisons

The following examples show that the order of the record fields is irrelevant for a comparison. As long as the records to be compared have the same fields, i.e. the same field names and values, the records are identical:

[ a = 1, b = 2 ] = [ b = 2, a = 1 ] → true
[ a = 1, b = 2, c = 3 ] <> [ a = 1, b = 2 ] → true

Record merge

Sometimes you don’t want to compare different records with each other, but combine them. This succeeds as follows:

[ a = 1, b = 2 ] & [ c = 3 ] → [ a = 1, b = 2, c = 3 ]
[ a = 1, b = 2 ] & [ a = 3 ] → [ a = 3, b = 2 ]. This can also be achieved by using the function Record.Combine({ [ a = 1, b = 2 ],[ a = 3 ]})

ATTENTION: The second example shows, that when combining/ merging records the fields from the most right record will override fields from the left record(s), should there be an overlap in field names.

Another aspect that is interesting with regard to records is how to access the fields in a record.

Accessing record fields

Once you have a record, it is sometimes necessary to access special fields within the record directly.

Using the lookup operator []

To access an field in a record, you can use the – so called – lookup operator „[]“ to search for record fields by name. The following examples explain this in detail:

Take the following record: MyRecord = [A=1, B=2].  Here are some examples of how to access the record fields of this record:

MyRecord[A] = 1,

MyRecord[B] = 2,

MyRecord[C] leads to the following error message, saying that an attempt is made to access an field of the record that does not exist in the record:

To overcome this last error message, you can use the so called optional-field-selection as follows, which returns null, if the selected field doesn’t exist:

MyRecord[C]? → null

Reference to record fields based on their position in the record

In lists, the corresponding item is referenced by its position within the list. Unfortunately, this is not so easy in records. In records the reference is (normally) based on the field name, but not on the position of the field within the record. If this is exactly what I need, I have to go a few detours.

The function Record.SelectFields() includes a record (MyRecord) in the first parameter from which certain fields are to be selected. The fields to be selected can be defined in the second parameter. The second parameter can be of any type. Therefore I can refer here to the function Record.FieldNames(), which returns all field names of the records as a list, and then use the positional index operator {} to select the item in the list (ATTENTION: zero-based index), which I want to have.

Using native M functions

Due to the fact that fields in records are referenced on the basis of their field names and not on the basis of their position as in lists, there are hardly any functions for records that would be equivalent to List.First(), ListLastN(), etc.. The only example I can think of on this topic is a modification of the above example to combine Record.SelectFields() with any second parameter.

Functions that use records as input

If I look at the functions that include a record as a function parameter at the moment, I see only 30 functions in M. If I add those functions where the record can also be „nullable“ (which means the parameter is optional), I get 87 functions.

An interesting representative of functions that record records as parameters is the function Web.Contents(). The function has the following syntax:

Web.Contents(url as text, optional options as nullable record) as binary 

While the first parameter only takes up the url as text, the optional parameter „options“ hides a record with a large variety of settings. Take a look at the following screenshot:

At this point I will not go into the individual fields of the record, because this could fill its own post. However, it is good to see that this record contains a large number of fields of different types and thus influences the behavior of the Web.Contents() function.

Functions that create records as output

The function DateTime.ToRecord() converts a datetime value into a record:

When I use this function in a calculated column within a table, it looks like this:

In step 1 I refer to the value in column „Now“ for the calculation of the new column. Then I expand the created record in step 2 and create the 6 new columns Year, Month, Day, Hour, Minute and Second.

Finally, let’s have a look at some very special forms of records.

Special records

M has special forms of records, which are needed for certain purposes.

Metadata record

Each value in M has a so-called metadata record, which is empty by default. You can add content to this record by using the keyword meta after an expression (in my example the expression 1). To retrieve the saved metadata information, you can use the function Value.Metadata(), with a reference to the value (Source), in which the expression was saved.

Using the metadata record to document custom functions

The metadata record can be used to document custom functions within the function and to provide help to users of the function. I have written a three-part article about this in the past that explains in great detail how this works.

The error record

M has an arsenal of error messages in place if it was not possible to get a value from an expression. For example, try to create a record with two identical record field names and you get the following error message:

The record couldn’t be evaluated, due to identical field names. But there are situations in M where the language itself cannot identify an error, but I want to raise an error in the logic of my code:

Imagine I want to write a custom function that does nothing more than return the given value. This is certainly not meaningful, but an easy to understand example: (x) => x.
If I now want this function to return all values, unless it is the value 0 (zero!), then I have to generate the error myself, if for M the inclusion of the value 0 as parameter is no error at all. Take a look at the follwing M code:

Within the function Error.Record() I define a record, which contains the 3 fields:

1. (error) reason,
2. (error) message and
3. (error) detail. 

I am then able to call the record created by this function  elsewhere with – in my example – „error ErrIfZero“ if it is necessary.

This was certainly not an exhaustive enumeration of the characteristics of records, but I hope they have brought you a little bit closer to this topic and also explained why you have to deal with records if you want to master the M language.

Greetings from Germany,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Records in Power Query – how, when and why erschien zuerst auf THE SELF-SERVICE-BI BLOG.

What is a list in M?

In M there are two kinds of values: primitive and structured values. Examples for primitive values are:

• „a“,
• 1,
• true.

They are primitive in that way, that they are not contructed out of other values. In contrast to primitive values, we have so called structured values in M, which are composed of other values, primitive and structured ones. A list is one of those structured values (the others are records and tables) and it is described as „an ordered sequence of values“. A defined list is introduced with an opening curly bracket „{“ and ended with a closing curly bracket „}“. Even if lists can be empty (it looks like this „{}“), lists are not limited in size, because M supports infinitely large lists.

An example of a very simple representative of a list is:

{1,2}

Before taking a more detailed look at lists of what they are and what they are for, let’s discuss why to use lists at all.

Why use lists at all?

In the vast majority of cases, a list will be used as an intermediate product for other queries. In any case, it is extremely rare for a list to be loaded into the data model.

There are many native functions in M that provide useful functionality and expect lists as one or more of their parameters. For these cases it is important to know how to create lists, so that you can provide those functions with their necessary parameters in a proper way.

In addition, there are many functions in M that return lists as return values. For these reasons it is necessary to know how to deal with them in order to use the M language safely.

Let’s take a look at how to create lists.

How to create lists in M

A list in M usually has a beginning and an end and in my experience there are 3 ways in M to create a list:

1. Using the list initialization syntax with curly braces

You can define a list by using an opening curly bracket „{„, an optional item-list and end it with a closing curly bracket „}“. The simplest list is an empty list.

Empty lists

The empty list is the simplest of all lists, in which the item-list is empty:

{} →List.Count({}) = 0

A practical use case of an empty list is, for example, creating a table in the Power BI data model that contains only measures. Unlike an empty table created in Power Query (which you can create like this: = #table({},{})), which simply doesn’t appear in the data model, an empty list is imported as a table into the data model with one column and no rows. Simply hide the resulting column in the data model and use this empty table as the measure table.

An empty list is of course more of a special case, because lists normally contain values, so-called items.

Lists with comma seperated items

To create a non-empty list, I can add comma separated values of any type to the list as follows:

If I want to add a large number of contiguous values to a list, the comma separated definition is quite complex. But there is help for that…

Lists with contiguous numbers

For example, if I want to create a list that shows the whole numbers from 1 to 100, I don’t have to manually add all these numbers to the list. Instead there is the following abbreviation:

{1..100}

If, for example, I would like the whole numbers to be listed from 1 to 100, but the number 48 should be excluded, then this works as follows:

{1..47, 49..100}

Such a sequence of contiguous values must not have more than 2^31 (2.147.483.647) values. For example, look at the following list definition:

{1..2147483648}

Due to the fact that a range of numbers greater than 2,147,483,647 is referenced here, the following error message is returned:

However, this does not mean that the list itself may not have more items, because as already mentioned, lists can be infinitely large. The following definition returns a valid result:

{1..2147483647, 1..2147483647}

The syntax „..“ for a contiguous range of integers can also be applied to characters.

Lists with contiguous characters

The procedure I have just shown for consecutive numbers is also possible for letters and characters. For example, it is valid to define the following list:

{"a".."m"}

The result then looks as follows:

It may not be surprising that Power Query „knows“ that after the „a“ comes the „b“, but look at the list {"Z".."a"} and its result:

Between the letters „Z“ and „a“ there are also a number of special characters. So what is the basis of this list and it’s sort order?

Character lists are based on Unicode

The M function Character.ToNumber() reveals the secret:

Lists of characters using the „..“-syntax internally use the Unicode definition to define both the characters and their order. So the list {"Z".."a"} is internally converted into {90..97} and finally to a list of unicode characters. If you want to see the corresponding Unicode character for a special numeric value, you can use the function Charater.FromNumber(). For example, Character.FromNumber(91) results in "[".

This ordered sequence of unicode characters is the reason, why {„a“..“Z“} produces an empty list: The list is internally converted to its numeric values, which in this case means {97..90}. Since the beginning of the defined list is before the end of the list, the result is an empty list.

Defining lists manually is one way. Let’s look at how to create lists with corresponding functions.

2. Using native M functions, that create lists

At the time of writing, we have 63 functions in M that return a list. The most obvious of them are the List.* functions (but not all of them create lists). Popular examples are:

• List.Dates(),
• List.Numbers(),
• List.Random().

Also easy to identify are the *.ToList() conversion functions (which I will discuss in more detail later), such as:

• Binary.ToList(),
• Table.ToList(),
• Text.ToList().

All those functions create lists based on the input parameters. But there is another way to create a list.

3. Referring to a column/ field in a table

The third way to create a list is to reference a column of a table using the following syntax: Table[ColumnName]:

Since this may not be obvious, I will mention here that the reference to a step name (if it returns a table) has the same effect as the reference to a table expression: #"step name"[Column].

Now that we know what lists are and how they are created, we are going to focus on dealing with lists.

Operators for lists + equivalent functions

There are 3 operators that can be used in conjunction with lists: „=“ and „<>“ make it possible to compare lists, while „&“ combines lists. Here are some examples of how to use that:

1. {1,2} = {1,2} → true
2. {1,2} <> {2,1} → true
3. {1,2} & {3,4,5} → {1,2,3,4,5}. This can also be achieved by using the function List.Combine({ {1,2}, {3,4,5} })

Another aspect that is interesting with regard to lists is how to access the items in a list.

Accessing list items

Once you have a list, it is sometimes necessary to access special items within the list directly.

Using the positional index operator {}

To access an item in a list, you can use the – so called – positional index operator „{}“ by its numeric index. The items in a list are refered to using a zero-based index. The following examples explain this in detail:

Take the following list: MyList = {1,2,3}. Because the internal index of the list items starts with zero, I get the following results:

MyList{0} = 1,

MyList{1} = 2,

MyList{2} = 3,

MyList{3} leads to the following error message, saying that an attempt is made to access an item of the list that does not exist in the list:

To overcome this last error message, you can use the so called optional-item-selection as follows, which returns null, if the selected item doesn’t exist:

MyList{3}? → null

To find out how many items are in a list, you can use the function List.Count(). List.Count(MyList) leads to result 3, so the last item in the list has index 2 (List.Count(MyList)-1), because the index is zero-based.

In addition to the examples shown above, there are also various native M functions that allow access to items in a list.

Using native M functions

The M library offers a wide range of list functions, some of which provide access to list items. For the following examples take this list as given: MyList = {1,2,3,-1, 8, 9}

List.First()

This function accesses the first item in a list, which is equivalent to using the positional index operator with index 0 (like MyList{0}).

List.FirstN()

This function works in two ways.

1. If the second parameter of the function is a number, the first list items are returned up to this number: List.FirstN(MyList, 2) = {1,2}
2. If the second parameter of the funtion is a condition, all those items are returned until the condition no longer applies for the first time: List.FirstN(MyList, each _ < 3) = {1,2}

List.Last()

This function accesses the last item in a list. The following two expressions are identical: List.Last(MyList) = MyList{List.Count(MyList)-1}

List.LastN()

Works as List.FirstN, but the other way around.

List.Range()

Returns a count items starting at an offset. List.Range(MyList, 2, 3) leads to the following result: {3,-1,8}

List.Select()

This function determines items of a list, not based on their position within the list, but based on certain conditions that this item must fulfill. The following example goes through a list and selects those items whose value is greater than 2:

List.Select(MyList, each _ > 2) which returns the following list as results: {3, 8, 9}

Functions that use lists as input

Sometimes you want to use a certain functionality in M and look for the suitable function. If, for example, I want to rename the column name of a table with dynamically vaying column names, I can do this in M with the function Table.RenameColumns(). If I ignore the last optional parameter of the function, then its structure can be described as follows:

Table.RenameColumns(Tables with column names to be changed, List with the structure {old column caption, new column caption})

The following screenshot shows an M script that renames the two existing columns of a table and is prepared for the column captions of the original table to be changed during the next run. The actual renaming of the column captions takes place in the last row of the let-expression (2). All the steps described in the red box (1) are used to create the dynamic list of old name/ new name pairs required by the Table.RenameColumns() function.

Let’s next look at functions that generate lists.

Functions that create lists as output

Many functions in M generate lists and the most obvious of them are the *.ToList functions:

*.ToList functions

Binary.ToList() – Creates a list out of binaries.

Record.ToList() – Converts a record into a list, containing the field values of the record.

Table.ToList() – Creates a list from a table by separating the columns row by row with a separator that can be defined in the optional combiner parameter (e.g., Combiner.CombineTextByDelimiter(","))

Text.ToList() – Creates a list from a text by adding each character individually as a list item.

However, the *.ToList functions are not the only functions that generate a list from a value of a certain type.

Other functions, that create lists

For some of the functions, the name does not suggest that they create lists. An example of this is the function Text.Split(). Imagine the following task: In a long text count the number of distinct words. The following script does this by using Text.Split() in combination with other list functions.

Other examples of functions, that return lists, but do not sound like that are:

• Table.Column(),
• Table.ColumnNames()
• Table.ToColumns()
• Table.ToRecords()
• Table.ToRows()

Even though there is much more to say about lists and their capabilities, I guess it has certainly become clear that lists have their charm and that it is necessary to deal with them in order to master the M language.

Greetings from Germany,

Lars

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag Lists in Power Query – how, when and why erschien zuerst auf THE SELF-SERVICE-BI BLOG.