You can download all code samples from here. If you already have my Power Query editor for Notepad++ (or want to build it) you can open the file in Notepad++. Otherwise open it in a simple text editor and paste it into the Advanced Editor of Power Query or Power BI Desktop.

Environments in custom M functions

Writing custom functions in M can be key to solving more advanced problems in Power Query. More often than you might think, good knowledge of the environment of the function is crucial for overcoming error messages and getting correct results. Let’s work with an example.

Creating a Running Total

Calculating a running total is not an easy task in Power Query and therefore it is a very good example for writing custom M code. There are easier and way faster methods to calculate a running total than the one we have chosen here, but the following code examples serve a diadactic approach. We will develop the solution together. In the end you will see how it works and what environments have to do with the solution.

The idea

The idea of a running total is pretty simple. Take a look at the following screenshot:

The table shows revenue by customer and date. The running total calculates the revenue by customer for the current date and all previous dates. Let’s see how this approach can be implemented.

1. Attempt – stuck in row reference

The first attempt is for didactic reasons only. We add a new column and create a sum over the revenue column, by using List.Sum([Revenue]).

You might be surprised, that you don’t even get a result, because the following error message appears:

List.Sum() expects a list, but calculating this formula inside a Table.AddColumn function (what automatically happens, when you push the Custom Column button in the UI) creates a calculation by row. Because the formula List.Sum() is calculated row wise, this leads to only one single value (not a list) per calculation. This is what the error message tells us: It gets the value 70 (in row one) but expects a list instead! Let’s try another attempt.

2. Attempt – the overall sum

To overcome the issue from attempt 1, we now do not refer to the revenue of the current table, but to the revenue of the table in our second step: CorrectTypes! The List.Sum() function (which is inside the Table.AddColumn() function, that you can only see in the formula bar and the advanced editor) looks as follows:

Let’s take a look at the result.

Okay, we don’t have an error message any more. But the result doesn’t look very promising, because we get the same number in each row of the newly calculated column. This number is the sum over the complete revenue column, which is not our desired result. What we need to calculate is a sum over a subset of rows of the source table, which is different for each row in the newly calculated column. Let’s do that!

3a Attempt – aggregating over subsets of the source table

The last attempt got us closer to the desired result, but we are not looking for the value 82.145 in each row of the newly calculated column. We are looking for the sum over only a subset of rows of the source table. Calculating a sum over a subset of a table, brings us to the function Table.SelectRows(). What we want to calculate is the following:

• for each row of the source table
• create the sum over
• a subset of the source table (in our example table CorrectTypes) where
• the customer in the current row (of the current table) is equal to the customer in the source table and the date in the source table is smaller than or equal to the date in the current row (of the current table).

So we have to iterate the source table twice. For each row in the source table we calculate a sum over a subset of the same table! Translated into code it could look like this:

Take a look at the result.

Again we get 82.145 as a result for each row, but why? To find the answer let’s take a look at the complete formular of this step (copied from the advanced editor):

We have two main parts here, the red part and the yellow part. Let’s spend some more time here:

• red part: Adds a new column to the table CorrectTypes and creates a sum for each row in CorrectTypes. That’s the first iteration over the source table CorrectTypes.
• yellow part: The sum, which is created in the red part, is built on a subset of rows of the same table CorrectTypes. This subset is built by using the Table.SelectRows() function, which checks row by row, if the specific row should be included in the sum or not. This is the second iteration over the source table.

The problem with this code is, that to get correct results, the inner iteration (yellow part) needs to access variables from the outer iteration (red part). What we want to obtain is the following:

Within the Table.SelectRows() function we want to compare Customer and Date from the inner iteration (blue arrows) of the source table (table CorrectTypes) with Customer and Date from the outer iteration (red arrows).

Unfortunately this is not what the code does! The code does the following:

The code only compares Customers and Dates from within the Table.SelectRows() function, which is the inner iteration. Because [Customer]  = [Customer] and [Date] <= [Date] is true for each row within the inner iteration, the result always is 82.145.

But how can we reach the outer iteration for comparison? Let’s see the final solution and its explanation.

3b Final Attempt

The following solution is the one that works properly. Please take a special look at the green marked Qs, which we will explain later.

The result is the desired running total calculation.

Okay, we’ve finally got the right calculation, but what does this strange looking Q stuff mean?

The reason why 3b Attempt solves our problems

This kind of solution has been the reason, why I (Lars) started investigating in the Power Query Formula Language Specification, because I didn’t understand, why this strange syntax worked. This led me to the Environment topic. Imke and I thought, that it is likely, that there are more people out there, having difficulties with this syntax as well.

Take a look at the following screenshot, which shows the function from the advanced editor. For now, the only important thing to notice is, that we replaced the second each by (Q)=>.

Each is a keyword to create simple functions. Each is an abbreviation for (_) =>, in which the underscore represents (if you are in a table environment, as we are) the current row. We’re doing the following:

• We replace the underscore bei Q. Q could have been anything else (e. g. Inner).
• We add Q as a prefix to the fields we want to bind to the inner table CurrectTypes (Q[Customer], Q[Date])

The reason, why we use a parameter (Q) inside the function initialization syntax is the following:

1. Everytime, when the function is invoked (which means, that the function is evaluated and delivers a value) the environment, in which this happens, creates one same-named variable for each parameter of the function. (9.2 Invoking function, in the official Power Query language specification).
2. The function-body can reference variables that are present in the environment when the function is initialized. (9.6 Functions and environments, in the official Power Query language specification)

For our example this means the following:

1. The function (Q)=> is invoked in the environment of the inner CorrectTypes. 
2. This environment creates a variable named Q, corresponding to the parameter Q of the function.
3. In this way, we bind the expression Q[Customer] and Q[Date] to the inner table CorrectTypes, as you can see in the following screenshot

Now you may think: Okay, but why all this to bind [Customer] and [Date] to the inner table CorrecTypes, even though they were bound before? That’s a good question. The reason is, that the non prefixed versions of [Customer] and [Date] are now automatically bound the outer table CorrectTypes. This becomes clearer, when you see the alternative writing of the screenshot above (see the official documentation for more details):

When the inner each existed, both fields were bound to the inner table CorrectTypes. Now that the inner each has been replaced by (Q) =>, both non prefixed fields are automatically bound to the outer each. Because (_) => is invoked in the environment of the outer table CorrectTypes, we bind _[Customer] and _[Date] to the outer table CorrectTypes, which makes our calculation of the running total work Done!

We know this was a very long and tough ride through the environment concept in M. Thanks for staying with us. We hope that we were able to convey the importance of this concept. We also hope that we have succeeded in making you understand our present state of understanding of this concept. Just keep in mind: There is always an environment and your understanding of it could be the reason why your current M-Code doesn’t work

Quick recap

What did we see so far?

• More complex custom M code often uses nested function, evaluated in different environments
• Those functions often need to reference to variables from other environments
• This can be obtained by using (Parameter)=> instaed of each. The environment, in which the (Parameter)=> is invoked, creates one same-named variable for each parameter of the function.
• The function-body can then reference variables from the environment, in which the function was defined.
• In this way, you can control which expression is bound to which environment.

Regards from Germany,

Lars & Imke

I write my posts for you, the reader. Please take a minute to help me write my posts as well as possible. Thank you

[yasr_visitor_multiset setid=2]

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag The Environment concept in M for Power Query and Power BI Desktop, Part 4 erschien zuerst auf THE SELF-SERVICE-BI BLOG.

You can download all code samples from here. If you already have my Power Query editor for Notepad++ (or want to build it) you can open the file in Notepad++. Otherwise open it in a simple text editor and paste it into the Advanced Editor of Power Query or Power BI Desktop.

The global environment

In the last two posts (Part1 and Part2) we discussed environments of expressions in records and let expressions. In addition to these very individual environments, there is (almost) always a global environment. Take a look at the following expression and its result:

There are two reasons, why the expression Table.FromRecords({[A=1, B=2],[A=3, B=4]}) returns a table with two columns and two rows:

1. The function Table.FromRecords() (like any other native M function) is part of the global environment.
2. The expression Table.FromRecords({[A=1, B=2],[A=3, B=4]}) is evaluated with the global environment being part of the expression’s environment.

That might sound a bit strange. But you will see later, when we talk about the function Expression.Evaluate(), what happens, if there is no global environment. Then come back to this line and read it one more time

What’s part of the global environment?

Knowing, that there is a global environment might make you think what is part of it. If you want to see the components of the global environment, you can use the (so called) intrinsic variable #shared. #shared returns a record containing all names and values of all the queries (tables, constants, records, lists, native M functions, custom functions, etc.) in the current Excel/ Power BI Desktop file.

All these components are part of the global environment. This is the reason why you can reference a Power Query queries‘ result in another Power Query query. You can’t influence #shared in Power BI Desktop/ Power Query by writing code, but you can create new queries, which then automatically belong to #shared. Now, knowing what the global environment is and knowing about #shared, let’s continue with the function Expression.Evaluate(), which can define its own environment.

Expression.Evaluate()

This M function evaluates a text expression and returns the evaluated value. You can find the official documentation of this function here. Before we move on, we have to give big credits to Chris Webb for his excellent post about Expression.Evaluate(), which very much enlightened us . Take a look at this very simple example of how to use this function:

Even though „1 + 1“ is a text, the function evaluates the expression and returns the value 2. This can get pretty handy, when you for example have to create the code to be evaluated on the fly.

Expression.Evaluate() and the global environment

Now let us get back to our former expression Table.FromRecords({[A=1, B=2],[A=3, B=4]}) and evaluate this inside Expression.Evaluate():

Instead of returning the expected table, we get the error message, that the function Table.FromRecords() doesn’t exist in the current context. Replace current context by current environment and it gets a bit clearer: The required function doesn’t exist in the current environment. Expression.Evaluate() is not evaluated in the global environment and this is why it doesn’t even know the function Table.FromRecord() exists. Its environment is empty.

How can we resolve that? Let’s take a look at the syntax of Expression.Evaluate():

Expression.Evaluate(expression as text, optional environment as [...]) as any

The function has a second, optional parameter: Environment. This parameter has to be a record. But how to define the environment with a record? There is good news for you. You already know a variable, that contains all the M functions and returns a record: #shared! Using #shared as second parameter resolves the problem:

But not all environmental problems come from the global environment.

Expression.Evaluate() and non global environments

The global environment is not the only possible environmental problem, when it comes to Expression.Evaluate(). Take a look at the following script:

The error message is the same (only the error code is slightly different), as in the example before, but this time it’s not about the global environment, but about the environment of the let expression. The expression behind variable Source cannot access the variables MyVariableA and MyVariableB, because the environment within the Expression.Evaluate() function is empty. You might think, that we could resolve this problem the same way we did last time: by adding #shared?! Unfortunately this is no solution. Even if you include #shared, you get the following error message:

The variables MyVariableA and MyVariablesB are not part of #shared, as this intrinsic variable only includes the results of queries, not single variables within the queries. So, how could a solution look like? Take a look at the following script:

As you can see this script delivers the desired result: 3! But honestly, what does this [MyVariableA = MyVariableA, MyVariableB = MyVariableB] mean? This is something we learned from Chris and the next screenshot will unravel the mystery:

The definition in squared brackets pulls the outer variables MyVariableA and MyVariableB into the environment of the Expression.Evaluate() function and creates a connection between those outer variables and the variables from inside the Expression.Evaluate() function. Usually the inner and outer variables have the same name (e. g. MyVariableA = MyVariableA) what made it hard to understand expresions like [MyVariableA = MyVariableA], but now you know what this syntax is used for. Please notice, that the order plays an important role: It has to be [InnerVariable = OuterVariable]. If you try to change the order, you will get an error message.

Expression.Evaluate with combined global and non global environments

You have seen how to define the global environment inside Expression.Evaluate (by adding #shared) and how to add variables from the same query to the Expression.Evaluate function. Now let’s push it a bit further. What if you need both combined? Let’s say, you want to access outer variables and additionally you want to use a native M function, to calculate a sum of a list:

We need to get the list MyListA inside the environment of Expression.Evaluate(). Additionally we need to add the global environment (#shared) to Expression.Evaluate(), because otherwise List.Sum() does not exist. We’ve got to do that in three steps:

1. Creating a record, that does the connection between inner and outer variables
2. Combining that record with #shared (the global environment), which is a record itself
3. Adding that new record as second parameter of Expression.Evaluate()

As you can see this leads to the desired result: 3! But you can shorten the code, by creating the combined environment record inside Expression.Evaluate():

Switching between Global and non global Environments

Imagine the following scenario: You have a query, that is called MyValue, which contains the value 12.

Now you write another query, which contains a variable (step) with the same name MyValue, that has the value 1.

How can you ensure, that the Result of Query1 refers to Query MyValue and not to the step MyValue, or in other words: How can Query1 return the value 12 instead of 1? With environments:

Because the query MyValue is part of the global environment, we can use Expression.Evaluate() and use #shared as environment. That way the query MyValue is evaluated, and not the variable (step) MyValue

Expression.Evaluate in a calculated column

Let’s say you have the following table and you need to calculate the sum of both columns by using Expression.Evaluate(). This could be the case, when you try to rebuild the example from this blog post from Imke.

The upcoming error message tells us, that the identifier is unknown, or in other words, [Column1] and [Column2] are not part of the environment of the Expression.Evaluate() statement. Inside a table, the underscore „_“ represents the current row, when working with line-by-line operations. The error can be fixed, by adding [_=_] to the environment of the Expression.Evaluate() function. This adds the current row of the table, in which this formula is evaluated, to the environment of the statement, which is evaluated inside the Expression.Evaluate() function.

At the time of writing Expression.Evaluate() is one of two native M functions, which is able to define its own environment. The other one is SqlExpression.ToExpression()…

SqlExpression.ToExpression()

SqlExpression.ToExpression() is no new native M function. It exists for a long time now. Unfortunately this function has an official documentation which only covers the syntax of the function:

SqlExpression.ToExpression(sql as text, environment as record) as text

Honestly we do not know more about this function, but we wanted to name it here, because it can define its own environment. So if you know more about it, feel free to comment below this post

Quick recap

What did we see so far?

• There is a global environment, which exists (almost always) in parallel to the individual environments of each expression. This is the reason why native M functions and other Power Query queries can be accessed in expressions
• #shared is a variable, returning a record with all the components of the global environment in the current Excel/ Power BI Desktop file
• There exist two native M functions, which can define their own environment: Expression.Evaluate(Expression, optional Environment) and SqlExpression.ToExpression(Sql-Expression, optional Environment)
• Without the optional second parameter, the environment of Expression.Evaluate() is empty. That’s why, unless you define the environment correctly, this function can neither access any variable from inside the Power Query query nor use a native or custom M function.
• The second parameter creates the environment for the defined expressions within the first parameter of the function (e. g. #shared, other variables within the Power Query query, etc.). The second parameter follows the pattern [inner variable = outer variable].
• If you want, that the environment of your Expression.Evaluate() function consists of the global environment ( e. g. to use native m functions) and an outer variable, then you can do that the following way: Use Record.Combine([inner variable = outer variable], #shared) and then use this as second parameter of the Expression.Evaluate() function.
• If you are using Expression.Evaluate() in a calculated column and want to reference columns of the current table, you need to use [_=_] as environment, to pull the current row inside the environment of the Expression.Evaluate() function.

The next post will discuss custom M functions and their environments. Stay tuned

Regards from Germany,

Lars & Imke

I write my posts for you, the reader. Please take a minute and rate the following 3 categories and press „submit“, to help me write even better posts. Thank you

[yasr_visitor_multiset setid=2]

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag The Environment concept in M for Power Query and Power BI Desktop, Part 3 erschien zuerst auf THE SELF-SERVICE-BI BLOG.

You can download all code samples from here. If you already have my Power Query editor for Notepad++ (or want to build it) you can open the file in Notepad++. Otherwise open it in a simple text editor and paste it into the Advanced Editor of Power Query or Power BI Desktop.

Environments in single nested records

A nested record is a record within a record. Here you see a record (represented bei the outer square brackets), containing the two variables a and b, while a names an expression, which is a record itself.

The environment for each sub-expression is formed by all the variables belonging to its parent-expression, except itself, and all the variables within the environment of the parent-expression. Let’s make this more plastic: In this example a and b are in the same environment, which is the outer record. Records are structured data, and all variables of the same structural level are in the same environment. The variables x, y and z name the sub-expression of the parent-expression a (the record itself). Each of these variables can access the other variables within their record (as we already know from the last post), except themself. New from here is, that x, y and z can also access b, because b is part of the environment of a, which is the parent-expression of x, y and z.

You can see the result in the following screenshot:

Merging variables in nested records

Now, that you know that sub-expression can access the variables within their parent-expressions environment: What if there is the same variable in different environments. In our last post we explained, that variables can be used as an identifier, because they have to be unique within their environment. But in two different environments, the same variable can exist twice. Take a look at the following nested record:

You see, that variable x is defined twice:

• line 28: within record a (which is correctly addressed a[x]) and
• line 33: definition outside record a

This is a valid definition, as both variables exist in different environments. But what if other variables refer to x? Which x will be used? This is where the knowledge of environments gets handy again. Let’s take a look at it…

Referring to x from inside record a

When x is refered from inside record a, the inner x is used. Inner means, we are looking from the perspective of the record a. Inner and outer always depends on the perspective you have taken. We’re looking from the perspective of record a.

When x is refered in this example, the expression uses the x from inside record a (which is correctly addressed by a[x]), to calculate the value 3:

Referring to x from outside record a

In this example we’re referring to x from outside record a.

This time the expression b = x referes to the outer x. Accordingly, the result is 4:

If b should refere to x from inside record a, instead of the outer x, then the correct expression would look like this:

Referring to the outer x from x inside record a

In this example x is refered from inside record a, again. But this time it is refered from the inner x.

Remember from our first post: Variables are not part of their own environment! This is why you could expect to get an error by the expression x = x. But in this example, it calculates the value 4, instead of  throwing an error: 

How can this behavior be explained?

What technically happens in these examples

When the environments of x, y and z within record a are formed, the following happens: The environment of each variable (x, y and z) gets merged with all the variables existing in the environment of parent-expression a (which is b and x). This leads to a conflict, because when merging these environments, x exists twice, what is not possible. Remember: Variables have to be unique in their environment, because they work as identifiers for their specific expression.

This conflict is resolved in that way, that whenever x is refered, it is always used the „closest“ x:

• When we refered to x from inside record a, a[x] was used
• When we refered to x from outside record a, x was used
• When we refered to x from x inside record a (a[x] = x), another rule kicked in (read about it here), which says, that a variable is not part of its own environment, so that the outer x is the closest x.

In addition to record, also let expression create their own environment. Let’s have a look at this topic too.

Environments in let-expressions

Let expressions define one or more expressions after the let, and return a value, which is calculated by the expression after the in. Regarding environments, the only important information for you is, that the expression after the in, can access all the variables defined in the let expression, because they belong to its environment.

Quick recap

What did we see so far?

• Variables can not only access other variables within their environment, but also all variables within the environment of their parent-expression (record a)
• All these variables are merged, to form the environment of the specific variable.
• This can lead to conflicts, as variables can occur multiple times in different environments.
• This conflict is resolved by always using the variable, that is closest to the expression, that refers to it.
• Let expressions: The expressions after the in can access all the variables defined in the let expression, because they belong to its environment.

The next post will discuss native M functions, which can define their own environment. Stay tuned

Regards from Germany,

Lars & Imke

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag The Environment concept in M for Power Query and Power BI Desktop, Part 2 erschien zuerst auf THE SELF-SERVICE-BI BLOG.

You can download all code samples from here. If you already have my Power Query editor for Notepad++ (or want to build it) you can open the file in Notepad++. Otherwise open it in a simple text editor and paste it into the Advanced Editor of Power Query or Power BI Desktop.

Before we start with environments: Expressions and variables

Expressions and variables are two other concepts, which build the basis to understand environments.

Expressions

Expressions are defined as follows: An expression is a formula used to construct values. Here are some examples:

While the example 123 consists of only a single number, the expression 1 + 2 is built from so called sub-expressions. The literals 1 and 2 are sub-expressions of the parent expression 1 + 2.

The parent- / sub-expression thing is not always very obvious. When creating the record [ x = 1, y  = 2 + 3], the record itself is the parent expression, while 1 and 2 + 3 are sub-expressions of the record.

Variables

To understand variables, take a look at the definition of the following record:

Variables are named values. The variables in this record are x, y and z. They name the values, which are the result of evaluating the expression behind the equal signs. All these variables within the parent expression (which is the record itself) form the environment of the record. Within this environment each variable has to be unique. If you try to define a variables twice, within this record, the following message appears:

Because each variable is unique within a certain environment, it works as an identifier within this environment, that can me referenced. We will be discussing this later throughout this post.

Environments in M

Understanding the concept of environments is crucial, to understand calculations and the occurrence of error messages in M. As mentioned before, the record builds its own environment, consisting of all the variables in the record.

For the following record this means, that the environment of the record consists of all the variables x, y and z:

This means, that the record itself can access all its sub-expressions by using their identifiers (variables). This leads to 6 as the result of the following expression:

For the expressions within the record, the environments are a slightly different thing:

First of all take a look at line 28. Because the variables are unique within their environment, they work like identifiers, which can be referenced. This is why z = x + y can be evaluated at all. But also take a look at the comments that show information about the different environments for each variable: Each variable within the record, can access all the other variables, except itself. Ignoring this, leads to the following error message:

Because z is not part of its own environment, it can’t evaluate the expression z + y. Please read the error message precisely: The variable z is simply not existing in the environment of z = z + y.

Quick recap

What did we see so far?

• Expressions produce values. They can consist of literals (like 1), or of more complex sub-expressions (like 1 + 2).
• Variables name those expressions (like a = 1 + 2).
• Variables are unique within their environment. This is why they can be used as identifiers.
• Parent expressions build their environment from all the variables, naming their sub-expressions. Sub-expressions can be evaluated in a different environment.
• Ignoring these concepts can result in error messages, which usually are not self-explanatory.

The next post will discuss environments in nested records and let-expressions. Stay tuned

Regards from Germany,

Lars & Imke

Lars Schreiber

Lars ist Berater, Entwickler und Trainer für Microsoft Power BI. Er ist zertifizierter Power BI-Experte und Microsoft Trainer. Für sein Engagement in der internationalen Community wurde Lars seit 2017 jährlich durch Microsoft der MVP-Award verliehen. Lies hier mehr…

ssbi-blog.de

Der Beitrag The Environment concept in M for Power Query and Power BI Desktop, Part 1 erschien zuerst auf THE SELF-SERVICE-BI BLOG.