
3.4 Higher order functions and lambda expressions
In this section, we are going to introduce a new and very powerful concept of Python (and other programming languages). This is the concept that functions can be used as parameters to other functions, similar to how other types of values like numbers, strings, or lists are used. Actually, you have already seen examples of this. In Lesson 1 with the pool.starmap(...) function and in Lesson 2 when passing the name of a function to the connect(...) method when connecting a signal to an event handler function. A function that takes other functions as arguments is often called a higher order function.
Let us immediately start with an example: Let’s say you need to apply certain string manipulation functions to each string in a list of strings. You may want to convert the string items in the list to be all upper-case characters, all lower-case, or Capital Case characters, or apply some completely different conversion. The following example shows how one can use a single function for these cases, and then pass the function to apply to each list element as a parameter to this new function:
1 2 3 4 5 6 7 8 | def applyToEachString(stringFunction, stringList): myList = [] for item in stringList: myList.append(stringFunction(item)) return myList allUpperCase = applyToEachString( str .upper, [ 'Building' , 'ROAD' , 'tree' ] ) print (allUpperCase) |
As you can see, the function definition specifies two parameters; the first one is for passing a function that takes a string and returns either a new string from it or some other value. The second parameter is for passing along a list of strings. In line 7, we call our function with using str.upper for the first parameter and a list with three words for the second parameter. The word list intentionally uses different forms of capitalization. upper() is a string method that turns the string it is called for into all upper-case characters. Since this a method and not a function, we have to use the name of the class (str) as a prefix, so “str.upper”. It is important that there are no parentheses () after upper because that would mean that the function will be called immediately and only its return value would be passed to applyToEachString(…).
In the function body, we simply create an empty list in variable myList, go through the elements of the list that is passed in parameter stringList, and then in line 4 call the function that is passed in parameter stringFunction to an element from the list. The result is appended to list myList and, at the end of the function, we return that list with the modified strings. The output you will get is the following:
1 | [ 'BUILDING' , 'ROAD' , 'TREE' ] |
If we now want to use the same function to turn everything into all lower-case characters, we just have to pass the name of the lower() function instead, like this:
1 2 | allLowerCase = applyToEachString( str .lower, [ 'Building' , 'ROAD' , 'tree' ] ) print (allLowerCase) |
Output: ['building', 'road', 'tree']
You may at this point say that this is more complicated than using a simple list comprehension that does the same function, like:
1 | [ s.upper() for s in [ 'Building' , 'ROAD' , 'tree' ] ] |
That is true in this case, but we are just creating some simple examples that are easy to understand here. One benefit of the complicated method is being able to add more string manipulations that could be hard to follow if done via list comprehension. Which method should you choose? It depends. If in doubt, choose the method that contains that is the most easily read and understood.
For converting all strings into strings that only have the first character capitalized, we first write our own function that does this for a single string. There actually is a string method called capitalize() that could be used for this, but let’s pretend it doesn’t exist to show how to use applyToEachString(…) with a self-defined function.
1 2 3 4 5 | def capitalizeFirstCharacter(s): return s[: 1 ].upper() + s[ 1 :].lower() allCapitalized = applyToEachString(capitalizeFirstCharacter, [ 'Building' , 'ROAD' , 'tree' ] ) print (allCapitalized) |
Output: ['Building', 'Road', 'Tree']
The code for capitalizeFirstCharacter(…) is rather simple. It uses slicing [:1] to make the first character of the given string s and makes it upper-case, takes the rest of the string s[1:] and turns it into lower-case. Finally, the two pieces are concatenated together again.
In a case where the function you want to use as a parameter is very simple, such as performing a single expression, and you only need this function at this one place in your code, you can skip the function definition completely and use a lambda expression. A lambda expression, (aka Anonymous function) defines a function without giving it a name using the format:
1 | lambda <parameters>: <expression for the return value> |
For capitalizeFirstCharacter(…), the corresponding lambda expression would be this:
1 | lambda s: s[: 1 ].upper() + s[ 1 :].lower() |
Note that the part after the colon does not contain a return statement; it is always just a single expression and the result from evaluating that expression automatically becomes the return value of the anonymous lambda function. That means that functions that require if-else or loops to compute the return value cannot be turned into lambda expression. When we integrate the lambda expression into our call of applyToEachString(…), the code looks like this:
1 | allCapitalized = applyToEachString( lambda s: s[: 1 ].upper() + s[ 1 :].lower(), [ 'Building' , 'ROAD' , 'tree' ] ) |
Lambda expressions can be used everywhere where the name of a function can appear, so, for instance, also within a list comprehension:
1 | [( lambda s: s[: 1 ].upper() + s[ 1 :].lower())(s) for s in [ 'Building' , 'ROAD' , 'tree' ] ] |
We here had to put the lambda expression into parenthesis and follow up with “(s)” to tell Python that the function defined in the expression should be called with the list comprehension variable s as parameter. There's a good first principles discussion on Lambda functions here(link is external) at RealPython) that is worth reviewing.
So far, we have only used applyToEachString(…) to create a new list of strings, so the functions we used as parameters always were functions that take a string as input and return a new string. However, this is not required. We can just as well use a function that returns, for instance, numbers like the number of characters in a string as provided by the Python function len(…). Before looking at the code below, think about how you would write a call of applyToEachString(…) that does that!
Here is the solution.
1 2 | wordLengths = applyToEachString( len , [ 'Building' , 'ROAD' , 'tree' ] ) print (wordLengths) |
len(…) is a function so we can simply put in its name as the first parameter. The output produced is the following list of numbers:
[8, 4, 4]
As shown in section 1.3.3, you can use Lambda with a dictionary to mimic the switch case construct when used with the .get() method:
1 2 3 4 5 6 | getTask = { 'daily' : lambda : get_daily_tasks(), 'monthly' : lambda : get_monthly_tasks(), 'weekly' : lambda : get_weekly_tasks()} task = ‘monthly’ getTask.get(task)() |
With what you have seen so far in this lesson the following code example should be easy to understand:
1 2 3 4 5 6 7 8 | def applyToEachNumber(numberFunction, numberList): l = [] for item in numberList: l.append(numberFunction(item)) return l roundedNumbers = applyToEachNumber( round , [ 12.3 , 42.8 ] ) print (roundedNumbers) |
Right, we just moved from a higher-order function that applies some other function to each element in a list of strings to one that does the same but for a list of numbers. We call this function with the round(...) function for rounding a floating point number. The output will be:
[12.0, 43.0]
If you compare the definition of the two functions applyToEachString(…) and applyToEachNumber(…), it is pretty obvious that they are exactly the same, we just slightly changed the names of the input parameters. The idea of these two functions can be generalized and then be formulated as “apply a function to each element in a list and build a list from the results of this operation” without making any assumptions about what type of values are stored in the input list. This kind of general higher-order function is already available in the Python standard library, map(…). We will go over this function along with two other popular higher-order functions: reduce(…) and filter(…).
Map
Like our more specialized versions, map(…) takes a function (or method) as the first input parameter and a list (or iterable) as the second parameter. It is the responsibility of the programmer using map(…) to make sure that the function provided as a parameter is able to work with the items stored in the provided list. In Python 3, a change to map(…) has been made so that it now returns a special map object rather than a simple list. However, whenever we need the result as a normal list, we can simply apply the list(…) function to the result like this:
1 | l = list ( map (function, iterable)) |
The three examples below show how we could have performed the conversion to upper-case and first character capitalization, and the rounding task with map(...) instead of using our own higher-order functions:
1 2 3 4 5 | map ( str .upper, [ 'Building' , 'Road' , 'Tree' ]) map ( lambda s: s[: 1 ].upper() + s[ 1 :].lower(), [ 'Building' , 'ROAD' , 'tree' ]) # uses lambda expression for only first character as upper-case map ( round , [ 12.3 , 42.8 ]) |
Map is actually more powerful than our own functions from above. It can take multiple lists as input together with a function that has the same number of input parameters as there are lists. It then applies that function to the first elements from all the lists, then to all second elements, and so on. We can use that to create a new list with the sums of corresponding elements from two lists as in the following example. The example code also demonstrates how we can use the different Python operators, like the + for addition with higher-order functions: The operator module(link is external) from the standard Python library contains function versions of all the different operators that can be used for this purpose. The one for + is available as operator.add(...).
1 2 | import operator map (operator.add, [ 1 , 3 , 4 ], [ 4 , 5 , 6 ]) |
Output: [5, 8, 10]
As a last example for map(...), let’s say you want to add a fixed number to each number in a single input list. The easiest way would be to use a lambda expression:
1 2 | number = 11 map ( lambda n: n + number, [ 1 , 3 , 4 , 7 ]) |
Output: [12, 14, 15, 18]
Filter
The goal of the filter(…) higher-order function is to create a new list with only certain items from the original list that all satisfy some criterion by applying a boolean function to each element (a function that returns either True or False) and only keeping an element if that function returns True for that element.
Below, we provide two examples for this. One for a list of strings and another for a list of numbers. The first example uses a lambda expression that uses the string method startswith(…) to check whether or not a given string starts with the character ‘R’:
1 2 | newList = filter ( lambda s: s.startswith( 'R' ), [ 'Building' , 'ROAD' , 'tree' ]) print (newList) |
Output: ['ROAD']
In the second example, we use is_integer() from the float class to take only those elements from a list of floating point numbers that are integer numbers. Since this is a method, we need to use the class name as a prefix (“float.”):
1 2 | newList = filter ( float .is_integer, [ 12.4 , 11.0 , 17.43 , 13.0 ]) print (newList) |
Output: [11.0, 13.0]
Reduce
The last higher-order function we are going to discuss here is reduce(…). In Python 3, it needs to be imported from the module functools. Its purpose is to combine (or “reduce”) all elements from a list into a single value by using an aggregation function taking two parameters that is used to combine the first and the second element, then the result with the third element, and so on until all elements from the list have been incorporated. The standard example for this is to sum up all values from a list of numbers. reduce(…) takes three parameters: (1) the aggregation function, (2) the list, and (3) an accumulator parameter. To understand this third parameter, think about how you would solve the task of summing up the numbers in a list with a for-loop. You would use a temporary variable initialized to zero and then add each number from that list to that variable which in the end would contain the final result. If you instead would want to compute the product of all numbers, you would do the same but initialize that variable to 1 and use multiplication instead of addition. The third parameter of reduce(…) is the value used to initialize this temporary variable. That should make it easy to understand the arguments used in the following two examples:
1 2 3 4 5 | import operator from functools import reduce result = reduce (operator.add, [ 234 , 3 , 3 ], 0 ) # sum print (result) |
Output: 240
1 2 3 4 5 | import operator from functools import reduce result = reduce (operator.mul, [ 234 , 3 , 3 ], 1 ) # product print (result) |
Output: 2106
Other things reduce(…) can be used for are computing the minimum or maximum value of a list of numbers or testing whether any or all values from a list of booleans are True. We will see some of these use cases in the practice exercises of this lesson. Examples of the higher-order functions discussed in this section will occasionally appear in the examples and walkthrough code of the remaining lessons.