I avoided decorators for so long. First, I pretended they didn't exist. Then, I treated them like a magic spell—I'd use some of the common ones and simply copy how they're used in the documentation. And each time I tried to learn how they work, I'd give up pretty quickly.
But eventually, I got there. And once I finally understood how decorators work, my reaction was, "Is that it?" As with many complex topics, the magic goes away once you get it, and what's left makes perfect sense.
So, let me take you on a journey similar to the one I took to demystify decorators.
I split this decorator quest into seven parts. This article covers Parts 1 and 2. These are the most important parts to understand decorators. Take your time. I'll publish Parts 3 to 7 in a separate article soon.
Decorators are powerful for adding reusable functionality to functions without the need to define a new function. You’ll see them often in Python, and you may even need to write your own one at some point.
Part 1 introduces one of the main characters in this story—closures. We'll get to decorators in Part 2 later in this article.
Let's say you want to keep track of all the arguments you pass to all the calls to print() throughout your code. And no, you don't want to have to do this manually. You want your code to automatically keep a record of any argument you use when you call print().
Start by defining a new function to replace the built-in print(). Although you could name this new function print, I'll add a trailing underscore in this example to keep the code focused on what really matters. First, here's a version that won't work as intended—the code runs without errors but doesn't achieve the result you need:
The new print_() function calls the built-in print(), but it also appends the argument to a list named data. Look for the trailing underscore in this article whenever I write print_ or print!
However, data is defined within the function definition. Therefore, data is a local variable. It exists only when you call the function and while the program executes it. It cannot be accessed from anywhere else in the program. This variable is local to each function call.
The code creates a new list each time you call the function. That won't work. You need a single list that contains all the arguments you pass to all the print_() function calls.
A second option is to define the list data outside the function definition:
The list data now exists in the global scope. And a function can access names defined in the global scope. Therefore, this solution works. Here's the output from this code:
I love Python
...and also The Python Coding Stack
['I love Python', '...and also The Python Coding Stack']The list data now holds a record of all the arguments you pass to print_().
This works. However, there are some drawbacks to this solution. The list data exists in the global scope and, therefore, can be accessed from anywhere in the code. There's the danger that another part of your code tries to access and modify this list, leading to bugs and unexpected behaviour:
The list data now includes an object that's not been used as an argument in print_():
['I love Python', '...and also The Python Coding Stack', 'Oops, modified globally!']You want to eliminate this risk.
Also, you can only use this list with one function, the function print_(). If you want to keep track of arguments passed to other functions, you'll need to create a separate list with a separate name for each function.
In the first (wrong) solution above, data was local to the function. In this solution, data is global. Ideally, we need something in between these two options.
Let me show the current version of the code again, but I'll visually highlight the function's scope:
The ideal solution for this problem—the in-between solution in which data is neither local nor global—encloses the list data along with the function's local scope. Here's a visual representation of what we'd like to achieve:
If we could create such a bubble or enclosure that includes the function's local scope and data, the function would still have access to data even though it's not a local variable. But the rest of the code won't be able to access and modify data.
You can achieve such an enclosure by defining another function. Here's the first step—we'll make some additions later:
Here are the changes from the earlier code block:
You enclose the original function and the definition of the list
datawithin a new function calledprint_with_memory()You rename the function
print_asinnerto clarify that this is an inner function, nested within an outer oneFor now, you comment out the final lines since
print_(), the function with the trailing underscore, doesn't exist (yet)
The outer function, print_with_memory(), has two local names: data and inner. However, there's a more interesting observation we can make about this code:
The function inner also has access to the variable data.
A function defined within another function—an inner function—also has access to variables defined within the enclosing function. So, inner() also has access to names defined in print_with_memory(). This is called a closure.
Clear? Does it all make sense?
No, I didn't think so. So let me add a bit more. When you define a standard function—one that has no nesting, like print_() earlier—the function has access to variable names defined in the global scope. A function has access to global variables, but the global scope can't access local variables defined in functions.
A closure does the same thing but with an extra layer of nesting. The inner function, such as inner(), has access to variables defined in the scope just outside of it—the enclosing scope defined by the outer function print_with_memory(). However, the function print_with_memory() doesn't have access to the inner function's local variables. And the global scope doesn't have access to any names defined either within print_with_memory() or inner().
Let's look at all the names you created in the latest version of the code.
The outer function, print_with_memory(), has two local variables:
datainner
The name of the inner function is also a name defined within the function. Therefore, the name inner is local to the function print_with_memory(). We'll return to this point shortly.
The inner function, inner(), has one local variable, some_obj. This local variable is the parameter you define in the function's signature. Parameters are assigned values when you call the function, and therefore, they're local variables.
The inner function, inner(), also has access to the outer function's local variable, data. This is the closure concept we discussed earlier.
If you compare your current version of the code—which includes the outer function print_with_memory() and the inner function inner()—with the previous version that only had one function, print_(), you'll note that it's the inner function inner() that performs a similar task to the print_() function in the first version.
You need inner() to be accessible in the global scope. You can achieve this by returning inner from print_with_memory():
The outer function now returns the inner function. Note that you don't call inner within print_with_memory(). There are no parentheses. Instead, you return the function by using just its name without parentheses.
In the global scope, you can now call print_with_memory(). This function returns another function—the inner function. You assign this inner function to the name print_ in the global scope. This is the same name you used in the first version of this code earlier.
Therefore, you can now use print_() to print objects and also keep a record of all the arguments passed to print_(). And the function print_() is the inner function inner() returned by print_with_memory(). There's a lot of juggling of functions around in this topic!
Right, so the function print_() now has access to the list data, which is defined in print_with_memory(). Recall that this is the characteristic of a closure, which you create when you define an inner function within an outer function.
And each time you call print_(), the function will have access to the same list data. There's only one list data even when you call print_() several times. The list data is attached to the function object print_ and isn't created when you call print_(). It's already there.
Since separate calls to print_() can access the same list data, closures allow calls of a function to communicate with previous and future calls through objects within the closures, such as data in this example.
So, how can you access this list if you need to see what's inside it? You'll deal with this properly in Part 2, but here's a hard way of accessing data through print_. First, let's look at the .__closure__ attribute:
Note that I'm using the standard built-in print() in the final line rather than the new print_() version.
The output shows that the function print_ has one element in its closure:
(<cell at 0x1006e96c0: list object at 0x1006377c0>,)As mentioned earlier, the closure is attached to the function object—print_ without parentheses—and not the function call.
The output is a tuple that contains only one element—note the trailing comma, which shows that this is a tuple. You can try adding a second local variable in print_with_memory() and then referring to that second variable within inner(). You'll see that .__closure__ will contain two elements.
In our case, there's only one element in .__closure__ and, therefore, you can access it by indexing .__closure__ using the index 0:
This now gives the cell object rather than a tuple containing the cell object. Don't worry about what a cell object is—it's not relevant to our discussion:
<cell at 0x1010c16c0: list object at 0x10100f7c0>All you need to know is that you can show the value using .cell_contents:
This shows the list that contains all the arguments used in all the calls to print_():
['I love Python', '...and also The Python Coding Stack']It's hard to get to this data. But that's a good thing. You don't want the data shared by all the calls to print_() to be easily accessible.
However, you'll see that you don't usually need to access this data directly using .__closure__. But more on this later in this series.
We're at the end of Part 1 in this journey through decorators, and we haven't even mentioned decorators. Don't worry. We'll get to them soon. First, let's wrap up Part 1 with a few summary-type observations:
A closure allows a function to access variables that aren't in its local scope or in the global scope. A closure has access to the enclosing scope and, therefore, to variables defined in the enclosing (outer) function when creating the closure.
A closure permits some data to persist when you call a function. Therefore, each call of the function can "communicate" with previous and future calls of the same function.
Great. Time to talk about decorators now. Closures have other uses in programming, but I introduced them here as they're central to the discussion on decorators, which is what I'll focus on!
Let's look at the code you have so far:
The outer function print_with_memory() creates a "new version" of the built-in print() function and adds some extra functionality. It decorates the print() function—the adornment, in this case, is the ability of the main program to keep track of all the arguments passed to the new function.
This code already has many of the hallmarks of a decorator, but let's make a few changes to make it a proper decorator. The first issue you need to address is that this function only works with the built-in print(). What if you also want to apply the same treatment to other functions? You don't want to repeat yourself and define similar decorator-like functions for each case.
The first step is to change the name of the outer function since you want it to apply to any function, not just print():
This doesn't change how the code works, of course. But the new name of the outer function is more generic. Great. Let's move on. The inner function still has print() hardcoded within it. So, this code still only works for the built-in print().
Instead of hardcoding print() within the code, you can add a parameter to store_arguments() to represent any function:
There are three changes in this code compared to the previous version:
The outer function,
store_arguments(), now has a parameter calledfunc. You can call the parameter anything you like, butfuncis often used to show that you should pass a function to this parameter.The
inner()function no longer callsprint(). Instead, it now callsfunc, which is a function—the function you pass when you callstore_arguments().The call to
store_arguments()at the end of the code, when you createprint_, now needs the name of a function as an argument. It's up to you to specify which function you want to use instore_arguments()since you can now use it for any function.
In this case, you pass print to store_arguments(). Therefore, this code performs the same task as the earlier version with print_with_memory(). You can confirm that this is the case:
The two calls to print_() perform the same task as the standard print(). And the last line allows you to access the list data, which you defined when you created the closure:
I love Python
...and also The Python Coding Stack
['I love Python', '...and also The Python Coding Stack']A quick note before you move on to Step 3. You can assign the function returned by store_arguments(print) directly to the name print, the one without the trailing underscore. This overwrites the built-in print() since the name print would now refer to the inner() function returned by store_arguments(print).
You'll see in Part 3 of this journey through decorators that reusing the original function's name is the most common way of using decorators. However, I'll stick with using different names by adding trailing underscores for now.
But print_() doesn't quite work like the built-in print(). Not yet. Here's an example:
