top of page
learn_data_science.jpg

Data Scientist Program

 

Free Online Data Science Training for Complete Beginners.
 


No prior coding knowledge required!

A Deep Dive into Python Decorators



Introduction

Our primary purpose in this article is to give insights into the Python Decorators, but it is somehow an advanced topic in Python that requires background knowledge. Therefore, we will cover some essential points before talking about Decorators.


Let's first start by asking

What is a decorator?

Decorators are used to wrap around another function to extend its functionality without explicitly changing its content. In more detail, decorators are just functions that take another function as an argument and have wrapped inner function that can use arguments of the passed function. Our decorator returns this wrapper function. The inner function with free variable (what is it? we will learn soon) is also called closure that we will also talk about later.


Functions and Arguments

Functions are considered as first-class objects(citizens) in Python. We say that functions can be handled uniformly, but what do we understand from this definition? It means we can store the function in a variable, pass it as an argument to another function, use it inside that function, return the function, store the functions in different data structures. Now let's practice what we have mentioned above.


Storing a function in a variable


# function to greet people
def greet(name):
    return f'Hi, {name}'

# storing in a variable
my_var = greet
print(my_var('Kanan'))
Output: 'Hi, Kanan'

Passing a function as an argument


from math import sqrt
# we will use first 2 functions as an argument
def sq_root(num):
    return sqrt(num)

def cube (num):
    return num ** 3

# after finding the result, functions uses arguments to give meaningful answer
def meaningful_func(func, num, message):
    res = func(num)
    return f'{message} of the {num} is the {res:.2f}'

print(meaningful_func(sq_root, 20, 'square root'))
nt(meaningful_func(cube, 5 , 'cube'))


Output 1: 'square root of the 20 is the 4.47'
Output 2: 'cube of the 5 is the 125.00'

Here we passed square root and cube functions into our last function as an argument and a message that explains what the passed functions do and returns an understandable result. We will talk about the returning functions inside a function while explaining nested functions in the closures part. We could also store the functions in data structures like lists, dictionaries, hash tables, etc. We will look at a straightforward case for practice.


Storing functions in data structures


from math import sqrt

# we will create 2 lambda functions and store them into a dictionary with boolean keys
bool_list = [True, False]
func1 = lambda x: x ** 2
func2 = lambda x: x ** 3
func_list = [func1, func2]
dic = dict(zip(bool_list, func_list))

# here we will use stored functions, if key is True
num = 10
for key, value in dic.items():
    if key:
        print(value(num))


Output: 100

Args and Kwargs

So far, we have defined several arguments while creating the functions. Therefore, Python will give an error if we have defined a function with two arguments but give more than two arguments while calling the function. However, sometimes we want to pass as many arguments as possible to the function and handle all of them. But at the creation process, we do not know how many arguments will be passed in the future. To handle this situation, Python has its syntax that we use when while defining function. It is called the unpacking operator - '*.' We can use this asterisk sign in front of any argument, and then Python will put all the arguments passed after that step into the tuple with the name we have defined. Generally, we use '*args,' as it is more understandable for us since we know that we will take non-keyword arguments here. If we need to pass arguments with default values, we use a double asterisk - "**" before any argument. It is also called named arguments or keyword arguments; therefore, generally, we use '**kwags.' But again, kwargs has no meaning for Python; we use any name. What Python needs to understand is that we need named arguments in the double-asterisk sign.


Using *args


# let's first look at the function with pre-defined number of arguments
def add(a, b):
    return a + b

print(add(1, 2, 3))
TypeError: add() takes 2 positional arguments but 3 were given

As we see from the error, it says we have only two arguments defined, but we want to pass three, and Python does not know what to do with that 3rd variable. So let's have an 'add' function that can add any number of elements. In our case, the args variable will be a tuple of arguments that we will pass.



def add(*args):
    print(args)
    return (sum(args))

print(add(1, 2, 3))
Output 1: (1, 2, 3)
Output 2: 6

# we can use non-keyword arguments after some predefined arguments
def func(first_argument, *args):
    print(f'{first_argument} is a predefined argument')
    print('Non-keyword arguments:')
    for i in args:
        print(i, end = ' ')

print(func('Python', 'Java', 'Julia', 'C++'))
Python is a predefined argument
Non-keyword arguments:
Java Julia C++ 

As we see, the first passed argument is stored in the first argument variable, and the rest of the arguments we passed are stored in the args tuple.


Using **Kwargs

When we use keyword arguments, Python stores them into a dictionary where given variables become keys, and their values become dictionary values.

def func(**kwargs):
    for key, value in kwargs.items():
        print(f'{value} has been stored at {key}')
        
print(func(argument1 = 'Python', argument2 = 'Java'))
Python has been stored at argument1
Java has been stored at argument2

# args and kwargs together
def func(*args, **kwargs):
    print('Non-keyword arguments:')
    for i in args:
        print(i)
    print('Keyword arguments:')
    for key, value in kwargs.items():
        print(f'{key}:{value}')

print(func(65, 80, 90, grade4=95, grade5=100))
Non-keyword arguments:
65
80
90
Keyword arguments:
grade4:95
grade5:100

Scopes

While working with the functions, we experience that Python doesn't recognize it when creating the variable inside the function. So we can ask how Python recognizes the variable inside of the function but not outside. To answer this question, we should first look at what happens when we create a variable and have a superficial knowledge of variables, bindings, namespaces, and, lastly, scopes. Let'sdoesn't''let's''" ''''let'sLet'sot mix everything and go through them one-by-one.


Variables, Bindings, and Namespaces


# let's create a variable and assign a value to it
my_var = 15

Python creates an object when we run the code, and our variable my_var points to that object. We say that our variable is bound to that object. Whenever we create a variable, it exists at some part of our code, and that portion of our code is called a scope. So variables belong to the scopes they exist inside; therefore, Python recognizes variables of our scope. Whenever we go outside, Python does not know what that variable is. We already know that variables exist in scopes, but they also need to be stored somewhere. The place variables, together with their bindings stored, are called namespaces. Each scope has its namespace.


Global Scopes

Global scope is an entire file scope. The variable we create inside of our file (module) can be accessed anywhere in our file. But in Python, there is no user-defined global scope that works across the files. For example, if we have several files in one project, each file has its global scope, and they are not shared. There is only one exception which is Python's built-in global objects that can be used everywhere. (some built-in objects: True, False, print, list, dict, None, etc.). We can think of the scopes in this way: first, we have a built-in scope with its namespace, then inside, we have global scopes, each module we create has its scope and namespace. When we want to access the objects, i.e., call the variable name, it looks from inside to outside.


print(None)
Output: None

Here, Python first looks at the global scope for print and None, but we have not defined them, so it looks to the built-in scope and finds them there and knows that print is a function and None is an object, and we print the value of that None object.


# let's see what happens when we overwrite the built-in function
print = lambda x: f'hello, {x}'
Output: hello, Kanan

Using two arguments:

print('Python', 'Java')
TypeError: <lambda>() takes 1 positional argument but 2 were given

As we see above, when we assign something to built-in names, Python will first find them while calling. Therefore in our case, we will have a global print inside our module, which has become a lambda function with one argument, and we won't' be able to use the built-in print function in our module if we don't' delete it. So we should be aware of the built-in namespace in Python and avoid using them as variable names.


# here is the list of the all built-in names predefined in Python
print(dir(__builtins__))

Let's delete the custom print function and continue.

del print
print('Python', 'Java')
Output: Python Java


Scope of the loops

When we create a variable inside the loop, will it be global or local to that loop?


for i in range(5):
    a = 3
print(a)

Output: 3


We don't have a particular scope for the loops; whatever we create inside these loops is global. But in other programming languages, if we do this, we will get an error: "a was not declared in this scope," where a is the variable we created inside the loop. The code below is the C++ version of the code we have written above:


#include <iostream> using 
std::cout;
int main() {
    for (int i = 0 ; i < 5; i++){       
        int a =3; 
    }
    cout << a << '\n';  
}

Local Scopes

When we create a function and call it, we also create a scope that belongs to that specific function. Inside of the global scope, we have local scopes for each function we create. The variables we create inside the functions exist only in that function's scope and don't exist outside. Since we only create a function and are binding in module scope while defining it, its scope(of course, and all the contents inside like variables) will not be created until the function is called. The reason behind it is that each time we call the function, we create a new local scope because we can pass different values to the function each time we call it. Therefore it should store them in different namespaces.


# creating a local variable
def add(a, b):
    c = a + b
    return c

We just defined our add function with two arguments and assigned their sum to the variable c. These a, b and c variables are local to the add function. We haven't called the function yet, so it hasn't run. Therefore we haven't created a local scope. Still, during the compile-time, Python already recognizes these a, b, and c variables as local to that function and, during the runtime, actually creates a local space where these variables will exist.


add(2, 3)

Output: 5


print(c)
Output: NameError: name 'c' is not defined

As we see, after running the function, we go outside of the local scope, and we don't have a variable c any longer.


Using a global variable inside a function

The variables we create inside the module are considered global, and we can use it inside as a function.


a = 5 
def multiply_a(x):
    return a * x

print(multiply_a(4))

Output: 20


As we see, without any error, we can use global variables inside the local scopes. But what will happen if want to change the value of that global variable inside the local scope? Let's code and analyze.


a = 5
b = 3
def multiply_a(x):
    b = a * x
    return b

# input 1
print(multiply_a(4))

# input 2
print(b)

Output 1: 20

Output2: 3


From the code above, we can see that we have variable b in the local and global scope. Here is what happens at that cell. When we call the function, it looks for the a and x to multiply, and Python always works with the inside out principle while working with scopes. Therefore it looks for a local scope, doesn’t find it, looks at the global scope and finds there; then it looks for the x, a local variable passed as an argument and multiplied them together, and assigns it to the local variable b. When we return b, it looks at the local scope, and we have b there whose value is 20 in our case and return 20. But outside of the local scope (in global) value of b is still three since we assigned 20 to the local b, which doesn’t affect global b. So we see that even though we can use global variables inside the local scope, we cannot change their values locally, and global value stays the same. But what if we also want to change the global variable’s value inside a function? How do we do that? For this purpose, we should use the ‘global’ keyword to tell Python: ‘Okay, here is my global variable that I will use in this local scope, but I also want any changes made to this variable to stay outside of this local scope.’


Using the 'global' keyword


a = 5
def func():
    global a
    a = 20
    
func()
print(a)

Output: 20


During the compile-time, Python tries to identify the variables. Python will consider them local if we have variables with values assigned to them inside the local scope. But if we have a variable, but nothing has been assigned to it inside the local scope. Python will assume it is nonlocal, but until the runtime, Python doesn’t know what it is; it only knows this is not a local variable and will look for it in outside scopes only when the function is called. However, when we use a global keyword, we say to Python. This is a global variable during the compile-time. When we run the function, even if we execute and go outside of the local loop, the value of variable a will also be updated globally because of the global keyword.


Tricky Part!

What will happen if we first approach the variable as global inside the local scope and then use it as local? How will Python recognize the variable? Let’s see.


a = 10
def my_func():
    print(a)
    a = 3
    return a

print(my_func())
UnboundLocalError: local variable 'a' referenced before assignment

If you remember, we said that Python would consider it a nonlocal variable if we directly use the variable without assigning any value to it. Still, if we assign value to the variable, Python will consider it a local variable. During the compile time, Python first sees print(a) and thinks it will be a nonlocal variable and but during the runtime, it sees an assignment, so it should be a local variable. Still, we also want to print the local variable before assigning a value to it. Therefore Python gives as an error that: "Hey, I am gonna consider this variable as local to my_func function, and you are printing variable a before assignment, therefore here is your error message, enjoy it."


Nonlocal scopes

As we talked about above, functions are first-class objects. Therefore we can create an inner function inside some outer function and return that inner function from the outer. In this case, we will create a local scope inside another one, i.e.; we will have nested scopes. We call the outer local scope as enclosing scope for the inner local scope. When we discussed local scopes, we said that local scope has access to global scope to use its variables, but we can only change them locally, not globally, unless we use the global keyword. The same happens in nested scopes. The inner scope has access to the enclosing scope and can use variables defined there, but if we want to change the value stored there, it will happen only inside the inner local scope. We also have a particular keyword to change the outer local scope, but we will talk about it later. This enclosing scope is neither local to the inner function nor global; it is called a nonlocal scope. Let's not mix up everything and have some practice on nested scopes to understand in detail.


Accessing to the global variable from inner local scope:

a = 10
def outer():
    def inner():
        print(a)
    inner()

print(outer())

Output: 10


What does Python does: look for variable inside the inner scope — not found — > look at the enclosing scope — not found -> look at the global scope — found!!! It is ten; print it out.


Accessing to the nonlocal scope from the inner local scope:


def outer():
    a  = 10
    def inner():
        print(a)
    inner()

print(outer())

Output: 10


What does Python do: look for variable inside the inner scope — not found -> look at the enclosing scope — found!!! It is ten; print it out.


Modifying global variable from enclosing scope:


a = 10
def outer():
    global a
    a = 5
    def inner():
        print(a)
    inner()
    
print(outer())
print(a)

Output 1: 5

Output 2: 5


Since we used the global keyword, changes apply everywhere.


Modifying the global variable from inner local scope:


a = 10
def outer():
    def inner():
        global a
        a = 3
    inner()
    print(a)
    
print(outer())
print(a)

Output 1: 3

Output 2: 3


Again using the global keyword enabled us to change the value of the variable globally. So we can define a variable as global from both inner and outer local scopes. We can ask that okay, we have access to global variables from the local scope, and we can modify them with the global keyword. In the same way, we also have access to enclosing scope variables from the inner local scope in nested functions, but what will happen if we want to change their values?


Let's try to change the value of a nonlocal variable from the inner local scope:


def outer():
    a = 10
    def inner():
        a = 3
        print(a)
    inner()
    print(a)

print(outer())

Output 1: 3

Output 2: 10


As we were only able to change the value of the global variable locally, inside the local scope, without affecting its global value if we haven't used a particular keyword (global in this case), we are also only able to change the value of the nonlocal variable inside the inner local scope. The nonlocal variable's value stays the same when outside, only modified in the inner local scope. To modify the nonlocal variable, we need another particular keyword — nonlocal. Python will understand it is nonlocal, and any change made inner local scope should affect the value of the original nonlocal variable.


Modifying the nonlocal variable:


def outer():
    a = 10
    def inner():
        nonlocal a
        print(a)
        a = 3
        print(a)
        
    inner()
    print(a)

print(outer())

Output 1: 10

Output 2: 3

Output3: 3


As we see, inside the inner function, the value of the variable a is ten before the modification. Then we change it to 3, print it out, and check the value of variable an at enclosing scope, and we see that the value of nonlocal a has also changed.


Trick again


def outer():
    a = 10
    def inner():
        print(a)
        a = 3
        
    inner()
    print(a)
    
print(outer())

Output:

UnboundLocalError: local variable 'a' referenced before assignment

Similar to the trick we did before, the same happens here. First, we want to print variable a, but Python sees an assignment during the runtime and considers the variable an as local to local inner function. Still, before we assign value to this variable, we have a print statement. Python considers a variable as an inner local variable; however, we haven’t assigned anything to it during the print, so Python gave an error that you are using a variable before assigning a value.


Deep nested functions

We can also have a function inside of another one. In this case, the outer function's local scope will contain a nested local scope. Even accessing and modifying the global variables won't be a challenge since we can use a global keyword at any level of nested scope and change its value. It will also change the value of the global variable outside. But when we use nonlocal keywords, we should be cautious because when we use nonlocal at some local scope, it will refer to its enclosing scope. Still, in deeply nested function, this enclosing scope will also have its enclosing scope. If we use the same name for one variable in scopes, we should know which scope this variable will refer to. Let's start with modifying global variables and then dive into the nonlocal variables. We can also have a function inside of another one. In this case, the outer function's local scope will contain a nested local scope. Even accessing and modifying the global variables won't be a challenge since we can use a global keyword at any level of nested scope and change its value. It will also change the value of the global variable outside. But when we use a nonlocal keyword, we should be cautious because when we use a nonlocal at some local scope, it will refer to its enclosing scope. Still, in a deeply nested function, this enclosing scope will also have its enclosing scope. If we use the same name for one variable in scopes, we should know which scope this variable will refer to. Let's start with modifying global variables and then dive into the nonlocal variables.


Modifying the global variable from the outer function's scope:



a = 10
def outer():
    global a
    a = 5
    def inner1():
        def inner2():
            print(a)
        inner2()
    inner1()
print(outer())
print(a)

Output 1: 5

Output 2: 5


Modifying the global variable from the first nested scope:


a = 10
def outer():
    def inner1():
        global a
        a = 5
        def inner2():
            print(a)
        inner2()
    inner1()
    
print(outer())
print(a)

Output 1: 5

Output 2: 5


Modifying the global variable from the second nested scope:


a = 10
def outer():
    def inner1():
        def inner2():
            global a 
            a = 5
            print(a)
        inner2()
    inner1()
    
print(outer())
print(a)

Output 1: 5

Output 2: 5


As we see, we modify the original value no matter where we declare our variable is global in a deeply nested scope. But it is not that easy when it comes to nonlocal variables. The following examples will explain what I mean.

Modifying a nonlocal variable in the outer function’s scope from the first nested scope:


def outer():
    a = 10
    def inner1():
        nonlocal a
        a = 5
        def inner2():
            print(a)
        inner2()
    inner1()
    print(a)
    
print(outer())

Output 1: 5

Output 2: 5


This is a simple example that resembles what we have done before. When we say variable a is nonlocal inside inner1, it looks like its enclosing scope, i.e., outer function’s scope, and finds it there. So when we assign a new value to variable a, it also affects the original. Therefore when we go outside the inner scope, we still have a modified value at variable a.

We can do the same from the second nested scope:


def outer():
    a = 10
    def inner1():
        def inner2():
            nonlocal a
            a = 2
            print(a)
        inner2()
    inner1()
    print(a)

print(outer())

Output 1: 2

Output 2: 2


Until now, we only had one variable, and when we used the nonlocal keyword in inner1 and inner2 functions' scopes, it referred to the same object — the variable and in the outer function's scope. But if we have two variables with the same name in both outer and inner functions' scopes, then nonlocal will refer to the variable in enclosing scope, and we should be careful what we are referring to.



# first we should be aware of using nonlocal keyword
# nonlocal means Python will only look at local scopes, not global
a = 10
def outer():
    def inner1():
        def inner2():
            nonlocal a
            a = 3
            print(a)
        inner2()
    inner1()

print(outer())

Output:


SyntaxError: no binding for nonlocal 'a' found (<ipython-input-45-8a1647e220a2>, line 7)

First, Python looks at inner1 function's scope, doesn't find variable a, look at outer function's scope, doesn't find it, then we finish with local scopes, and next is the global scope, but we used nonlocal keyword. That means we say Python that looks only at local scopes; since there is no value assigned to variable an in local scopes, there is no object it is bound to. Therefore Python gives us a meaningful error.


Having the same variable in both outer and inner local scopes:


def outer():
    a = 'Python'
    def inner1():
        a = 'Java'
        def inner2():
            nonlocal a
            a = 'C++'
        print('Before inner2:', a)
        inner2()
        print('After inner2:', a)
        
    inner1()
    print('Outer a:',  a)

print(outer())

Output 1: Before inner2: Java

Output 2: After inner2: C++

Output 3: Outer a: Python


Let's explain what happens here. During the runtime, we assign 'Python' to the variable a in the outer scope, and we call the inner1 so that Python will go to that function and here we assigned 'Java' to the variable a; therefore, Python will consider this as a local variable of inner1, then we print the value of the variable a before calling inner2. As we have a local variable a=’Java' there, before calling the inner2, we print Java. Then we call inner2; inside inner2, we declare the variable an as nonlocal. But which variable is it? As we know, Python looks from inside to outside. Therefore it will first look at enclosing the scope of the inner2, which is inner1's scope that has the local variable a. Now we modify the variable a to 'C++.' It will also affect the variable's value in inner1's scope, as we were pointing to the object that various be bound to. Therefore after calling the inner2, when we print the variable a, we will point to the variable in inner1's scope that has been changed to 'C++,' and we print 'C++.' We executed the inner1 and go outside of the nested scopes. Now we are again in the outer function's scope, and we print the value of the variable a, which we had assigned 'Python' to. The modification we made in inner2 only affected the variable in inner1 and did not affect the outer scope. Therefore the value is still 'Python,' and we print it.


Using the nonlocal keyword in both nested scopes:


def outer():
    a = 'Python'
    def inner1():
        nonlocal a
        a = 'Java'
        def inner2():
            nonlocal a
            a = 'C++'
        print('Before inners:', a)
        inner2()
        print('After inner2:', a)
        
    inner1()
    print('outer:', a)
    
print(outer())

Output 1: Before inners: Java

Output 2: After inner2: C++

Output 3: outer: C++


Let's examine this example. First, we say that variable a in outer function's scope has value 'Python,' then we call inner1, and we say variable a is nonlocal, which point to the object that the variable a in outer function's scope is bound to. We change the value of the variable a to the 'Java.' So the variable a both in outer and inner1 functions' scope has value 'Java' now. Next, we print the value of the variable a that 'Java.' Then we cal the inner2, and inside it, we again say nonlocal, this will point to the object that the variable a in inner1's scope is bound to. So when we change the value of the variable a in inner2 to the 'C++,' it will also change the value of the variable in inner2's scope since we were pointing to it. But remember that in inner1's scope, the variable a was also pointing to the object that variable a, in outer function's scope, is bound to. Therefore, we also change the variable's value in the outer function's scope to the 'C++.'

What if we also had a global variable with the same name?


a = 'Python'
def outer():
    a = 'Java'
    def inner1():
        nonlocal a
        a = 'C++'
        def inner2():
            global a
            a =