The Python import system is pretty straightforward… to a point. Importing code present in the same directory you’re working in is very different from importing between multiple files present in multiple directories. Through this post, I analyse some scenarios commonly encountered when working with imports, hopefully making it easier for you to create your own packages.

The sample package created in this post can be found here.

https://github.com/allitnils/blogcode/pythonimportexample

GITHUB.COM

An Example

We’ll start with a simple example and build on it throughout the post. Let’s say we have two simple Python files in a directory called PythonImportExample.

PythonImportExample/
    file1.py
    file2.py

Let’s say file1.py contains the following:.

print("This is file1.py")

And file2.py imports file1.py.

import file1print("This is file2.py")

he import looks something like this:

What Happens When You Import a Python File?

When a Python file is imported, it is executed, and then added to the namespace of the file importing it.

For example, when file2.py is executed, we get the following output:

$ cd PythonImportExample
$ python file2.py

This is file1.py
This is file2.py

The imported file is executed right before its imported. So if we put the import statement in file2.py below the print statement like so:

print("This is file2.py")import file1

We end up with a swapped output:

This is file2.py
This is file1.py

Of course, the entire process of importing a Python module (or even a package, which we’ll get to) is a little more complex. The search for a module goes something like this:

Once the module or package is found, it is executed. If it’s a module, the module is run. If its a package, the __init__.py file of that package is run.

Then, the imported items are added to the namespace of that module, allowing you to import it and use its attributes.

There is a minor difference in the first element of sys.path . If we launch the interpreter interactively, the first element is '' . It represents the current directory from where the interpreter was launched.

If we were to run a script, instead of '' , sys.path would contain the directory of the script as its first element.

Let’s look at a bit of terminology first.

Terminology

A basic Python package can contain sub-packages, modules, init files, and a setup.py file. A basic package structure might look something like this:

Module

The Python documentation says the following about modules:

“A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py appended.”

Modules are objects that encourage modular code. A module can contain variables, functions and classes, and these components are part of the namespace defined by that module.

Due to this fact, naming issues are not an issue, since two different modules can have variables, functions, and classes with the same name.

Package

A package is a hierarchical structure of modules and packages. Just like how a module defines a namespace so variables, functions, and classes can have the same names in two different modules, a package does the same for its constituent packages and modules.

Modules and packages inside the main package can be accessed via dot notation.

__init__.py

__init__.py is a file placed inside packages and sub-packages. Before Python 3.3, it was necessary for this file to be present in every package and sub-package, though this is no longer the case.

When a package is imported, it’s __init__.py file, if present, is executed. This fact can be used for several things, like importing specific packages.

setup.py

This file is present in the main directory where your package resides. It contains config information like required dependencies, scripts and sub-packages. You can specify meta-data about your package as well, like the name of the package, the author, a description, etc.

This file is what pip (Python’s standard package manager) uses to install your package. It’s located in the main directory of your project, along with your package code.

ProjectDir
   setup.py
   package/
       ..
       .

sys.path

sys.path is a list of paths as strings. When an interpreter sees the import statement, it looks for the module or package to be imported in the paths present in sys.path .

Like we already discussed, the first element of the sys.path list is:

• '' if we run the interpreter interactively
• the path to the script, if we run it.

The documentation states that sys.path is:

“Initialised from the environment variable PYTHONPATH, plus an installation-dependent default.”

Some properties

• sys.path doesn’t depend on our current directory, just the path of the script we are running.
• it doesn’t change between imported modules. If a module imports another module, which in turn imports another, the sys.path for the first module is where the interpreter searches for the second import statement.

Import Scenarios

Let’s consider a few scenarios that you might encounter while structuring the imports in your directory. In each scenario, we’ll start with a basic case, and improve upon it if it doesn’t give us the results we expect.

We’ll discuss:

• Importing within the same sub-package
• Importing within the same package but in different sub-packages
• Importing between different levels in the project hierarchy

Importing within the same sub-package

Let’s add a sub-directory to our main directory called subpackage1 . Note that we aren’t calling these packages and sub-packages because they are only directories at this point.

Let’s add two sub-modules file3.py and file4.py . Also, file4.pyimports file3.py. We end up with the following structure:

PythonImportExample/    file1.py
    file2.py    subpackage1/
        file3.py
        file4.py

The two files look like this:

file3.py
print("This is file3.py")
file4.py
import file3
print("This is file4.py")

file3.py

print("This is file3.py")

file4.py

import file3print("This is file4.py")

Now, when we run file4.py , the interpreter looks for file3.py . Since they are present in the same directory, file3.py is easily found, since a module’s directory is the first entry in sys.path .

Creating a package isn’t necessary here, but we’ll see how to do it later.

Importing within the same package but different sub-packages

Let’s say we added another sub-package to our project. We’ll name it subpackage2 , and inside it, we’ll have file5.py and file6.py . Now, if one of these files imported the other, we wouldn’t have any issues executing, since that would essentially be the previous scenario.

But what if we wanted to import a file that is in a different subpackage, say, file5.py importing file3.py ?

The structure of our project is currently like so:

PythonImportExample/    file1.py
    file2.py    subpackage1/
        file3.py
        file4.py    subpackage2/
        file5.py
        file6.py

file5.py

import subpackage1.file3print("This is file5.py")

If file5.py imported file3.pydirectly, what would we end up with?
It’s clear it would be a ModuleNotFoundError . The sys.path for file5.py would contain its directory — which is subpackage2 — and when importing, subpackage1 would not be found, giving the error.

Putting all our code into a package would solve this issue, but is there another way?

It’s obvious when you think about what led to the import error in the first place: sys.path . So, if we could dynamically change this, it would technically be possible to run a script importing a file from anywhere.

file5.py

import sysprint(sys.path)sys.path.insert(1, "/Users/test_user/Documents/PythonImportExample/subpackage1")print(sys.path)import file3print("This is file5.py")

Running file5.py now works fine. The output looks something like this:

['/Users/test_user/Documents/PythonImportExample/subpackage2', ...['/Users/test_user/Documents/PythonImportExample/subpackage2', '/Users/test_user/Documents/PythonImportExample/subpackage1',....This is file3.py
This is file5.py

You can see subpackage1 added to sys.path above.

Importing between different levels in the project hierarchy

Let’s have another look at our current structure.

PythonImportExample/    file1.py
    file2.py    subpackage1/
        file3.py
        file4.py    subpackage2/
        file5.py
        file6.py

We’ll look at two scenarios here:

• file2.py importing file6.py
• and the other way around (file6.py importing file2.py )

Case 1:file2.py importing file6.py

Let’s modify file2.py so that it imports file6.py. It now looks like this.

file2.py

import file1print("This is file2.py")# added code
import subpackage2.file6

The output looks like this:

This is file1.py
This is file2.py
This is file6.py

The output is as expected. file1.py is successfully imported just as before. Coming to the second import statement, the interpreter looks for subpackage2 in sys.path . It is found immediately, since subpackage2 is located in the same directory as our script file2.py , so the corresponding entry is already present in sys.path .Pretty straightforward.

Case 2:file6.py importing file2.py

This is just the reverse of the previous case. A direct import like we did in Case 1 won’t work, since file2.py isn’t on sys.path for file6.py . We would have to either create a package for this or modify sys.path ourselves at runtime, just like we did in the previous scenario.

Note: This is one issue in the Python import system. You can’t import a module present in your current script’s parent directory without modifying sys.path or PYTHONPATH.

Analysing these Scenarios

We made a few decisions as we tried to solve the import errors we encountered. Our decisions were generally in regard to modifying sys.path dynamically. Creating a package was also a suggestion.

Is it necessary to build a package?

Wouldn’t just creating a hierarchy of directories and modules do?

Not exactly.

Some simple cases might work without creating a package, but more complex ones would quickly run into issues similar to what we discussed in the scenarios above.

Creating a package offers several benefits:

• better structure and organisation
• fewer issues like import errors and naming conflicts
• easier to share code.

Turning Our Project Into a Package

We’ll follow the steps below to make a very basic package.

1. Move our current project into a directory which would serve as our main package.
2. Add blank __init__.py to each package and subpackage and fix imports
3. Add a setup.py file. We’ll discuss this in a bit.

Modifying our project structure and adding __init__.py files

This step is straightforward. We move our python_import_example package into another directory PythonImportExampleProject , which would house config related info like .gitignore , LICENCE , setup.py , etc, along with our package.

We also add __init__.py files to each package and sub-package. Since we have a single main package (python_import_example) and two sub-packages, we end up with three init files.

In the next step, we’ll talk about our package’s setup file. I’ve included it in the tree below to get an idea of where setup.py exactly goes.

PythonImportExampleProject/
    setup.py
    
    config related files (gitignore, LICENCE, etc.)    pythonimportexample/
        __init__.py        file1.py
        file2.py        subpackage1/
            __init__.py            file3.py
            file4.py        subpackage2/
            __init__.py            file5.py
            file6.py

This would look like this in a chart.

Adding a setup file

Lets add a simple setup file. We’ll just set the name for now, though you can do a lot more.

from setuptools import find_packages, setup
setup(    name='pythonimportexample',
    packages=find_packages(),)

The find_packages function returns all packages and subpackages in our project. This is useful in that we don’t have to list them ourselves.

The Syntax of Your Import Statement

There are a few different ways you can structure your imports. Some examples:

• import abc.def.xyz
• from abc.def import xyz
• from ..abc import pqr
• import abc.def as mymodule

Let’s come back to our example. We’ll look at file5.py and file3.py .

PythonImportExample/    file1.py
    file2.py    subpackage1/
        file3.py
        file4.py    subpackage2/
        file5.py
        file6.py

Absolute imports

Absolute imports let you specify the entire path of the package, module, or object you are importing.

If you wanted to import file5.py into file3.py , you could use this statement:

import PythonImportExample.subpackage2.file5

or

from PythonImportExample.subpackage2 import file5

Pros

• Absolute imports improve readability. Looking at the above statement, it’s clear that file5.py resides in subpackage2 of PythonImportExample .
• They work regardless of where your script is located. Even if the above import statement was put into file1.py , which is located in the main package.

Cons

• Absolute imports get long, fast. Imagine importing a class from a nested series of subpackages four levels deep. from package.subpackage1.subpackage2.subpackage3.... import TestClass

Relative Imports

A relative import works relative to the script you are importing into. It uses the dot “.” notation.

In our example, import statements using a relative approach would look something like:

import ..subpackage2.file5

or

from ..subpackage2 import file5

Pros

• They are concise and don’t get as long as absolute imports. Since we don’t have to specify the entire path of whatever we are importing, fetching stuff between files deep within our package hierarchy is simpler.

Cons

• They won’t work when the location of the execution script changes. Since the statement is with respect to our execution script, any changes in our file’s location would break our code, since its relative position with respect to the file we want to import changes.

Notes and Resources

Note #1: Is an __init__.py file even necessary? And if yes, what do you put in it?

Regarding the first part, well yes… and no. Since Python3.3, there are two kinds of packages: regular and implicit namespace packages. The former kind require __init__.py files, while the latter don’t.

But for pretty much every use-case, only the regular kind of packages would be required. The namespace packages would come in handy in really specific cases like when multiple packages at different locations are contributing modules to your package.

Besides, its a good idea to include init files anyway, since testing libraries like pytest may give unexpected results.

Coming to the second part of the question, leaving your init file empty is perfectly fine. But for more advanced cases, you could specify import statements and code there, making it easier to use your package overall.

Note #2: what does pip install -e do?

This command lets you build and install your package in development mode. This means that if you make changes to your package, you won’t have to rebuild your package for the changes to reflect.

To use this, simply execute:

pip install -e <path to your package>

Note #3: python file.py vs python -m file.py?

A module can be run either as a script or as an imported module. When we specify the path of a module, we are running it as a script.

In our example,

PythonImportExample/    file1.py
    file2.py    subpackage1/
        file3.py
        file4.py    subpackage2/
        file5.py
        file6.py

if we import file6.py into file2.py and execute it as:

$ python subpackage2/file6.py

we are executing file6.py as a script and file2.py as an imported module. Simply put, to execute a module as a script, we would need to specify the entire path to that module. This can prove to be tricky if we wanted to execute something deep inside a package.

Instead, we could run a module using the -m flag. If we used our example as a package,

PythonImportExample/
    python_import_example/        file1.py
        file2.py        subpackage1/
            file3.py
            file4.py        subpackage2/
            file5.py
            file6.py

we could run file6.py like so:

$ python -m python_import_example.subpackage2.file6

Notice that since we are running the file as a module, we won’t specify the .py file extension at the end.

Conclusion

Packages are an important part of working with Python. Whether you’re using someone else’s code or sharing your own, understanding how imports and packages work is key.

I hope this post helps you avoid import errors in the future, along

It should help you avoid those annoying import errors, and more importantly, make you a better Python developer.