The Python import system is pretty straightforward… to a point. Importing code present in the same directory you’re working in is very different from importing between multiple files present in multiple directories. Through this post, I analyse some scenarios commonly encountered when working with imports, hopefully making it easier for you to create your own packages.
The sample package created in this post can be found here.
We’ll start with a simple example and build on it throughout the post. Let’s say we have two simple Python files in a directory called
PythonImportExample/ file1.py file2.py
file1.py contains the following:.
print("This is file1.py")
import file1print("This is file2.py")
he import looks something like this:
What Happens When You Import a Python File?
When a Python file is imported, it is executed, and then added to the namespace of the file importing it.
For example, when
file2.py is executed, we get the following output:
$ cd PythonImportExample $ python file2.py This is file1.py This is file2.py
The imported file is executed right before its imported. So if we put the import statement in
file2.py below the print statement like so:
print("This is file2.py")import file1
We end up with a swapped output:
This is file2.py
This is file1.py
Of course, the entire process of importing a Python module (or even a package, which we’ll get to) is a little more complex. The search for a module goes something like this:
- built-in modules from the Python Standard Library.
sys.pathdirectories and files.
- modules and packages not part of the Standard Library
Once the module or package is found, it is executed. If it’s a module, the module is run. If its a package, the
__init__.py file of that package is run.
Then, the imported items are added to the namespace of that module, allowing you to import it and use its attributes.
There is a minor difference in the first element of
sys.path . If we launch the interpreter interactively, the first element is
'' . It represents the current directory from where the interpreter was launched.
If we were to run a script, instead of
sys.path would contain the directory of the script as its first element.
Let’s look at a bit of terminology first.
A basic Python package can contain sub-packages, modules, init files, and a
setup.py file. A basic package structure might look something like this:
The Python documentation says the following about modules:
“A module is a file containing Python definitions and statements. The file name is the module name with the suffix
Modules are objects that encourage modular code. A module can contain variables, functions and classes, and these components are part of the namespace defined by that module.
Due to this fact, naming issues are not an issue, since two different modules can have variables, functions, and classes with the same name.
A package is a hierarchical structure of modules and packages. Just like how a module defines a namespace so variables, functions, and classes can have the same names in two different modules, a package does the same for its constituent packages and modules.
Modules and packages inside the main package can be accessed via dot notation.
__init__.py is a file placed inside packages and sub-packages. Before Python 3.3, it was necessary for this file to be present in every package and sub-package, though this is no longer the case.
When a package is imported, it’s
__init__.py file, if present, is executed. This fact can be used for several things, like importing specific packages.
This file is present in the main directory where your package resides. It contains config information like required dependencies, scripts and sub-packages. You can specify meta-data about your package as well, like the name of the package, the author, a description, etc.
This file is what pip (Python’s standard package manager) uses to install your package. It’s located in the main directory of your project, along with your package code.
ProjectDir setup.py package/ .. .
sys.path is a list of paths as strings. When an interpreter sees the import statement, it looks for the module or package to be imported in the paths present in
Like we already discussed, the first element of the
sys.path list is:
''if we run the interpreter interactively
- the path to the script, if we run it.
The documentation states that
“Initialised from the environment variable
PYTHONPATH, plus an installation-dependent default.”
sys.pathdoesn’t depend on our current directory, just the path of the script we are running.
- it doesn’t change between imported modules. If a module imports another module, which in turn imports another, the
sys.pathfor the first module is where the interpreter searches for the second import statement.
Let’s consider a few scenarios that you might encounter while structuring the imports in your directory. In each scenario, we’ll start with a basic case, and improve upon it if it doesn’t give us the results we expect.
- Importing within the same sub-package
- Importing within the same package but in different sub-packages
- Importing between different levels in the project hierarchy
Importing within the same sub-package
Let’s add a sub-directory to our main directory called
subpackage1 . Note that we aren’t calling these packages and sub-packages because they are only directories at this point.
Let’s add two sub-modules
file4.py . Also,
file3.py. We end up with the following structure:
The two files look like this:
file3.py print("This is file3.py") file4.py import file3 print("This is file4.py")
print("This is file3.py")
import file3print("This is file4.py")
Now, when we run
file4.py , the interpreter looks for
file3.py . Since they are present in the same directory,
file3.py is easily found, since a module’s directory is the first entry in
Creating a package isn’t necessary here, but we’ll see how to do it later.
Importing within the same package but different sub-packages
Let’s say we added another sub-package to our project. We’ll name it
subpackage2 , and inside it, we’ll have
file6.py . Now, if one of these files imported the other, we wouldn’t have any issues executing, since that would essentially be the previous scenario.
But what if we wanted to import a file that is in a different subpackage, say,
The structure of our project is currently like so:
import subpackage1.file3print("This is file5.py")
file3.pydirectly, what would we end up with?
It’s clear it would be a
ModuleNotFoundError . The sys.path for
file5.py would contain its directory — which is
subpackage2 — and when importing,
subpackage1 would not be found, giving the error.
Putting all our code into a package would solve this issue, but is there another way?
It’s obvious when you think about what led to the import error in the first place:
sys.path . So, if we could dynamically change this, it would technically be possible to run a script importing a file from anywhere.
import sysprint(sys.path)sys.path.insert(1, "/Users/test_user/Documents/PythonImportExample/subpackage1")print(sys.path)import file3print("This is file5.py")
file5.py now works fine. The output looks something like this:
['/Users/test_user/Documents/PythonImportExample/subpackage2', ...['/Users/test_user/Documents/PythonImportExample/subpackage2', '/Users/test_user/Documents/PythonImportExample/subpackage1',....This is file3.py
This is file5.py
You can see
subpackage1 added to
Importing between different levels in the project hierarchy
Let’s have another look at our current structure.
We’ll look at two scenarios here:
- and the other way around (
file2.py so that it imports
file6.py. It now looks like this.
import file1print("This is file2.py")# added code
The output looks like this:
This is file1.py
This is file2.py
This is file6.py
The output is as expected.
file1.py is successfully imported just as before. Coming to the second import statement, the interpreter looks for
sys.path . It is found immediately, since
subpackage2 is located in the same directory as our script
file2.py , so the corresponding entry is already present in
sys.path .Pretty straightforward.
This is just the reverse of the previous case. A direct import like we did in Case 1 won’t work, since
file2.py isn’t on
file6.py . We would have to either create a package for this or modify
sys.path ourselves at runtime, just like we did in the previous scenario.
Note: This is one issue in the Python import system. You can’t import a module present in your current script’s parent directory without modifying
Analysing these Scenarios
We made a few decisions as we tried to solve the import errors we encountered. Our decisions were generally in regard to modifying
sys.path dynamically. Creating a package was also a suggestion.
Is it necessary to build a package?
Wouldn’t just creating a hierarchy of directories and modules do?
Some simple cases might work without creating a package, but more complex ones would quickly run into issues similar to what we discussed in the scenarios above.
Creating a package offers several benefits:
- better structure and organisation
- fewer issues like import errors and naming conflicts
- easier to share code.
Turning Our Project Into a Package
We’ll follow the steps below to make a very basic package.
- Move our current project into a directory which would serve as our main package.
- Add blank
__init__.pyto each package and subpackage and fix imports
- Add a
setup.pyfile. We’ll discuss this in a bit.
Modifying our project structure and adding __init__.py files
This step is straightforward. We move our
python_import_example package into another directory
PythonImportExampleProject , which would house config related info like
setup.py , etc, along with our package.
We also add
__init__.py files to each package and sub-package. Since we have a single main package (
python_import_example) and two sub-packages, we end up with three init files.
In the next step, we’ll talk about our package’s setup file. I’ve included it in the tree below to get an idea of where
setup.py exactly goes.
config related files (gitignore, LICENCE, etc.) pythonimportexample/
This would look like this in a chart.
Adding a setup file
Lets add a simple setup file. We’ll just set the name for now, though you can do a lot more.
from setuptools import find_packages, setup
find_packages function returns all packages and subpackages in our project. This is useful in that we don’t have to list them ourselves.
The Syntax of Your Import Statement
There are a few different ways you can structure your imports. Some examples:
from abc.def import xyz
from ..abc import pqr
import abc.def as mymodule
Let’s come back to our example. We’ll look at
Absolute imports let you specify the entire path of the package, module, or object you are importing.
If you wanted to import
file3.py , you could use this statement:
from PythonImportExample.subpackage2 import file5
- Absolute imports improve readability. Looking at the above statement, it’s clear that
- They work regardless of where your script is located. Even if the above import statement was put into
file1.py, which is located in the main package.
- Absolute imports get long, fast. Imagine importing a class from a nested series of subpackages four levels deep.
from package.subpackage1.subpackage2.subpackage3.... import TestClass
A relative import works relative to the script you are importing into. It uses the dot “.” notation.
In our example, import statements using a relative approach would look something like:
from ..subpackage2 import file5
- They are concise and don’t get as long as absolute imports. Since we don’t have to specify the entire path of whatever we are importing, fetching stuff between files deep within our package hierarchy is simpler.
- They won’t work when the location of the execution script changes. Since the statement is with respect to our execution script, any changes in our file’s location would break our code, since its relative position with respect to the file we want to import changes.
Notes and Resources
Note #1: Is an __init__.py file even necessary? And if yes, what do you put in it?
Regarding the first part, well yes… and no. Since Python3.3, there are two kinds of packages: regular and implicit namespace packages. The former kind require
__init__.py files, while the latter don’t.
But for pretty much every use-case, only the regular kind of packages would be required. The namespace packages would come in handy in really specific cases like when multiple packages at different locations are contributing modules to your package.
Besides, its a good idea to include init files anyway, since testing libraries like
pytest may give unexpected results.
Coming to the second part of the question, leaving your
init file empty is perfectly fine. But for more advanced cases, you could specify import statements and code there, making it easier to use your package overall.
Note #2: what does pip install -e do?
This command lets you build and install your package in
development mode. This means that if you make changes to your package, you won’t have to rebuild your package for the changes to reflect.
To use this, simply execute:
pip install -e <path to your package>
Note #3: python file.py vs python -m file.py?
A module can be run either as a script or as an imported module. When we specify the path of a module, we are running it as a script.
In our example,
if we import
file2.py and execute it as:
$ python subpackage2/file6.py
we are executing
file6.py as a script and
file2.py as an imported module. Simply put, to execute a module as a script, we would need to specify the entire path to that module. This can prove to be tricky if we wanted to execute something deep inside a package.
Instead, we could run a module using the
-m flag. If we used our example as a package,
we could run
file6.py like so:
$ python -m python_import_example.subpackage2.file6
Notice that since we are running the file as a module, we won’t specify the
.py file extension at the end.
Packages are an important part of working with Python. Whether you’re using someone else’s code or sharing your own, understanding how imports and packages work is key.
I hope this post helps you avoid import errors in the future, along
It should help you avoid those annoying import errors, and more importantly, make you a better Python developer.