Python Modules: Overview

There are actually three different ways to define amodule in Python:

1. A module can be written in Python itself.
2. A module can be written inC and loaded dynamically at run-time, like there (regular expression) module.
3. Abuilt-in module is intrinsically contained in the interpreter, like theitertools module.

A module’s contents are accessed the same way in all three cases: with theimport statement.

Here, the focus will mostly be on modules that are written in Python. The cool thing about modules written in Python is that they are exceedingly straightforward to build. All you need to do is create a file that contains legitimate Python code and then give the file a name with a.py extension. That’s it! No special syntax is necessary.

For example, suppose you have created a file calledmod.py containing the following:

mod.py

s="If Comrade Napoleon says it, it must be right."a=[100,200,300]deffoo(arg):print(f'arg ={arg}')classFoo:pass

Several objects are defined inmod.py:

• s (a string)
• a (a list)
• foo() (a function)
• Foo (a class)

Assumingmod.py is in an appropriate location, which you will learn more about shortly, these objects can be accessed byimporting the module as follows:

>>>importmod>>>print(mod.s)If Comrade Napoleon says it, it must be right.>>>mod.a[100, 200, 300]>>>mod.foo(['quux','corge','grault'])arg = ['quux', 'corge', 'grault']>>>x=mod.Foo()>>>x<mod.Foo object at 0x03C181F0>

The Module Search Path

Continuing with the above example, let’s take a look at what happens when Python executes the statement:

importmod

When the interpreter executes the aboveimport statement, it searches formod.py in alist of directories assembled from the following sources:

• The directory from which the input script was run or thecurrent directory if the interpreter is being run interactively
• The list of directories contained in thePYTHONPATH environment variable, if it is set. (The format forPYTHONPATH is OS-dependent but should mimic thePATH environment variable.)
• An installation-dependent list of directories configured at the time Python is installed

The resulting search path is accessible in the Python variablesys.path, which is obtained from a module namedsys:

>>>importsys>>>sys.path['', 'C:\\Users\\john\\Documents\\Python\\doc', 'C:\\Python36\\Lib\\idlelib','C:\\Python36\\python36.zip', 'C:\\Python36\\DLLs', 'C:\\Python36\\lib','C:\\Python36', 'C:\\Python36\\lib\\site-packages']

Thus, to ensure your module is found, you need to do one of the following:

• Putmod.py in the directory where the input script is located or thecurrent directory, if interactive
• Modify thePYTHONPATH environment variable to contain the directory wheremod.py is located before starting the interpreter
  • Or: Putmod.py in one of the directories already contained in thePYTHONPATH variable
• Putmod.py in one of the installation-dependent directories, which you may or may not have write-access to, depending on the OS

There is actually one additional option: you can put the module file in any directory of your choice and then modifysys.path at run-time so that it contains that directory. For example, in this case, you could putmod.py in directoryC:\Users\john and then issue the following statements:

>>>sys.path.append(r'C:\Users\john')>>>sys.path['', 'C:\\Users\\john\\Documents\\Python\\doc', 'C:\\Python36\\Lib\\idlelib','C:\\Python36\\python36.zip', 'C:\\Python36\\DLLs', 'C:\\Python36\\lib','C:\\Python36', 'C:\\Python36\\lib\\site-packages', 'C:\\Users\\john']>>>importmod

Once a module has been imported, you can determine the location where it was found with the module’s__file__ attribute:

>>>importmod>>>mod.__file__'C:\\Users\\john\\mod.py'>>>importre>>>re.__file__'C:\\Python36\\lib\\re.py'

The directory portion of__file__ should be one of the directories insys.path.

Theimport Statement

Module contents are made available to the caller with theimport statement. Theimport statement takes many different forms, shown below.

import<module_name>

>>>importmod>>>mod<module 'mod' from 'C:\\Users\\john\\Documents\\Python\\doc\\mod.py'>

>>>sNameError: name 's' is not defined>>>foo('quux')NameError: name 'foo' is not defined

>>>mod.s'If Comrade Napoleon says it, it must be right.'>>>mod.foo('quux')arg = quux

import<module_name>[,<module_name>...]

from<module_name>import<name(s)>

>>>frommodimports,foo>>>s'If Comrade Napoleon says it, it must be right.'>>>foo('quux')arg = quux>>>frommodimportFoo>>>x=Foo()>>>x<mod.Foo object at 0x02E3AD50>

>>>a=['foo','bar','baz']>>>a['foo', 'bar', 'baz']>>>frommodimporta>>>a[100, 200, 300]

from<module_name>import*

>>>frommodimport*>>>s'If Comrade Napoleon says it, it must be right.'>>>a[100, 200, 300]>>>foo<function foo at 0x03B449C0>>>>Foo<class 'mod.Foo'>

from<module_name>import<name>as<alt_name>[,<name>as<alt_name>…]

>>>s='foo'>>>a=['foo','bar','baz']>>>frommodimportsasstring,aasalist>>>s'foo'>>>string'If Comrade Napoleon says it, it must be right.'>>>a['foo', 'bar', 'baz']>>>alist[100, 200, 300]

import<module_name>as<alt_name>

>>>importmodasmy_module>>>my_module.a[100, 200, 300]>>>my_module.foo('qux')arg = qux

>>>defbar():...frommodimportfoo...foo('corge')...>>>bar()arg = corge

>>>defbar():...frommodimport*...SyntaxError: import * only allowed at module level

>>>try:...# Non-existent module...importbaz...exceptImportError:...print('Module not found')...Module not found

>>>try:...# Existing module, but non-existent object...frommodimportbaz...exceptImportError:...print('Object not found in module')...Object not found in module

Thedir() Function

The built-in functiondir() returns a list of defined names in a namespace. Without arguments, it produces an alphabetically sorted list of names in the currentlocal symbol table:

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>qux=[1,2,3,4,5]>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'qux']>>>classBar():...pass...>>>x=Bar()>>>dir()['Bar', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'qux', 'x']

Note how the first call todir() above lists several names that are automatically defined and already in the namespace when the interpreter starts. As new names are defined (qux,Bar,x), they appear on subsequent invocations ofdir().

This can be useful for identifying what exactly has been added to the namespace by an import statement:

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>importmod>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'mod']>>>mod.s'If Comrade Napoleon says it, it must be right.'>>>mod.foo([1,2,3])arg = [1, 2, 3]>>>frommodimporta,Foo>>>dir()['Foo', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'a', 'mod']>>>a[100, 200, 300]>>>x=Foo()>>>x<mod.Foo object at 0x002EAD50>>>>frommodimportsasstring>>>dir()['Foo', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'a', 'mod', 'string', 'x']>>>string'If Comrade Napoleon says it, it must be right.'

When given an argument that is the name of a module,dir() lists the names defined in the module:

>>>importmod>>>dir(mod)['Foo', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__','__name__', '__package__', '__spec__', 'a', 'foo', 's']

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>frommodimport*>>>dir()['Foo', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'a', 'foo', 's']

Executing a Module as a Script

Any.py file that contains amodule is essentially also a Pythonscript, and there isn’t any reason it can’t be executed like one.

Here again ismod.py as it was defined above:

mod.py

s="If Comrade Napoleon says it, it must be right."a=[100,200,300]deffoo(arg):print(f'arg ={arg}')classFoo:pass

This can be run as a script:

C:\Users\john\Documents>python mod.pyC:\Users\john\Documents>

There are no errors, so it apparently worked. Granted, it’s not very interesting. As it is written, it onlydefines objects. It doesn’tdo anything with them, and it doesn’t generate any output.

Let’s modify the above Python module so it does generate some output when run as a script:

mod.py

s="If Comrade Napoleon says it, it must be right."a=[100,200,300]deffoo(arg):print(f'arg ={arg}')classFoo:passprint(s)print(a)foo('quux')x=Foo()print(x)

Now it should be a little more interesting:

C:\Users\john\Documents>python mod.pyIf Comrade Napoleon says it, it must be right.[100, 200, 300]arg = quux<__main__.Foo object at 0x02F101D0>

Unfortunately, now it also generates output when imported as a module:

>>>importmodIf Comrade Napoleon says it, it must be right.[100, 200, 300]arg = quux<mod.Foo object at 0x0169AD50>

This is probably not what you want. It isn’t usual for a module to generate output when it is imported.

Wouldn’t it be nice if you could distinguish between when the file is loaded as a module and when it is run as a standalone script?

Ask and ye shall receive.

When a.py file is imported as a module, Python sets the specialdunder variable__name__ to the name of the module. However, if a file is run as a standalone script,__name__ is (creatively) set to the string'__main__'. Using this fact, you can discern which is the case at run-time and alter behavior accordingly:

mod.py

s="If Comrade Napoleon says it, it must be right."a=[100,200,300]deffoo(arg):print(f'arg ={arg}')classFoo:passif(__name__=='__main__'):print('Executing as standalone script')print(s)print(a)foo('quux')x=Foo()print(x)

Now, if you run as a script, you get output:

C:\Users\john\Documents>python mod.pyExecuting as standalone scriptIf Comrade Napoleon says it, it must be right.[100, 200, 300]arg = quux<__main__.Foo object at 0x03450690>

But if you import as a module, you don’t:

>>>importmod>>>mod.foo('grault')arg = grault

Modules are often designed with the capability to run as a standalone script for purposes of testing the functionality that is contained within the module. This is referred to asunit testing. For example, suppose you have created a modulefact.py containing afactorial function, as follows:

fact.py

deffact(n):return1ifn==1elsen*fact(n-1)if(__name__=='__main__'):importsysiflen(sys.argv)>1:print(fact(int(sys.argv[1])))

The file can be treated as a module, and thefact() function imported:

>>>fromfactimportfact>>>fact(6)720

But it can also be run as a standalone by passing an integer argument on the command-line for testing:

C:\Users\john\Documents>python fact.py 6720

Reloading a Module

For reasons of efficiency, a module is only loaded once per interpreter session. That is fine for function and class definitions, which typically make up the bulk of a module’s contents. But a module can contain executable statements as well, usually for initialization. Be aware that these statements will only be executed thefirst time a module is imported.

Consider the following filemod.py:

mod.py

a=[100,200,300]print('a =',a)

>>>importmoda = [100, 200, 300]>>>importmod>>>importmod>>>mod.a[100, 200, 300]

Theprint() statement is not executed on subsequent imports. (For that matter, neither is the assignment statement, but as the final display of the value ofmod.a shows, that doesn’t matter. Once the assignment is made, it sticks.)

If you make a change to a module and need to reload it, you need to either restart the interpreter or use a function calledreload() from moduleimportlib:

>>>importmoda = [100, 200, 300]>>>importmod>>>importimportlib>>>importlib.reload(mod)a = [100, 200, 300]<module 'mod' from 'C:\\Users\\john\\Documents\\Python\\doc\\mod.py'>

Python Packages

Suppose you have developed a very large application that includes many modules. As the number of modules grows, it becomes difficult to keep track of them all if they are dumped into one location. This is particularly so if they have similar names or functionality. You might wish for a means of grouping and organizing them.

Packages allow for a hierarchical structuring of the module namespace usingdot notation. In the same way thatmodules help avoid collisions between global variable names,packages help avoid collisions between module names.

Creating apackage is quite straightforward, since it makes use of the operating system’s inherent hierarchical file structure. Consider the following arrangement:

Here, there is a directory namedpkg that contains two modules,mod1.py andmod2.py. The contents of the modules are:

mod1.py

deffoo():print('[mod1] foo()')classFoo:pass

mod2.py

defbar():print('[mod2] bar()')classBar:pass

Given this structure, if thepkg directory resides in a location where it can be found (in one of the directories contained insys.path), you can refer to the twomodules withdot notation (pkg.mod1,pkg.mod2) and import them with the syntax you are already familiar with:

import<module_name>[,<module_name>...]

>>>importpkg.mod1,pkg.mod2>>>pkg.mod1.foo()[mod1] foo()>>>x=pkg.mod2.Bar()>>>x<pkg.mod2.Bar object at 0x033F7290>

from<module_name>import<name(s)>

>>>frompkg.mod1importfoo>>>foo()[mod1] foo()

from<module_name>import<name>as<alt_name>

>>>frompkg.mod2importBarasQux>>>x=Qux()>>>x<pkg.mod2.Bar object at 0x036DFFD0>

You can import modules with these statements as well:

from<package_name>import<modules_name>[,<module_name>...]from<package_name>import<module_name>as<alt_name>

>>>frompkgimportmod1>>>mod1.foo()[mod1] foo()>>>frompkgimportmod2asquux>>>quux.bar()[mod2] bar()

You can technically import the package as well:

>>>importpkg>>>pkg<module 'pkg' (namespace)>

But this is of little avail. Though this is, strictly speaking, a syntactically correct Python statement, it doesn’t do much of anything useful. In particular, itdoes not place any of the modules inpkg into the local namespace:

>>>pkg.mod1Traceback (most recent call last):  File"<pyshell#34>", line1, in<module>pkg.mod1AttributeError:module 'pkg' has no attribute 'mod1'>>>pkg.mod1.foo()Traceback (most recent call last):  File"<pyshell#35>", line1, in<module>pkg.mod1.foo()AttributeError:module 'pkg' has no attribute 'mod1'>>>pkg.mod2.Bar()Traceback (most recent call last):  File"<pyshell#36>", line1, in<module>pkg.mod2.Bar()AttributeError:module 'pkg' has no attribute 'mod2'

To actually import the modules or their contents, you need to use one of the forms shown above.

Package Initialization

If a file named__init__.py is present in a package directory, it is invoked when the package or a module in the package is imported. This can be used for execution of package initialization code, such as initialization of package-level data.

For example, consider the following__init__.py file:

__init__.py

print(f'Invoking __init__.py for{__name__}')A=['quux','corge','grault']

Let’s add this file to thepkg directory from the above example:

Now when the package is imported, the global listA is initialized:

>>>importpkgInvoking __init__.py for pkg>>>pkg.A['quux', 'corge', 'grault']

Amodule in the package can access the global variable by importing it in turn:

mod1.py

deffoo():frompkgimportAprint('[mod1] foo() / A = ',A)classFoo:pass

>>>frompkgimportmod1Invoking __init__.py for pkg>>>mod1.foo()[mod1] foo() / A =  ['quux', 'corge', 'grault']

__init__.py can also be used to effect automatic importing of modules from a package. For example, earlier you saw that the statementimport pkg only places the namepkg in the caller’s local symbol table and doesn’t import any modules. But if__init__.py in thepkg directory contains the following:

__init__.py

print(f'Invoking __init__.py for{__name__}')importpkg.mod1,pkg.mod2

then when you executeimport pkg, modulesmod1 andmod2 are imported automatically:

>>>importpkgInvoking __init__.py for pkg>>>pkg.mod1.foo()[mod1] foo()>>>pkg.mod2.bar()[mod2] bar()

Importing* From a Package

For the purposes of the following discussion, the previously defined package is expanded to contain some additional modules:

There are now four modules defined in thepkg directory. Their contents are as shown below:

mod1.py

deffoo():print('[mod1] foo()')classFoo:pass

mod2.py

defbar():print('[mod2] bar()')classBar:pass

mod3.py

defbaz():print('[mod3] baz()')classBaz:pass

mod4.py

defqux():print('[mod4] qux()')classQux:pass

(Imaginative, aren’t they?)

You have already seen that whenimport * is used for amodule,all objects from the module are imported into the local symbol table, except those whose names begin with an underscore, as always:

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>frompkg.mod3import*>>>dir()['Baz', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'baz']>>>baz()[mod3] baz()>>>Baz<class 'pkg.mod3.Baz'>

The analogous statement for apackage is this:

from<package_name>import*

What does that do?

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>frompkgimport*>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']

Hmph. Not much. You might have expected (assuming you had any expectations at all) that Python would dive down into the package directory, find all the modules it could, and import them all. But as you can see, by default that is not what happens.

Instead, Python follows this convention: if the__init__.py file in thepackage directory contains alist named__all__, it is taken to be a list of modules that should be imported when the statementfrom <package_name> import * is encountered.

For the present example, suppose you create an__init__.py in thepkg directory like this:

pkg/__init__.py

__all__=['mod1','mod2','mod3','mod4']

Nowfrom pkg import * imports all four modules:

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>frompkgimport*>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'mod1', 'mod2', 'mod3', 'mod4']>>>mod2.bar()[mod2] bar()>>>mod4.Qux<class 'pkg.mod4.Qux'>

Usingimport * still isn’t considered terrific form, any more forpackages than formodules. But this facility at least gives the creator of the package some control over what happens whenimport * is specified. (In fact, it provides the capability to disallow it entirely, simply by declining to define__all__ at all. As you have seen, the default behavior for packages is to import nothing.)

By the way,__all__ can be defined in amodule as well and serves the same purpose: to control what is imported withimport *. For example, modifymod1.py as follows:

pkg/mod1.py

__all__=['foo']deffoo():print('[mod1] foo()')classFoo:pass

Now animport * statement frompkg.mod1 will only import what is contained in__all__:

>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__']>>>frompkg.mod1import*>>>dir()['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__','__package__', '__spec__', 'foo']>>>foo()[mod1] foo()>>>FooTraceback (most recent call last):  File"<pyshell#37>", line1, in<module>FooNameError:name 'Foo' is not defined

foo() (the function) is now defined in the local namespace, butFoo (the class) is not, because the latter is not in__all__.

In summary,__all__ is used by bothpackages andmodules to control what is imported whenimport * is specified. Butthe default behavior differs:

• For a package, when__all__ is not defined,import * does not import anything.
• For a module, when__all__ is not defined,import * imports everything (except—you guessed it—names starting with an underscore).

Subpackages

Packages can contain nestedsubpackages to arbitrary depth. For example, let’s make one more modification to the examplepackage directory as follows:

The four modules (mod1.py,mod2.py,mod3.py andmod4.py) are defined as previously. But now, instead of being lumped together into thepkg directory, they are split out into twosubpackage directories,sub_pkg1 andsub_pkg2.

Importing still works the same as shown previously. Syntax is similar, but additionaldot notation is used to separatepackage name fromsubpackage name:

>>>importpkg.sub_pkg1.mod1>>>pkg.sub_pkg1.mod1.foo()[mod1] foo()>>>frompkg.sub_pkg1importmod2>>>mod2.bar()[mod2] bar()>>>frompkg.sub_pkg2.mod3importbaz>>>baz()[mod3] baz()>>>frompkg.sub_pkg2.mod4importquxasgrault>>>grault()[mod4] qux()

In addition, a module in onesubpackage can reference objects in asibling subpackage (in the event that the sibling contains some functionality that you need). For example, suppose you want to import and execute functionfoo() (defined in modulemod1) from within modulemod3. You can either use anabsolute import:

pkg/sub__pkg2/mod3.py

defbaz():print('[mod3] baz()')classBaz:passfrompkg.sub_pkg1.mod1importfoofoo()

>>>frompkg.sub_pkg2importmod3[mod1] foo()>>>mod3.foo()[mod1] foo()

Or you can use arelative import, where.. refers to the package one level up. From withinmod3.py, which is in subpackagesub_pkg2,

• .. evaluates to the parent package (pkg), and
• ..sub_pkg1 evaluates to subpackagesub_pkg1 of the parent package.

pkg/sub__pkg2/mod3.py

defbaz():print('[mod3] baz()')classBaz:passfrom..importsub_pkg1print(sub_pkg1)from..sub_pkg1.mod1importfoofoo()

>>>frompkg.sub_pkg2importmod3<module 'pkg.sub_pkg1' (namespace)>[mod1] foo()

Conclusion

In this tutorial, you covered the following topics:

• How to create a Pythonmodule
• Locations where the Python interpreter searches for a module
• How to obtain access to the objects defined in a module with theimport statement
• How to create a module that is executable as a standalone script
• How to organize modules intopackages andsubpackages
• How to control package initialization

This will hopefully allow you to better understand how to gain access to the functionality available in the many third-party and built-in modules available in Python.

Additionally, if you are developing your own application, creating your ownmodules andpackages will help you organize and modularize your code, which makes coding, maintenance, and debugging easier.

If you want to learn more, check out the following documentation atPython.org:

• Theimport system
• The Python tutorial: Modules

Happy Pythoning!

Take the Quiz: Test your knowledge with our interactive “Python Modules and Packages” quiz. You’ll receive a score upon completion to help you track your learning progress:

---

Interactive Quiz

Python Modules and Packages

In this quiz, you'll test your understanding of Python modules and packages, which are mechanisms that facilitate modular programming. Modular programming involves breaking a large programming task into smaller, more manageable subtasks or modules. This approach has several advantages, including simplicity, maintainability, and reusability.