How to handle packaging in python using __init__.py

Keeping in mind the current situation across the world. I hope everyone is doing good. Please take precautions and stay at home and keep your self busy in whatever way you want to be.

I was reading the book "Python for DevOps" and came across the topic "Packaging". In every business, packaging plays a big role while it comes to product distribution. 

While it comes to IT software usually, below are the few things which should take care of :

• Descriptive Versioning 
• In Python packages, the following two variants are used:
• major.minor
• major.minor.micro
• major - for backward-incompatible changes
• minor - adds features that are also backward compatible
• micro - adds backward-compatible bug fixes.

• The Changelog
• This is a simple file that keeps track of all the changes we will be doing for each version upgrade.

Not going in detail here, coming directly to implementation on how we can handle packaging in python using the "__init__.py" file. 

The tool used here for packaging is "setuptools" python module.

Now we'll create python virtual environment and add "setuptools" there as below - 

$ python3 -m venv /tmp/packaging
$ source /tmp/packaging/bin/activate

$ pip3 install setuptools

Tip - you can cross check the list of installed modules using pip3 as below.

Now, let's see the code. I have simple hello-world examples as below

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $tree .
.
├── README
├── hello_world
│   ├── __init__.py
│   ├── hello_python.py
│   └── hello_world.py
└── setup.py

1 directory, 5 files

Note -

• README - simple instructions
• hello_world(directory) - module name
• __int__.py - organize modules while keeping them in directory
• hello_*.py - two different module with different functionality
• setup.py - required by "setuptools" to build a package.

Source code is available on GitHub page as well.

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat hello_world/hello_world.py 
def helloworld():
    return "HELLO WORLD"

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat hello_world/hello_python.py 
def hellopython():
    return "HELLO PYTHON3" 

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat hello_world/__init__.py 
from .hello_python import hellopython
from .hello_world import helloworld

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat setup.py 
from setuptools import setup, find_packages

setup(
    name="hello_example",
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    url="example.com",
    description="A hello-world example package",
    packages=find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
)

Now, lets start packaging our hello world program:

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $python3 setup.py sdist
running sdist
running egg_info
creating hello_example.egg-info
writing hello_example.egg-info/PKG-INFO
writing dependency_links to hello_example.egg-info/dependency_links.txt
writing top-level names to hello_example.egg-info/top_level.txt
writing manifest file 'hello_example.egg-info/SOURCES.txt'
reading manifest file 'hello_example.egg-info/SOURCES.txt'
writing manifest file 'hello_example.egg-info/SOURCES.txt'
running check
creating hello_example-0.0.1
creating hello_example-0.0.1/hello_example.egg-info
creating hello_example-0.0.1/hello_world
copying files to hello_example-0.0.1...
copying README -> hello_example-0.0.1
copying setup.py -> hello_example-0.0.1
copying hello_example.egg-info/PKG-INFO -> hello_example-0.0.1/hello_example.egg-info
copying hello_example.egg-info/SOURCES.txt -> hello_example-0.0.1/hello_example.egg-info
copying hello_example.egg-info/dependency_links.txt -> hello_example-0.0.1/hello_example.egg-info
copying hello_example.egg-info/top_level.txt -> hello_example-0.0.1/hello_example.egg-info
copying hello_world/__init__.py -> hello_example-0.0.1/hello_world
copying hello_world/hello_python.py -> hello_example-0.0.1/hello_world
copying hello_world/hello_world.py -> hello_example-0.0.1/hello_world
Writing hello_example-0.0.1/setup.cfg
creating dist
Creating tar archive
removing 'hello_example-0.0.1' (and everything under it)

After this, you will see that the above command has created other folders as well and our packaged module has been stored in "dist" folder.

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $tree .
.
├── README
├── dist
│   └── hello_example-0.0.1.tar.gz
├── hello_example.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── hello_world
│   ├── __init__.py
│   ├── hello_python.py
│   └── hello_world.py
└── setup.py

3 directories, 10 files

Now let's install and list down the modules using "pip3".

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $pip3 install dist/hello_example-0.0.1.tar.gz 
Processing ./dist/hello_example-0.0.1.tar.gz
Installing collected packages: hello-example
    Running setup.py install for hello-example ... done
Successfully installed hello-example-0.0.1
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $ pip3 list --format=columns
Package       Version
------------- -------
hello-example 0.0.1  
pip           20.0.2 
setuptools    41.2.0 

As the module has been installed, let's test this first using "ipyhon" console and then using the python program.

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $ipython3
/usr/local/lib/python3.7/site-packages/IPython/core/interactiveshell.py:931: UserWarning: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
  warn("Attempting to work in a virtualenv. If you encounter problems, please "
Python 3.7.7 (default, Mar 10 2020, 15:43:03) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import hello_world as hw                                                                                       

In [2]: hw.hellopython()                                                                                               
Out[2]: 'HELLO PYTHON3'

In [3]: hw.helloworld()                                                                                                
Out[3]: 'HELLO WORLD'

In [4]:

Here you saw that I can call the function using my custom module "hello_world". 

Using this module in the program-

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat example.py 
import hello_world as hw

print("Calling Hello Python Function: "+hw.hellopython())
print("Calling Hello World Function: "+hw.helloworld())
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $python3 example.py 
Calling Hello Python Function: HELLO PYTHON3
Calling Hello World Function: HELLO WORLD
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $

Now, let's suppose you want to upgrade to version "0.0.2", then we will follow the below steps -

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat hello_world/hello_python.py 
def hellopython():
    return "HELLO PYTHON3 with version **0.0.2**"
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat hello_world/hello_world.py 
def helloworld():
    return "HELLO WORLD with version ** 0.0.2 **"
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $cat setup.py 
from setuptools import setup, find_packages

setup(
    name="hello_example",
    version="0.0.2",
    author="Example Author",
    author_email="author@example.com",
    url="example.com",
    description="A hello-world example package",
    packages=find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
)

Package it again - 

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $ python3 setup.py sdist
running sdist
running egg_info
writing hello_example.egg-info/PKG-INFO
writing dependency_links to hello_example.egg-info/dependency_links.txt
writing top-level names to hello_example.egg-info/top_level.txt
reading manifest file 'hello_example.egg-info/SOURCES.txt'
writing manifest file 'hello_example.egg-info/SOURCES.txt'
running check
creating hello_example-0.0.2
creating hello_example-0.0.2/hello_example.egg-info
creating hello_example-0.0.2/hello_world
copying files to hello_example-0.0.2...
copying README -> hello_example-0.0.2
copying setup.py -> hello_example-0.0.2
copying hello_example.egg-info/PKG-INFO -> hello_example-0.0.2/hello_example.egg-info
copying hello_example.egg-info/SOURCES.txt -> hello_example-0.0.2/hello_example.egg-info
copying hello_example.egg-info/dependency_links.txt -> hello_example-0.0.2/hello_example.egg-info
copying hello_example.egg-info/top_level.txt -> hello_example-0.0.2/hello_example.egg-info
copying hello_world/__init__.py -> hello_example-0.0.2/hello_world
copying hello_world/hello_python.py -> hello_example-0.0.2/hello_world
copying hello_world/hello_world.py -> hello_example-0.0.2/hello_world
Writing hello_example-0.0.2/setup.cfg
Creating tar archive
removing 'hello_example-0.0.2' (and everything under it)

Install new version -

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $pip3 install dist/hello_example-0.0.2.tar.gz 
Processing ./dist/hello_example-0.0.2.tar.gz
Installing collected packages: hello-example
  Attempting uninstall: hello-example
    Found existing installation: hello-example 0.0.1
    Uninstalling hello-example-0.0.1:
      Successfully uninstalled hello-example-0.0.1
    Running setup.py install for hello-example ... done
Successfully installed hello-example-0.0.2

Verify the installation - 

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $ pip3 list --format=columns
Package       Version
------------- -------
hello-example 0.0.2  
pip           20.0.2 
setuptools    41.2.0 

Test again using the python program -

(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $python3 example.py 
Calling Hello Python Function: HELLO PYTHON3 with version **0.0.2**
Calling Hello World Function: HELLO WORLD with version ** 0.0.2 **
(packaging) kulsharm2@WKMIN5257929:/tmp/hello_world$ ⚙️  $