Ubuntu Hacks — Working with multiple python versions on same system

While doing any development work, switching versions for dependencies is the major issue confronted by the developers. Ubuntu comes with default python2.7 version. It does have python3.5 on it, whose console can be invoked by running “python3” as a command from terminal but this is not a default version that implies all your installations and development work will be done on top of python2.7 by default. However,obsolescence of python2 has made python3 a prerequisite for most of the development work. Also, different pieces of codes demands for different versions of python to be executed. Hence, it is a necessity to know how can you switch between versions of python, while working on Ubuntu.

There exist many ways in which you may to and fro from one version of python to another being on same system. I will elaborate few of them in this blog with their respective limitations.

1. Creating python virtual environments

This is most preferable method of making your dependencies installed on top of python version other than the default python version of ubuntu. Python virtual environments create a separate workspace on your OS, where you can install any version of any software or library without making it clash with the existing version of same software/library in root directory.

Virtual environments are very prominent:

• if you are working on some libraries which clashes with each others versions.
• if you want to work on different versions of same library for different projects.

Creating virtual environments on Ubuntu:

You need to have pip installed on your system in order to make a python virtual environment on it.

Update package index by running command:

$ sudo apt update

Install pip for Python with:

$ sudo apt install python-pip  

Installing virtualenv

$ pip install virtualenv

Test your installation:

$ virtualenv --version

Using virtualenv

You can create a virtualenv using the following command:

$ virtualenv my_name

After running this command, a directory named my_name will be created. This is the directory which contains all the necessary executables to use the packages that a Python project would need. This is where Python packages will be installed.
If you want to specify Python interpreter of your choice, for example Python 3, it can be done using the following command:

$ virtualenv -p /usr/bin/python3 virtualenv_name

Instead of python3 in above command write the version desired by you.

Now after creating virtual environment, you need to activate it. Remember to activate the relevant virtual environment every time you work on the project. This can be done using the following command:

$ source virtualenv_name/bin/activate

You can now install required packages inside the virtual environment created. In order to quit, you need to execute following command:

(virtualenv_name)$ deactivate

Limitations of using virtual environment :

• Virtual environments are not apt for building a projects which demands different versions of python within.
• Upgrading the version of python may lead to the data loss stored inside the virtual environment.
• The packages and libraries stored within virtualenv can’t be used outside it. So you need to repeat installation work in order to use those packages for some other projects. This is time and memory consuming.

2. Creating Pyenv

Pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.

pyenv does…

• Let you change the global Python version on a per-user basis.
• Provide support for per-project Python versions.
• Allow you to override the Python version with an environment variable.
• Search commands from multiple versions of Python at a time. This may be helpful to test across Python versions with tox.

PyEnv is install per user, where you can use these different ways:

• global: default for when starting a shell for the current user
• shell: updates the current session
• local: use the local project version specified in a .python-version file. Python is set at the current directory level, useful for managing python per project.

Installation:

Check out pyenv where you want it installed. A good place to choose is $HOME/.pyenv (but you can install it somewhere else).

$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

Define environment variable PYENV_ROOT to point to the path where pyenv repo is cloned and add $PYENV_ROOT/bin to your $PATH for access to the pyenv command-line utility.

$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc

Add pyenv init to your shell to enable shims and autocompletion. Please make sure eval "$(pyenv init -)" is placed toward the end of the shell configuration file since it manipulates PATH during the initialization.

$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bashrc

Restart your shell so the path changes take effect. You can now begin using pyenv.

$ exec "$SHELL"

Install Python build dependencies before attempting to install a new Python version.

$ sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Install Python versions into $(pyenv root)/versions. For example, to download and install Python 2.7.8, run:

$ pyenv install 2.7.8

Setting up versions:

pyenv local

Sets a local application-specific Python version by writing the version name to a .python-version file in the current directory. This version overrides the global version, and can be overridden itself by setting the PYENV_VERSION environment variable or with the pyenv shell command.

$ pyenv local 2.7.6

When run without a version number, pyenv local reports the currently configured local version. You can also unset the local version:

$ pyenv local --unset

Previous versions of pyenv stored local version specifications in a file named .pyenv-version. For backwards compatibility, pyenv will read a local version specified in an .pyenv-version file, but a .python-version file in the same directory will take precedence.

pyenv local (advanced)

You can specify multiple versions as local Python at once.

Let’s say if you have two versions of 2.7.6 and 3.3.3. If you prefer 2.7.6 over 3.3.3,

$ pyenv local 2.7.6 3.3.3
$ pyenv versions
  system
* 2.7.6 (set by /Users/yyuu/path/to/project/.python-version)
* 3.3.3 (set by /Users/yyuu/path/to/project/.python-version)
$ python --version
Python 2.7.6
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

or, if you prefer 3.3.3 over 2.7.6,

$ pyenv local 3.3.3 2.7.6
$ pyenv versions
  system
* 2.7.6 (set by /Users/yyuu/path/to/project/.python-version)
* 3.3.3 (set by /Users/yyuu/path/to/project/.python-version)
  venv27
$ python --version
Python 3.3.3
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

pyenv global

Sets the global version of Python to be used in all shells by writing the version name to the ~/.pyenv/version file. This version can be overridden by an application-specific .python-version file, or by setting the PYENV_VERSION environment variable.

$ pyenv global 2.7.6

The special version name system tells pyenv to use the system Python (detected by searching your $PATH).

When run without a version number, pyenv global reports the currently configured global version.

pyenv global (advanced)

You can specify multiple versions as global Python at once.

Let’s say if you have two versions of 2.7.6 and 3.3.3. If you prefer 2.7.6 over 3.3.3,

$ pyenv global 2.7.6 3.3.3
$ pyenv versions
  system
* 2.7.6 (set by /Users/yyuu/.pyenv/version)
* 3.3.3 (set by /Users/yyuu/.pyenv/version)
$ python --version
Python 2.7.6
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

or, if you prefer 3.3.3 over 2.7.6,

$ pyenv global 3.3.3 2.7.6
$ pyenv versions
  system
* 2.7.6 (set by /Users/yyuu/.pyenv/version)
* 3.3.3 (set by /Users/yyuu/.pyenv/version)
  venv27
$ python --version
Python 3.3.3
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

pyenv shell

Sets a shell-specific Python version by setting the PYENV_VERSION environment variable in your shell. This version overrides application-specific versions and the global version.

$ pyenv shell pypy-2.2.1

When run without a version number, pyenv shell reports the current value of PYENV_VERSION. You can also unset the shell version:

$ pyenv shell --unset

Note that you’ll need pyenv’s shell integration enabled (step 3 of the installation instructions) in order to use this command. If you prefer not to use shell integration, you may simply set the PYENV_VERSION variable yourself:

$ export PYENV_VERSION=pypy-2.2.1pyenv shell (advanced)

You can specify multiple versions via PYENV_VERSION at once.

Let’s say if you have two versions of 2.7.6 and 3.3.3. If you prefer 2.7.6 over 3.3.3,

$ pyenv shell 2.7.6 3.3.3
$ pyenv versions
  system
* 2.7.6 (set by PYENV_VERSION environment variable)
* 3.3.3 (set by PYENV_VERSION environment variable)
$ python --version
Python 2.7.6
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

or, if you prefer 3.3.3 over 2.7.6,

$ pyenv shell 3.3.3 2.7.6
$ pyenv versions
  system
* 2.7.6 (set by PYENV_VERSION environment variable)
* 3.3.3 (set by PYENV_VERSION environment variable)
  venv27
$ python --version
Python 3.3.3
$ python2.7 --version
Python 2.7.6
$ python3.3 --version
Python 3.3.3

Limitations of pyenv :

• Pyenv is not an apt solution for package management.
• You can not use two versions of same package for different projects. In order to do so, you are required to build a virtualenv using the plugin of pyenv on top of it.

3. Switching versions using GNOME terminal

Ubuntu comes with both Python 2.7 and Python 3.5 by default. You can install other versions of python along with them via a third-party PPA by doing following steps:

1. Open terminal via Ctrl+Alt+T or searching for “Terminal” from app launcher. When it opens, run command to add the PPA:

sudo add-apt-repository ppa:jonathonf/python-3.6

Instead of python-3.6 type the version of python required by you.

2. Then check updates and install Python 3.6 via commands:

sudo apt-get update
sudo apt-get install python3.6

Now you have three Python versions, use python command for version 2.7, python3 for version 3.5, and/or python3.6 for version 3.6.1.

3. To make python3 use the new installed python 3.6 instead of the default 3.5 release, run following 2 commands:

sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.5 1
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 2

Finally switch between the two python versions for python3 via command:

sudo update-alternatives --config python3

After selecting version 3.6:

python3 -V

NOTE: due to this bug, gnome-terminal won’t launch after step 3, a workaround is running following commands to recreate the symlink:

sudo rm /usr/bin/python3
sudo ln -s python3.5 /usr/bin/python3

Limitations of configuring through GNOME terminal:

Linux and Python run into messy situations due to the two versions. To illustrate, the request which python on a new Ubuntu server yields the response:

/usr/bin/python

The Ubuntu server runs Python 2. An admin can install Python 3 in addition and use Python 3. To run a Linux system with Python 3, always include python3 instead of python, or create an alias that points to python3 or a soft link that enables the user simply to type python.

When users install Python packages, they go into two folders, depending on whether they use the package manager pip2 or pip3:

/usr/lib/python3.4
/usr/lib/python2.7

Put the wrong version of Python into the wrong folder on the Linux box, and you can end up with an unrecoverable situation (see Figure 1).

An admin might drill down and try to install the Depends on packages in an attempt to fix the situation, but that can create a further mess (see Figure 2). Admins typically fix broken dependencies via sudo apt-get install -f, but this method is usually ineffective with Python and Linux.

To resolve Python and Linux problems, install the virtualenv Python environment isolation tool. Virtualenv creates multiple Python environments for multiple users. It works by installing Python 2 and Python 3 binaries in the target folder, which is preferably the user’s home folder. Then, it installs Python packages in (target folder)/lib. It also can share systemwide packages.

Conclusion

Every method has its pros and cons, you should choose one of above methods wisely in order to customise your system’s python version. As Ubuntu server runs on python2, so it is always recommended to not to manipulate the default versions using gnome terminal, rather make separate environments on top of default versions.

Virtual environments go handy for package management within a project and pyenv are best source to trigger the version of python as per requirements.

The best way to handle all clashes is to build virtualenv on top of pyenv using the plugins.