Summary

Most people’s computers have multiple Python versions installed (macOS comes with Python pre-installed, often termed system Python). Keeping these versions (the pre-installed system Python version and other user-installed Python versions) separate and controlling them independently (e.g. installing or updating a given Python package for a specific Python version only) requires an upfront solution.

Working with virtual environments, i.e. having a separate workspace for each individual project, necessitates actively defining the Python version that will be used to create a specific virtual environment. Controlling Python versions is thus linked to the task of managing virtual environments. All packages installed in a specific virtual environment will not conflict with other virtual environments (all packages being contained in that particular workspace).

Following the steps outlined here below, your computer should be in a state where different Python versions coexist peacefully and where you can safely install, upgrade, or remove libraries without affecting other projects. In fact, a working setup that is simple to maintain.

Introduction

A certainty, especially in the world of programming, is that there are numerous paths to the same destination. As a result, there is a plethora of advice available on how to arrive at a working (and hopefully maintainable) setup of a local computer to do data science using Python.

Speaking from personal experience, I was always too busy doing the actual work to contemplate about the “ideal” setup, so I decided for the moment to ignore any best practices that might exist somewhere. Besides, given the vast amount of information available, determining the right blueprint to follow is not an easy task (just considering the combinatorial nightmare of different operating systems, python versions, installed or assumed pre-installed software, various IDEs etc etc). In short, there are many flying pieces and thus many degrees of freedom, which inevitably leads to complexity.

However, sooner or later the day always comes when you can’t afford to ignore the Python setup issue any longer simply because something broke (e.g. after installing a new Python package or upgrading an installed one). From that moment on you will be busy finding a way to continue your work. Sometimes a quick fix is possible (after consulting numerous entries at Stack Overflow or similar websites), but sometimes band-aid is not enough, so that you need a fundamental solution.

In order to spare you time, research effort and hopefully some nerves I will concisely summarize all steps that resulted into my tested setup. Each step builds on different sources of advice that I will cite along the way (i.e. credits, attribution and thanks are given by explicit references).

Note

I used the phrase “fundamental solution” above for a reason: in case you got totally lost with quick fixes that resulted in even more problems later on a fresh installation of the operating system is certainly a radical approach. However, it provides a clean start from where you can begin building your new robust setup that is maintainable for all your future projects. I wrote the following lines with a fresh installation of the operating system in mind.

I wrote this summary using macOS (version: 12.6), but you should be able to follow equivalent steps on Linux or Windows (you might need to make some modifications).

Overview

We’ll go through three main steps. The list below is intended to provide you with a high-level overview. Each of the three main steps will be explained in detail in the next sections (see Details):

Install the homebrew package manager (macOS/Linux)
(If you are not sure if you have please check and eventually update it: see details below)
Install pyenv (using homebrew)
1. Running the installation command in the terminal
2. Configure your terminal (setup your .zsrh file for pyenv)
3. Using pyenv to install a new Python version (in addition to your system Python version)
4. Switching between different Python versions available/installed on your computer

About: pyenv

The library pyenv is a wonderful tool that allows you to control and switch between all different Python versions installed on your computer. Each new Python version you install using pyenv will be neatly isolated from all other versions; this is equally and importantly true for the system Python version that comes pre-installed on macOS (located in /usr/bin/python3). In this way you can also make sure you don’t mess with this system Python version that is a constituting part of your operating system. That is, your computer depends on it. A great summary about pyenv can be found here [*].

Install virtualenvwrapper using your system Python version
(and its associated pip version)
1. Running the installation command in the terminal
2. Configure your terminal to load the virtualenvwrapper commands
3. Setup a new virtual environment on the basis of a specific Python version

About: virtualenvwrapper

The library virtualenvwrapper is extremely useful tool to create and manage (such as activating/deactivating) virtual environments? Why should we care about virtual environments? The main reason is that virtual environments represent independent, totally separate workspaces for individual projects: any package installed in a given workspace will not interfere with the ones installed in other workspaces. Thus, in this way you are able to install different versions of the same package for each of your projects. This would be impossible when all of your projects’ requirements (i.e. packages) were installed in the very same environment. Please find a fine and elaborate summary about this tool here [*].

Details

1. Install the homebrew package manager

As a first step you should install homebrew which is a very useful and straightforward package manager (for macOS/Linux). In order to enter the code below open a terminal window (you can do this by opening macOS spotlight - or via your keyboard: just hit both keys command + space at once - then type “terminal” and open this application).

Now that you have your terminal window open, you can enter the following commands shown below (select the commands matching your case):

To install homebrew on your computer: paste this command in your terminal window and hit enter. After hitting enter the script will explain what it will do and then pauses before it does it:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)

To verify and check whether you have homebrew already installed, you can enter the command shown below. In the case you have homebrew installed, the output will tell you which version it is. If you don’t have it installed yet, the output will tell you something like command not found.

brew --version

Example output in case your computer has homebrew already installed (your version might differ):

Homebrew 3.6.6 
Homebrew/homebrew-core (git revision 7ec3c2d24e8; last commit 2022-10-24) 
Homebrew/homebrew-cask (git revision c3048a8013; last commit 2022-10-24)

After having verified that your computer has homebrew installed, update it to make sure you have the latest version. Just enter the following command in your terminal window and hit enter [*].

brew update

2. Install pyenv (via homebrew)

2a: Run the following installation command in the terminal [*].

brew install pyenv

2b: Now you need to configure your terminal (or shell) in a way so that the pyenv commands will be loaded. You do this by editing the configuration file of your terminal represented by .zshrc, .bashrc or other file types (sometimes subsumed under the term “shell’s rc file”, the affix means “run commands”). Given that macOS default is based on the zsh Unix shell - also known as Z shell - we will focus to edit only this configuration file (for other options please see this manual: [*]). The configuration file for zsh is called .zshrc and is located in your home folder (~/.zshrc).

Before trying to edit a file we should make sure that the hidden .zshrc file is actually there:

To view the hidden .zshrc file in your home directory you can do the following within macOS Finder: Open Finder and navigate to your home directory (e.g, something like /Users/MyUserName): hit the following keys at once: Command + Shift +.

By using this command you told Finder to display usually hidden files (i.e. files starting with a dot). If you see the .zshrc file within Finder you can open it by selecting any text editing application (e.g. TextEdit). In case there is no .zshrc file in your home directory, please create one: just see the instructions in this section (tab).

Open a terminal window (the hidden file should be located in your home folder: something like /Users/MyUserName/) and enter the following command:

open ~/.zshrc

In case this file doesn’t exist you will receive a respective feedback (“file doesn’t exist”). In case there is no such .zshrc file in your home directory, please create one: just see the instructions in this section (tab).

Within your terminal window enter the following command:

touch ~/.zshrc

The operation starting with touch will create the hidden .zshrc file in your home directory. You can verify the creation of this file by following one of the options shown in this section (see other tabs).

Since we made sure that the configuration file for our shell is indeed available (i.e., the .zshrc file) we can now add the following three expressions to this configuration file. To add these modifications to your shell’s configuration enter the following commands in your terminal:

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

After having executed these commands your .zshrc file will have three new lines included.

Note

You can verify the successful installation and setup of pyenv by entering the following command in your terminal window:

pyenv

If all went fine, you should receive the output of the actual version of pyenv installed. In my case the output is as follows:

pyenv 2.3.5

In addition, you should see a list of command options (e.g. “--version”, “commands”, “exec” and several more).

With pyenv installed successfully you can get a nice overview about all installed Python versions: just enter the following pyenv command:

pyenv versions

Since we haven’t yet installed any additional Python version, pyenv will show you the system Python version (pre-installed on our macOS): you will see something like this (exemplary output):

  * system (set by /Users/MyUserName/.pyenv/version)

Given that no other version of Python is installed pyenv is enlisting the Python system version. The star * symbol marks the Python version set to be global, that is the Python version that serves as the default version of Python which macOS uses when running Python applications.

You can use the following command to confirm the location of the Python system version:

pyenv which python3

This will show the following output:

/usr/bin/python3

This shows the actual path to your system Python version.

You might be tempted to enter the traditional terminal command which python3 to locate the Python’s location. However, this command will reveal only the pyenv path modifications inserted in your shell’s configuration. In fact you will see something like this: /Users/MyUserName/.pyenv/shims/python3

2c: We are now ready to pyenv in order to install a new Python version. In case you know the precise Python version you would like to install (for the sake of this example let’s say 3.10.5) simply execute the following pyenv command:

pyenv install 3.10.5

To verify which Python versions are installed and available on your computer just enter the following command we saw before:

pyenv versions

This time the output should look like along these lines:

* system (set by /Users/ggs/.pyenv/version)
  3.10.5

The output provides a list of all installed Python versions, while the star * symbol marks the Python version set to be global (default).

Nice to know: Which Python versions are available to install?

In case you would like to get a list of all Python versions that are available to be installed via pyenv you can use the following command (as a shortcut you can also type: pyenv install -l):

pyenv install --list

This command will give you a list of available Python versions (and additionally showing other libraries we can ignore for the moment) such as:

Available versions:
  2.1.3
  2.2.3
  2.3.7
...
  3.10.5
  3.10.6
  3.10.7
...

2d: For our current example we have seen two versions of Python installed on our computer. Using pyenv we can switch between these two Python versions, that is we can select one version and set it the global default: this global Python version will become the one macOS uses when running Python applications. Set a given Python version to global by executing the following command:

pyenv global 3.10.5

When running the previously shown command pyenv versions again we will see that the Python version marked to be global has changed accordingly (exemplary output):

system
* 3.10.5 (set by /Users/MyUserName/.pyenv/version)

As intended the star * symbol marking the Python version set as global has changed. In case you have several more Python versions installed you can change to each one of them using the pyenv global command shown above.

You can change back to using your system Python version as the global one via the following command:

pyenv global system

3. Install virtualenvwrapper using your system Python version (and its associated pip version)

3a: Before you continue please consider the following important requirement:

Important

Before continuing with installing the virtualenvwrapper library make sure that your system Python version is set to global (just run the command pyenv versions and verify that the star * symbol is shown next to the system Python version - as detailed in step 2 above). If that is not the case, you can simply set the system Python version to global by executing this command: pyenv global system.

Being sure that you are using the correct Python version i.e. the system Python version, you can now run the following installation command. It will install the virtualenvwrapper library on your computer [*]
(Note: The usage of the sudo command will require that you enter your admin password before the installation can be executed):

sudo pip3 install virtualenvwrapper

3b: Next, we need to configure your terminal to load the virtualenvwrapper commands, i.e. we will modify the existing .zsrh file of your terminal in a way that the virtualenvwrapper commands can be used (we saw already a similar procedure for pyenv - see the details of step 2b). To add three modifications to your ~.zshrc file please execute the following three commands within your terminal window [*]:

echo 'export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3' >> ~/.zshrc
echo 'export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv' >> ~/.zshrc
echo 'source /usr/local/bin/virtualenvwrapper.sh' >> ~/.zshrc

You can verify the three newly appended lines by opening the .zshrc configuration file: just enter this command in your terminal:

open ~/.zshrc

3c: Finally, we ready to create our first virtual environment on the basis of our now established setup. However, before creating the virtual environment we need to make one last decision: Which Python version do we want to use in order to create this virtual environment?

Remember: We can see which Python versions are actually installed on our computer by running the pyenv command pyenv versions. Executing this command within your terminal will in our example case lead to the following output:

* system (set by /Users/MyUserName/.pyenv/version)
  3.10.5

As detailed before (please check 2c) the output provides a list of installed Python versions, while one named “system” is marked by the star * symbol indicating that this version is set to global so that it serves at the moment as the default version of Python. That is, the currently set default Python version is the system Python version. From here, we have two options to proceed:

In case we make the decision to use the system Python version to create a virtual environment we can now proceed with executing the respective virtualenvwrapper command (please see the tab below: Option I).
In case we want to use another Python version available on our computer (in our example: Python version 3.10.5), we first need to set this version to global. Only after having changed the default Python version (in our case to 3.10.5) we can then proceed with creating the virtual environment. Please see everything summarized in the tab below (Option II).

Option I (using the system Python version)
Option II (using a user-installed Python version)

The prerequisite for this option is that the system Python version is set to global (or in other words: when you have entered the terminal command pyenv versions and you see the star * symbol marking the Python version named “system”). If this is not the case, just set the system Python version to global by using the pyenv command as shown before: pyenv global system.

Now you can simply run the following virtualenvwrapper command with a name for the virtual environment of your choice (in the example below I will use the name myenv_test1) to create a new virtual environment:

mkvirtualenv myenv_test1

Note that executing this command above will create and at the same time activate the new virtual environment: you will notice that the prompt line in your terminal window has changed:

myusername@mbp ~ % (before)
(myenv_test1) myusername@mbp ~ % (after)

The fact that (myenv_test1) appears before your prompt shows that this virtual environment is in fact activated. A virtual environment constitutes a separate and independent workspace: Any packages you install using pip will be installed within this workspace alone, so that they will not interfere with your other projects, other virtual environments, or your system installation. To work smoothly with virtual environment please see the most important commands (such as activating, deactivating, etc.) in the summary table below.

First, we need to know which Python version is set to be the global default: as demonstrated before (see 2d) we get a list of all installed Python versions using this pyenv command: pyenv versions. The output is a list of Python versions, where one is marked by the star * symbol to indicate the version currently set to the global default. For our example case, let’s assume the output looks like the following:

 * system (set by /Users/MyUserName/.pyenv/version)
  3.10.5

We decided that we want to use a new user-installed Python version to create a virtual environment. Therefore, based on the output shown above, we need to change the global Python version to 3.10.5. We will do this again by using the pyenv global command (see 2d):

pyenv global 3.10.5

With this user-installed Python version (3.10.5) set to global, we can now create a new virtual environment based on this exact Python version. To that end, we will call the virtualenvwrapper command mkvirtualenv together with an additional flag argument -p and specifying the actual Python version to be used for creating the new virtual environment (I will name it this time myenv_test2):

mkvirtualenv -p python3.10.5 myenv_test2

Executing this command above will create and at the same time activate the new virtual environment: this fact is also reflected by the change in the prompt line of your terminal window:

myusername@mbp ~ % (before)
(myenv_test2) myusername@mbp ~ % (after)

To work smoothly with virtual environment please see the most important commands (such as activating, deactivating, etc.) in the summary table below.

The following table summarizes some important operations working with virtual environments provided by virtualenvwrapper (for a complete list please see the manual [*]).

Managing virtual environments (*virtualenvwrapper*)
Desired Outcome	Command	Comment
Create a virtual environment named “myenv_test1”	`mkvirtualenv myenv_test1`	The creation of the virtual environment is based upon the Python system version
Create a virtual environment named “myenv_test2” which is based on a specified Python version	`mkvirtualenv -p python3.10.5 myenv_test2`	Before specifying a user-installed Python version that version need to be set to global (see 2d)
Deactivate the currently running virtual environment	`deactivate`	After executing this command the prompt line of your window will have changed (see 3c)
Re-enable an already created virtual environment named “myenv_test1”	`workon myenv_test1`	After executing this command the prompt line of your window will have changed (see 3c)
List all virtual environments currently available on your computer	`lsvirtualenv`	As an alternative command you can also use `workon` without any additional arguments
Remove the virtual environment named “myenv_test1”	`rmvirtualenv myenv_test1`	Before removing a given virtual environment you first need to deactivate it

All these steps together should give you a clean and maintainable setup for all your Python coding work. I hope that my explanations helped you on the way. Happy coding & Cheers!