In my earlier post, I wrote about how pyenv is a great tool for running multiple versions of Python on the same host. It makes it simple to install multiple versions of Python on your workstation or server and control which version executes in a shell. But as a Python developer, the Python version is only one part of the environment. Most Python developers will work on more than one project at a time and want to install a number of Python packages for use in each project. Installing modules globally is rarely a good idea, especially if you are likely to use that Python version for more than one project. What happens when one projects wants a specific version of a package that won’t work with another project? Instead, using virtualenv or anaconda is the way to go. Luckily, both work well with pyenv. In this post I’ll look at using basic virtualenv, the pyenv-virtualenv plugin, and anaconda to build an isolated virtual environment that has a package installed in it that will be isolated to that environment.
Before discussing these details, I’ll mention that this post does not talk about the complexity of maintaining package compatibility within virtual environments. That is a topic for another post (or set of posts).
virtualenv
Assuming you followed the installation steps in the first post on pyenv, you should already know how to setup your shell to use a specific version of Python. As in that post, I’ll go ahead and install a unique Python version. Then, using that version I’ll
- install virtualenv (globally)
- use the virtualenv command to make a virtualenv
- activate the virtualenv
- use pip to install packages into that virtualenv
Note that all of these examples were run on a Mac running macOS Catalina and using zsh. This can all be run using the shell of your choice on Mac, Linux, or Windows using WSL.
❯ pyenv install --list | grep 3.8 # look for the latest 3.8 version 3.8.0 3.8-dev 3.8.1 3.8.2 3.8.3 3.8.4 3.8.5 3.8.6 miniconda-3.8.3 miniconda3-3.8.3 ❮ pyenv install 3.8.6 python-build: use [email protected] from homebrew python-build: use readline from homebrew Installing Python-3.8.6... python-build: use readline from homebrew python-build: use zlib from xcode sdk Installed Python-3.8.6 to /Users/mcw/.pyenv/versions/3.8.6 ❯ pyenv shell 3.8.6 # sets the version just for this shell ❯ pyenv which pip # show which executable is running for pip, it's the newly installed one /Users/mcw/.pyenv/versions/3.8.6/bin/pip ❯ pip install virtualenv Collecting virtualenv Using cached virtualenv-20.1.0-py2.py3-none-any.whl (4.9 MB) Collecting appdirs<2,>=1.4.3 Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB) Collecting filelock<4,>=3.0.0 Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB) Collecting distlib<1,>=0.3.1 Using cached distlib-0.3.1-py2.py3-none-any.whl (335 kB) Collecting six<2,>=1.9.0 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Installing collected packages: appdirs, filelock, distlib, six, virtualenv Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 six-1.15.0 virtualenv-20.1.0
Now that virtualenv is installed in the Python environment, I can setup a virtualenv for our test project. For this example, I’m putting the virtualenv in a projects directory, but you can put it anywhere you want, including in a hidden directory in our source tree like .env.
❯ cd projects
❯ virtualenv myenv
created virtual environment CPython3.8.6.final.0-64 in 407ms
creator CPython3Posix(dest=/Users/mcw/projects/myenv, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mcw/Library/Application Support/virtualenv)
added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
❯ . myenv/bin/activate # activates the 3.8.6 virtualenv to isolate our pip installs
❯ pip install requests
Collecting requests
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting idna<3,>=2.5
Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Installing collected packages: idna, chardet, urllib3, certifi, requests
Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 urllib3-1.25.11
❯ python
Python 3.8.6 (default, Nov 1 2020, 17:41:10)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__file__
'/Users/mcw/projects/myenv/lib/python3.8/site-packages/requests/__init__.py'
>>> requests.__version__
'2.24.0'Now just to show you quickly why virtualenvs are great, we’ll install a different version of requests in a different virtualenv. It doesn’t take much work at all.
❯ deactivate # deactivate our old environment ❯ virtualenv myenv2. # same output as above, but for 2nd environment ❯ . myenv2/bin/activate # activate our new one ❯ pip install requests==2.23.0. # pick a different version than last time ❯ python Python 3.8.6 (default, Nov 1 2020, 17:41:10) [Clang 11.0.3 (clang-1103.0.32.62)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.__version__ '2.23.0'
So now there are two isolated environments, each with a different version of the requests module. We can easily switch between the two as needed.
pyenv-virtualenv plugin
It turns out that pyenv supports plugins, and the pyenv-virtualenv plugin helps you use pyenv with virtualenv (or conda, which we’ll talk about next). Take a look at their docs for the installation process. It’s very similar to the installation of pyenv itself (I used brew to install it in my environment). Once installed, you get some new commands available in pyenv.
activate Activate virtual environment deactivate Deactivate virtual environment virtualenv Create a Python virtualenv using the pyenv-virtualenv plugin virtualenv-delete Uninstall a specific Python virtualenv virtualenv-init Configure the shell environment for pyenv-virtualenv virtualenv-prefix Display real_prefix for a Python virtualenv version virtualenvs List all Python virtualenvs found in `$PYENV_ROOT/versions/*'.
If we use these commands to replicate what we did above, it would look like this.
❯ pyenv virtualenv 3.8.6 myenv3 # makes a 3rd virtualenv using the 3.8.6 version # output as before ❯ pyenv virtualenvs 3.8.6/envs/myenv3 (created from /Users/mcw/.pyenv/versions/3.8.6) myenv3 (created from /Users/mcw/.pyenv/versions/3.8.6) ❯ pyenv deactivate # if you still have another virtualenv activated from earlier, you can deactivate this way ❯ pyenv activate myenv3 pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior. ❯ pip install requests # output as before ❯ python Python 3.8.6 (default, Nov 1 2020, 17:41:10) [Clang 11.0.3 (clang-1103.0.32.62)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.__file__ '/Users/mcw/.pyenv/versions/myenv3/lib/python3.8/site-packages/requests/__init__.py'
conda
Another nice thing about pyenv is that it makes working with Anaconda pretty straightforward. I won’t get into Anaconda in detail here, but if you plan on working on data science projects that have a large number of dependencies (like numpy, pandas, scikit-learn, tensorflow, etc.), it is usually going to be much easier to just use anaconda or miniconda to get all of your dependencies installed. Anaconda is the full install, miniconda will just install the bare necessities for you to pick and choose the packages you want quicker installs and need to save disk space.
Pyenv makes it easy to search for all versions of anaconda and miniconda without having to wade through the web site to search for an installer. Once you install a version of Anaconda using pyenv, using conda (the command for managing environments and dependencies) fits in pretty well with the environment so you don’t have to use conda commands for the basic environment creation.
❯ pyenv install --list | grep conda # to see what's available ❯ pyenv install miniconda3-4.7.12 Downloading Miniconda3-4.7.12-MacOSX-x86_64.sh... -> https://repo.anaconda.com/miniconda/Miniconda3-4.7.12-MacOSX-x86_64.sh Installing Miniconda3-4.7.12-MacOSX-x86_64... # <lots of other output snipped> ❯ pyenv versions system 3.6.10 3.8.6 3.8.6/envs/myenv3 3.9.0 miniconda3-4.7.12 * myenv3 (set by PYENV_VERSION environment variable) ❯ pyenv deactivate # if we had an environment activated already (myenv3 from above) ❯ pyenv virtualenv myconda # this will generate lots of output as conda builds the environment ❯ pyenv activate myconda pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
Now the conda tools are available in the shell and can be used instead of pyenv to create environments. Conda is also used to install packages. I won’t get into details here, but to be complete, let’s replicate the earlier task that was done using virtualenv, but this time with conda.
❯ conda env list # see that we have our new environment activated # conda environments: # base /Users/mcw/.pyenv/versions/miniconda3-4.7.12 myconda * /Users/mcw/.pyenv/versions/miniconda3-4.7.12/envs/myconda ❯ conda install requests # instead of pip # this generates a lot of output showing all the dependencies being installed ❯ python Python 3.8.5 (default, Sep 4 2020, 02:22:02) [Clang 10.0.0 ] :: Anaconda, Inc. on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import requests >>> requests.__file__ '/Users/mcw/.pyenv/versions/myconda/lib/python3.8/site-packages/requests/__init__.py'
Pyenv is a useful tool for not only installing and isolating multiple versions of Python, but can easily be used to manage virtual environments, both in Anaconda and using virtualenv. Once a Python developer needs more than one version of Python on a workstation or server, it’s a great way to manage that complexity.
I have a question – why is pyenv so “hidden”. After using it for a while I wonder why this is not the default tool managing python and why it does not come with every python distribution. Are there some drawbacks? Why is it not part of every python beginner tutorial or part of the official python docs?
It seems such a great tool – but I feel that I do miss some “dark secret” why the “official” python world seems to ignore this tool – what is it?
I’m not really sure why it’s not more commonly used, that’s a good question. I can’t even remember how I first discovered it, but found it useful enough that it made sense to write about it. Hopefully it will become more standard. I think I discovered it out of necessity when I needed to run 2-3 versions of Python 3 for different projects and didn’t see better ways.
New to python; appreciate your site, and have a very basic question: is “pyenv” an Python IDE like PYCharm and others or is it an environment one loads within a Python IDE?
Hi William, thanks for reading.
You can think of pyenv as a way to manage versions of Python available on your computer. When you use the pyenv-virtualenv plugin, you can create virtualenvs that use the versions of Python you installed with pyenv. Then if you use an IDE like PyCharm, you have to figure out how to tell it to use that virtualenv in your project. I’m not a PyCharm user, but any good IDE will have a way to point to an existing virtualenv.
Thanks for your post, it is very instructive, I just have one question. Why you recommend to install Anaconda/miniconda from pyenv. I read about other people who don’t recommend it. For example the answer from Simba to this question from Stack Overflow: https://stackoverflow.com/questions/58044214/installing-anaconda-with-pyenv-unable-to-configure-virtual-environment
Would you elaborate more why this option is better for you, from usability, maintenance, etc. Thanks!
Hi David, my experience with using anaconda with pyenv is fairly limited, but I haven’t had the issues that are mentioned in the stack overflow link. In the past, I usually used anaconda for work, and would only have 1-2 environments running with limited changes. I find pyenv very helpful for managing lots of little environment for smaller projects and scripts. I’ve found the pyenv support for anaconda useful for the few times that I choose to use anaconda, but mostly build isolated environments using pyenv and pyenv-virtualenv, then manage my dependencies with pip/requirements.txt or poetry.
I’d say if you are mostly using anaconda, then don’t bother with pyenv. If you mostly use pyenv and occasionally need anaconda, then use it with pyenv, but if it gives you trouble, just go the full anaconda route. I hope that helps, and have to admit that in my most recent jobs I’ve not been using anaconda at all, so take my advice very lightly.