I’ve started using a new approach to managing Python virtual environments, which includes a couple of tools to help me keep track of which environment I’m in as I move between projects. The core idea is a set of 2-letter shortcuts for common virtual environment tasks:
- NE = New Environment
- AE = Activate Environment
- DE = Deactivate Environment
- SE = Show Environment
This approach is based on something I saw Dan Bader do in one of his courses. He used Bash aliases, and I’ve done essentially the same thing with Windows batch files. I also added a few things.
I’ve used virtual environments at times in the past (based on both virtualenv and venv-based), but I’ve always been inclined to not use a virtual environment unless I really need to. I use Python mostly for task-oriented projects, and I like having all my favorite tools at hand at all times. And I’m not tied down to a big legacy code base, so I’m free to standardize on a single Python version, which simplifies switching between projects. I also have the luxury of working by myself, without any need to deploy my Python code to another machine (except when I change dev machines myself) — my deliverables are mostly charts and spreadsheets and CSV files to drive Power BI dashboards, and not Python code. Compared to most Python developers, my needs are simple.
Nonetheless, as a consequence of my infrequent use of virtual environments, I now have over 120 packages installed in my global Python environment, including those I use in almost every project (e.g., Requests) as well as those I installed to test-drive and forgot to uninstall later (e.g., adodbapi). Time to clean up this mess! I don’t plan to drive that number to 0, but “under 20” seems a reasonable goal for making things more manageable going forward.
To get there, I’m going to add virtual environments to about 25-30 projects. I’m using the venv functionality that has come pre-installed with Python since version 3.3, and as usual I’m working on Windows 10 and doing most everything at a command prompt.
Then, once my global environment has been de-cluttered, I plan to keep it that way. Instead of only using virtual environments when I feel I have to, I’m going to use virtual environments by default.
This simplified workflow is intended to help me get there, and help me stay there.
There are a few fundamental recurring tasks in working with virtual environments, as listed above. Let’s look at how to automate each of those tasks with a batch file.
New Environment (NE). This batch file creates a new virtual environment, by running the command
python -m venv env. This creates a virtual environment in the env subfolder.
Activate Environment (AE). This batch file activates the current virtual environment, by executing the command
env\Scripts\activate. Any other active virtual environment is deactivated first.
Deactivate Environment (DE). This batch file deactivates the currently active virtual environment (if any).
Show Environment (SE). This batch file shows which virtual environment is currently active, and it also shows whether the current folder is within that same project or not.
Creating those batch files, and deciding where to put them, sounds tedious. So I wrote a Python script to create all of them. Just download venv_setup.py and run it, and you’ll have the NE, AE, DE, and SE commands set up and ready to use.
The SE command uses venv_setup.py itself, so you should put venv_setup.py in a spot you plan to leave it in, then run it from there to set up the batch files. You can move venv_setup.py and run it from a new location any time, and then the SE command will know where to find it going forward.
For those unfamiliar with batch file idioms, here are a few things to know about the details of these batch files:
- An @ at the beginning of command causes the command itself to not be displayed on the console.
- The %cd% token expands to the drive/path string for the current directory.
- prefacing a command with call causes the batch file to continue executing even if that command exits with an error. So we use this in the deactivate command of ae.bat, since there may or may not be an active virtual environment to deactivate.
- The string nul 2>&1 captures both stderr (nul 2) and stdout (nul 1) from a command. We use it to hide the output of the deactivate command (which writes stderr if it fails).
Here are a few examples of using these shortcuts. First, creating a new virtual environment:
If you are in a folder outside the virtual environment you’ve activated, the SE (show environment) command will warn you of this. It also tells you whether there’s a virtual environment under the current folder, and you can activate that one with AE (activate environment) if desired.
Finishing the Migration
Now that I have these shortcuts in place, I’m going through my projects and getting new virtual environments in place retroactively for most of them. The workflow for each project goes something like this:
- Create a new environment (NE)
- Test the project, figure out what needs to be installed, repeat until everything works.
- Generate a requirements.txt file in the project folder (pip freeze > requirements.txt). This will make it easy to re-create the virtual environment for deployments, etc.
- Push changes to GitHub (if the project has a GitHub repo).
Keeping Requirements.txt Current
I added one more feature to SE (show environment), to help keep requirements.txt files up to date. When you execute the SE command, it checks which packages are installed in the current virtual environment, and verifies that they’re all included in the current folder’s requirements.txt file (if any). Any missing packages are displayed, as a reminder to add them or re-generate requirements.txt.
Customizing the Python Interpreter
As a final step in automating my virtual environment workflow, I’ve made my Python interpreter check at startup whether there’s a virtual environment under the current folder, and if so whether that environment is currently active. If there is a virtual environment present but I haven’t activated it, this warning is displayed when I launch the interpreter:
And if I currently have a different virtual environment active than the one in the current folder, this warning is displayed:
To set that up, I’ve added a system environment variable called PYTHONSTARTUP, which points to interpreter_startup.py, which you can download here. Note that no messages are displayed if there is not a virtual environment under the current folder, or if there is one present and I already have it active; a message is only displayed if something doesn’t appear to make sense.
These two scripts have helped me automate my use of virtual environments, and help me avoid common mistakes such as accidentally installing a package in the wrong environment.