When your Python program uses more memory than expected, you can use memory_profiler to find out where memory is allocated.
Jupyter notebooks can easily have hidden state. Use these methods to see all the variables that exist in your Jupyter notebooks.
Using the %autoreload magic in IPython or Jupyter can help you continue working without restarting your session after making local changes.
It is possible to unit test Python code that lives in a Jupyter notebook. This article looks at three reasonable choices.
You can use py-spy to profile already running Python code without restarting your process or modifying the source code.
Removing one or more columns from a pandas DataFrame is a pretty common task, but it turns out there are a number of possible ways to perform this task. I found that this StackOverflow question, along with solutions and discussion in it raised a number of interesting topics. It is worth digging in a little bit to the … Continue reading How to remove a column from a DataFrame, with some extra detail
Once we have debugged, working, readable (and hopefully testable) code, it may become important to examine it more closely and try to improve the code's performance. Before we can make any progress in determining if our changes are an improvement, we need to measure the current performance and see where it is spending its time. … Continue reading Profiling Python code with line_profiler
We would love for our Python programs to run as fast as possible, but figuring out how to speed things up requires gathering information about the current state of our code and knowing techniques to speed things up. First and foremost, we need to know where our program is spending its time, and what is … Continue reading Profiling Python with cProfile, and a speedup tip
If you've done any work in pandas, you've surely seen the SettingWithCopyWarning. This is an explanation of what's happening and how to fix it.
The query method in pandas DataFrames provides some flexibility in code, and potential speedups using numexpr.