Connecting to your notebook kernel using Jupyter console

Jupyter notebooks are a great way to explore data using Python (and other languages as well). Having a visual representation of your code and output, along with documentation and formatting in one view can be extremely helpful. However, there are some things that are just much better to do in a console session. In this post, I’m going to show you how to connect to the kernel that your notebook is using when you want to make use some of the convenience and speed of using a console.

Jupyter basics

First, let’s just establish a few basic details about Jupyter and its components. It’s good to know that the Jupyter project is a large project with a number of components. For now, we’ll just concentrate on two main portions: user interfaces and kernels. If you’ve always only used notebooks, you may not realize that the Jupyter projects supports a number of diffent user interfaces to the underlying kernel that is executing code. For now, we’ll look at just two user interfaces: notebooks and the console.

Notebook basics

I’m assuming you already know how to run a Jupyter notebook. If not, check out the basic docs.

Now, let’s assume you’re in a running Jupyter notebook. I’m writing this sentence now in a notebook, so I’ll first create a code cell and execute it.

x = 5

At this point, you should know that there is a kernel executing that the notebook front end is connected to. In that IPython kernel, there is now a variable x that has a value of 5. Now if you want to inspect that variable, you can create another cell, type x in it, and then execute it.

x
5

Why would I want to connect with the console?

There are a lot of great things about running code in a notebook, but there are a number of cases where notebooks just don’t cut it. One example is when you’re deep into debugging some isolated code. In this case, using a quick Read-Eval-Print Loop (REPL) environment like a console allows you to work much faster. Maybe you want to execute a function with 5 different parameters quickly and see the return values. If you’re using a notebook, you’ll have to either create or edit a cell five different times and then execute it. If you’re in a console, you can just quickly execute those lines and see the results.

A second area that a console shines is your command history. In the notebook, once you change a cell, you don’t see the previous value. In the console, you can see (and edit) the history of code that you’ve executed. It’s a big win to quickly and efficiently edit and re-execute those lines, especially when you’re making small changes and honing in on a solution.

Another example is when exploring values that are already defined in your session. In a notebook, you have to edit a cell and execute it each time you want to see anything. In a terminal, you can very quickly just view the values by executing a line with that variable to inspect it.

I’d recommend looking through the IPython tutorial for some more examples of using IPython. Much of this functionality will be helpful both in a console and notebook, but some features, like traversing command history, make using a console very helpful. 

Getting connected

Now, I’ll show you how you can connect to this kernel with another front end, a Jupyter console instead of a notebook. So what do you need?

Indexing in pandas can be so confusing

There are so many ways to do the same thing! What is the difference between .loc, .iloc, .ix, and []?  You can read the official documentation but there's so much of it and it seems so confusing. You can ask a question on Stack Overflow, but you're just as likely to get too many different and confusing answers as no answer at all. And existing answers don't fit your scenario.

You just need to get started with the basics.

What if you could quickly learn the basics of indexing and selecting data in pandas with clear examples and instructions on why and when you should use each one? What if the examples were all consistent, used realistic data, and included extra relevant background information?

Master the basics of pandas indexing with my free ebook. You'll learn what you need to get comfortable with pandas indexing. Covered topics include:

  • what an index is and why it is needed
  • how to select data in both a Series and DataFrame.
  • the difference between .loc, .iloc, .ix, and [] and when (and if) you should use them.
  • slicing, and how pandas slicing compares to regular Python slicing
  • boolean indexing
  • selecting via callable
  • how to use where and mask.
  • how to use query, and how it can help performance
  • time series indexing

Because it's highly focused, you'll learn the basics of indexing and be able to fall back on this knowledge time and again as you use other features in pandas.

Just give me your email and you'll get the free 57 page e-book, along with helpful articles about Python, pandas, and related technologies once or twice a month. Unsubscribe at any time.

Invalid email address

First, make sure you have a console running using the same virtualenv as your Jupyter notebook instance. You can do this by starting a new terminal session and choosing your virtualenv as you did to launch the notebook instance above. Another option is to browse to the Jupyter home page on your local machine (probably something like http://localhost:8888) and launch a new terminal from the dropdown in the upper right corner.

Next, you can run this command: jupyter console --existing

At this point, you should connect to your existing kernel. Type x and hit return, and you should see that the variable already exists and has a value of 5. You’ll even see that the prompt history number should be the next number. So if the last command you ran in the notebook had a prompt of Out[5]:, then the console should be Out[6].

What if I’m running lots of notebooks?

Many times you’ll be running multiple notebooks, and you want to connect to a specific instance. To do this, you need to know which specific kernel to connect to. In your notebook, run the %connect_info magic, and a helpful message will show you how to do this.

{
  "shell_port": 57293,
  "iopub_port": 57294,
  "stdin_port": 57295,
  "control_port": 57297,
  "hb_port": 57296,
  "ip": "127.0.0.1",
  "key": "c2dc5435-fc2c47faa20a16212f0c92ac",
  "transport": "tcp",
  "signature_scheme": "hmac-sha256",
  "kernel_name": ""
}

Paste the above JSON into a file, and connect with:
    $> jupyter <app> --existing <file>
or, if you are local, you can connect with just:
    $> jupyter <app> --existing kernel-b5e78585-d011-4876-97cd-cb130fbd26ff.json
or even just:
    $> jupyter <app> --existing
if this is the most recent Jupyter kernel you have started.

So in our case, we’ll run the command jupyter console --existing kernel-b5e78585-d011-4876-97cd-cb130fbd26ff.json.

Once we’re connected, we can modify the value of x, and see that the change makes its way back to our notebook frontend by inspection (in a new cell or by re-executing an earlier cell).

One word of warning, notebooks are already an easy way to make a big mess of your code. If you start executing code in your notebook and your console session in a non-linear order, and then try to reproduce that output at a later time, you’ll find yourself very frustrated. Make sure that you only use this technique when you’re drilling into a specific issue where having console access is going to help you solve a specific problem. And when you have solved that problem, it usually makes sense to ensure all code changes are persisted to your notebook and saved, and if possible, you should restart your kernel and run the notebook from the top.

Don't miss any articles!

If you like this article, give me your email and I'll send you my latest articles along with other helpful links and tips with a focus on Python, pandas, and related tools.

Invalid email address
I promise not to spam you, and you can unsubscribe at any time.

2 thoughts on “Connecting to your notebook kernel using Jupyter console

    1. If you are on the same host where Jupyter is running, and you have the same jupyter in your path, you can run this shell command:

      ls -ltr $(jupyter --runtime-dir)

      This will list the contents of the runtime directory, and the json file that you get from running %connect_info will be there. However, I don’t know how you can tell different instances apart from one another, besides looking at the timestamp or connecting and poking around to see what is in that running kernel.

Have anything to say about this topic?