TableGen: the playground .ipynb file is not working as expected

Beginners
1

So here is what I have done:

Step 1

I have the following environment:

> llvm-tblgen --version
LLVM (http://llvm.org/):
  LLVM version 16.0.6
  Optimized build with assertions.

> python --version
Python 3.10.8

> jupyter --version
Selected Jupyter core packages...
IPython          : 8.14.0
ipykernel        : 6.23.3
ipywidgets       : not installed
jupyter_client   : 8.3.0
jupyter_core     : 5.3.1
jupyter_server   : not installed
jupyterlab       : not installed
nbclient         : not installed
nbconvert        : not installed
nbformat         : not installed
notebook         : not installed
qtconsole        : not installed
traitlets        : 5.9.0

Further, since I built LLVM 16.x locally, with the built binaries under the directory llvm-project/installed, I have set the following environment variables:

> echo $PATH
<user home>/llvm-project/installed:<other stuff>

> echo $LLVM_TBLGEN_EXECUTABLE
<user home>/llvm-project/installed/llvm-tblgen

> echo $PYTHONPATH
<user home>/llvm-project/llvm/utils/TableGen/jupyter

Step 2

I moved to the path: llvm-project/llvm/utils/TableGen/jupyter/

I first executed the command:

> python -m tablegen_kernel.install
Installing llvm-tblgen IPython kernel spec

> jupyter kernelspec list
Available kernels:
  python3     <user home>/custom_py_envs/py310env/share/jupyter/kernels/python3
  tablegen    <user home>/.local/share/jupyter/kernels/tablegen

Step 3

In my VSCODE, I launched the LLVM_TableGen.ipynb notebook. I executed the very first block:

%reset
// This is some tablegen
class Foo {}

It gave me the following output:

  Cell In[8], line 2
    // This is some tablegen
    ^
SyntaxError: invalid syntax


Errors printed to stderr are shown.

I believe the correct output should have been:

    ------------- Classes -----------------
    class Foo {
    }
    ------------- Defs -----------------


Errors printed to stderr are shown.

Please help me understand what mistake I made in setting up the TableGen playground.

2

This is strange, I will try it myself. I’ve not used it via VSCode but I can’t imagine it’s too different.

1 Like
3

Btw this is a thorough report, thank you for that.

I think what is happening is it’s running the notebook using the iPython kernel. The error is coming from the Python interpreter.

$ python3
>>> def foo(?):
  File "<stdin>", line 1
    def foo(?):
            ^
SyntaxError: invalid syntax

What I had to do was click a button on the top right of the “code - markdown - run all…” bar. At first it said “Select Kernel”, then I selected “LLVM Tablegen” from the menu.

You might have to select a Python environment first if the kernel is not listed, it seemed like I had to do that some times but could be confusion on my part.

The other thing to note is that I think you’ll have to restart all instances of VSCode to get it to see any new environment variables.

1 Like
4

When I selected the LLVM TableGen kernel from the dropdown,

image
image799Ă—437 33 KB

(disclaimer: the above screenshot was taken after fixing the error!)

and executed the block, I am receiving the following error:

The kernel died. Error: Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 185, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.8/runpy.py", line 144, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.8/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "<user home>/llvm-project/llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py", line 6, in <module>
    from .kernel import __version__
  File "<user home>/llvm-project/llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py", line 9, in <module>
    from ipykernel.kernelbase import Kernel
ModuleNotFoundError: No module named 'ipykernel'... View Jupyter log for further details.

The scenario is this: I have created a virtual python environment and am using that. Even jupyter was installed inside that environment. However, upon selecting the LLVM TableGen kernel, for some reason, it is selecting the system python3 and not the one in my environment.

Here is the tablegen kernel’s .json file:

{
    "argv": [
        "python3", "-m", "tablegen_kernel", "-f", "{connection_file}"
    ],
    "display_name": "LLVM TableGen",
    "language": "tablegen",
    "language_info": {
        "name": "tablegen",
        "codemirror_mode": "tablegen",
        "mimetype": "text/x-tablegen",
        "file_extension": ".td",
        "pygments_lexer": "text"
    }
}

Any heads up regarding how to fix that?

5

I tried uninstalling the LLVM TableGen kernel and reinstalling it:

> cd <user home>/llvm-compiler

> jupyter kernelspec uninstall tablegen
Kernel specs to remove:
  tablegen              <user home>/.local/share/jupyter/kernels/tablegen
Remove 1 kernel specs [y/N]: y
Removed <user home>/.local/share/jupyter/kernels/tablegen

Then, I tried to reinstall it with the specific Python interpreter:

> cd llvm/utils/TableGen/jupyter

> $(which python) -m tablegen_kernel.install
Installing llvm-tblgen IPython kernel spec

> jupyter kernelspec list
Available kernels:
  python3     <user home>/custom_py_envs/py310env/share/jupyter/kernels/python3
  tablegen    <user home>/.local/share/jupyter/kernels/tablegen

The problem is right there - as seen in the output of last command, the path of the python3 kernel and the tablegen kernel is different.

To fix the kernel’s execution within the environment’s Python interpreter, I am currently using a hack:

In the kernel.json file located at: <user home>/.local/share/jupyter/kernels/tablegen, I changed the value python3 to the complete path of the python interpreter of choice (here path of python in the virtual environment):

{
    "argv": [
        "<user home>/custom_py_envs/py310env/bin/python", "-m", "tablegen_kernel", "-f", "{connection_file}"
    ],
    "display_name": "LLVM TableGen",
    "language": "tablegen",
    "language_info": {
        "name": "tablegen",
        "codemirror_mode": "tablegen",
        "mimetype": "text/x-tablegen",
        "file_extension": ".td",
        "pygments_lexer": "text"
    }
}

Here, is the final screenshot of the working .ipynb:

image
image807Ă—382 24.4 KB

6

This might explain why I had to select a Python first sometimes.

There is an option to add a new Python, “Python Environments” → “Create Python Environment”. However it seems to only be for creating a new virtualenv rather than pointing to one.

Potentially you could select an interpreter this way: Using Python Environments in Visual Studio Code too. Though I think vscode is being smart and knowing which kernels are tied to which interpreters.

I will start my local setup fresh and see if I can improve the setup instructions to avoid the problems you’ve hit.

7

Actually now I see what you mean specifically. The install json should include the python you ran the install command with but it’s hardcoded to python3. There is a way I can fix that.

8

https://reviews.llvm.org/D154351

If you can check that on your setup that would be great. VSCode was still a bit weird in that if I selected the kernel first, it wouldn’t work, but once I selected the virtualenv, then the kernel, that was fine.

Could just be a me problem, some leftover files perhaps.

1 Like
9

To test it, I first uninstalled the tablegen kernel and then copied the contents of the install.py file from the link shared above.

Then I reinstalled and tried the .ipynb file. Everything worked just fine. Thank you.

10

[llvm][TableGen][Jupyter] Record current python when kernel is installed · llvm/llvm-project@535693f · GitHub fixes the Python path and [llvm][TableGen][Jupyter] Note an easily encountered error · llvm/llvm-project@dacbf4a · GitHub notes the initial error you saw.

Thanks again for the report. I also use VSCode for other things but hadn’t tried a notebook in it, very glad I can now!

1 Like
11

I believe the README.md file should also mention that when working with an IDE like tool (such as VSCode) after installing the LLVM TableGen kernel, the IDE needs to be restarted.

1 Like
12

Agreed - [llvm][TableGen][Jupyter] Note needed restart when using an IDE · llvm/llvm-project@2c93cfa · GitHub.

1 Like