RFC: A Jupyter kernel for Tablegen

LLVM Project
1

Hi all, myself and some colleagues at Arm were discussing how we can help people learn how to do the work that we do in llvm. Which for architecture support involves a lot of TableGen.

I thought it would be great to have more “explorer” type tools along the lines of Compiler Explorer, which I use all the time to experiment with various languages.

There is a system in the same lane called Jupyter (https://jupyter.org/). If you don’t know already, Jupyter lets you make “notebooks” which are like the notes you might take when you’re studying. Except that you can include the code you were using, and the result of it, inside the document itself.

(meaning, if someone else’s notes don’t cover what you want, just edit the code and find out for yourself!)

Jupyter has a Python focus (hence the name) but you can add what’s called a “kernel” for almost anything. So I did one for Tablegen. Here’s an example of what you can do with it:

%reset
%json
class Root {}
def thing : Root {}

{
    "!instanceof": {
        "Root": [
            "thing"
        ]
    },
    "!tablegen_json_version": 1,
    "thing": {
        "!anonymous": false,
        "!fields": [],
        "!name": "thing",
        "!superclasses": [
            "Root"
        ]
    }
}```

Or you can see a full notebook here: llvm-project/LLVM TableGen.md at 7eb80c910e923c395b0807068d7485a246989cf3 · DavidSpickett/llvm-project · GitHub

I am looking for anyone who would like to review the changes (either because you want to review some Python code or you have used other notebooks in the past).

You can also use the tree here to try it out: GitHub - DavidSpickett/llvm-project at tablgen-jupyter

I welcome any suggestions for future features. I am new to notebooks myself so I am still impressed at just being able to embed the output, I’m sure there’s more to it.

My next goal is to write a notebook that covers the basics of Tablegen. Following the same topics as 1   TableGen Programmer’s Reference — LLVM 16.0.0git documentation. That will be checked into the repo next to the kernel itself.

4 Likes
2

And I should mention there is a kernel for MLIR llvm-project/mlir/utils/jupyter at main · llvm/llvm-project · GitHub which I used as the base for this.

3

That’s really neat. I suspect one of the most important features for large scale usability is to have ways to filter the output, e.g. search for specific record names, or for records defined (directly or indirectly) by specific lines in the TableGen source.

4

Can you give an example of what that might look like?

My guess at it is that you might be building up a large example and want to say “ok let’s see what the Foo records look like” then “and here’s what the Bar records look like”. Without printing the entire output out each time. That sort of thing?

Which is also what a tablegen backend tends to do so that’s a good direction. There’s a lot of “show me all the things that are instances of X class”.

A subset of that might be to see only the things that the current cell changed. So you get the benefit of the previous code but don’t have to look at a lot of output each time.

5

I like this idea!

I think you’re referring to dumping the (parsed) records rather than a specific TG backend e.g. showing the corresponding SelectionDAGISel C++ code of a Pattern record. Because that would require quite a lot TG backend changes for the latter case.

6

Yes.

Thinking about this a bit more though, I believe where this tool would have been useful for me in the past is when trying to define complex “computational” TableGen classes and multiclass constructs. For those, it’s almost always desirable to have larger context. So perhaps the first step is actually making sure that one can include TableGen source from disk, to pull in e.g. a backend’s instruction classes.

But what you edit in the notebook is just some snippet, not the entire source on disk. So perhaps the second step is to (by default) filter the output to only show those records that are defined (directly or indirectly) by statements in the snippet of the notebook; i.e., hide the records that are defined by included TableGen files.

7

Exactly. I’m not suggesting we start adding filters to the backends but look at allowing people to do what backends tend to do e.g. search for instances of a class.

So as it stands you can include "..." in the cell as you would in a normal .td file but you’re right that the important thing is what one does with what that file provides. The new stuff you make with it. I can see it being useful for us for new instructions. Include ARMInstrInfo.td and start building.

Similar idea to silencing warnings from system includes.

Perhaps includes using a magic directive could have that behaviour, though I need to think about how that composes with the model of having the cells linked.

1 Like
8

Update for anyone wondering about the status here.

Handling larger examples will be important but I want to make sure I don’t let that obscure the initial goal of smaller tutorials. So I am going to keep the patches in review while I write a first tutorial notebook and use that experience to inform the next features.

(of course if someone does want to hack on what’s there go for it, let me know what you find)

9

Quick update start of 2023.

The lightning talk from US llvm dev is now up 2022 LLVM Dev Mtg: Interactive Programming for LLVM TableGen - YouTube

The kernel is now in the repo, you can find it at llvm-project/llvm/utils/TableGen/jupyter at main · llvm/llvm-project · GitHub

It comes with a demo notebook: llvm-project/LLVM_TableGen.ipynb at main · llvm/llvm-project · GitHub (you can read this on directly on Github)

The first part of the tutorial notebooks is in review ⚙ D137085 [LLVM][TableGen] Add first tutorial notebook.

If anyone does have time to follow it and review that would be amazing, but I appreciate it’s a lot more effort than a code review. I hope to find a colleague who is just starting to learn TableGen and have them review it as they go.

3 Likes
10

Unrelated to the kernel, there is now a notebook about writing a backend using tablegen’s JSON output - llvm-project/sql_query_backend.ipynb at main · llvm/llvm-project · GitHub

1 Like
11

Hi David,
I’m in the process of learning more about TableGen and I think this can be useful for people like me. I would happily follow along the tutorials, give feedback and possibly contribute.

Also I am not that familiar with Phabricator and if I do something awkward please let me know :slight_smile:

12

Yes please!