TorchDynamo Developer Hour Slides and Notes

Incubator torch-mlir
1

On the 2022-09-19 we discussed TorchDynamo and the long-term technical direction path for Torch-MLIR. Overall, there was strong alignment on the direction between Torch-MLIR and upstream PyTorch.

Meeting Slides

Attendees:

  • PyTorch core devs: Ed Yang, Jason Ansel, Horace He, Natalia Gimelshein, Mike Ruberry, Michael Voznesensky
  • Nod.ai: Anush Elangovan
  • Cruise: Sambhav Jain
  • Google: Sean Silva, Ramiro Leal-Cavazos
  • ByteDance: Ziheng Jiang
  • … (this is from memory – apologies if your name is not listed – please reply and we can update this list)

Meeting summary (focusing on what is not on the slides):

  • Ramiro showed his PoC integrating TorchDynamo, Torch-MLIR, and IREE to achieve a 2.4x speedup on a BERT model on GPU.
  • We discussed the Torch-MLIR requirement for Ahead-of-Time compile + dynamic shapes
    • Horace mentioned that the FakeTensor stored in node.meta["val"] on the FX graphs produced by make_fx should have symbolic shape information, but there is more work to be done to make this path work in full generality.
    • Action Item: Sean will follow up with Horace to distill a “north star” example of this for progress tracking
  • Mike Ruberry asks about the UX for dynamic shapes in AoT use cases
    • How to prevent user footguns if the example inputs don’t cover all possible shapes?
    • One possibility is for compilation to error if TorchDynamo requires any non-trivial guards on dynamically shaped dimensions
    • Michael Voznesensky mentioned that he was developing a pass to convert the guards into internal asserts in the program as well.
  • PyTorch folks mentioned that FX graphs will eventually support “cond”-like operators
  • Ops with data-dependent shapes like “nonzero” – how to handle?
    • Horace said that TorchDynamo will create a new symbolic size for the output and guard on any control flow against that value, but otherwise it should work fine.
  • Horace mentioned the use case of specifying that 2 dynamic shapes are equal on inputs is often needed
    • Torch-MLIR doesn’t have a way to represent this. No action item but keeping an eye on that.
  • Similarly, the symbolic shapes produced by TorchDynamo are full SymPy expressions, but Torch-MLIR doesn’t have a way to represent this, so it will discard that rich information and just use “unknown”. No action item but keeping an eye on that.
  • PyTorch devs mentioned that TD should be in mainline PyTorch by 2022EOY
    • This will allow Torch-MLIR to build a TorchDynamo frontend on by default.
2

Thank you for the summary, Sean. I missed this meeting somehow :frowning:

A couple of questions: what’s the use case for dynamic shapes? When you use dynamo in “normal” execution (eager), we don’t need dynamic shapes, right?
You want this to capture the whole model at once? So you can execute the model afterwards outside of Python with a shape to be defined later?

Or do you envision multiple dynamo frames that are equal modulo types and thus you want to save resources on archs with long compile times (like TPU, and other accelerators)?

Thank you!

3

You’re exactly right :slight_smile:

4

Sounds good, thank you!

5

Glad to see this progress!