MLIR JIT/ExecutionEngine performance issue

I notice that for compiling the same program this line here takes 10m when 2 processes are running vs 27s when 1 processes is running. It’s just pass manager running all passes so I am not sure why the execution time is so different.

More context on this:
We launch python processes that uses ExecutionEngine from MLIR with Python binding to execute multiple python programs compiled to LLVM IR. We notice that on a particular machine, if more than one processes are launched to execute the python programs there are significant more ticks from this method. All the processes are doing the same work which is compiling and executing a python program fetched from a job queue. More investigation shows the ticks ended up in PM.run(M); in SimpleCompiler::operator()(Module &M) I linked above.

Does anyone have a clue what might be the reason behind this?

You may be overloading the machine with a large number of threads: each process will likely create as many threads as there are core in the machine.

If you know you’re already exploiting cross-process parallelism in a way that saturates the machine, it is probably best to disable threading in the MLIRContext.

Note: this is mostly a wild guess based on the information you provided, it may be a very different root cause but that’s hard to tell right now.

Thanks Mehdi for the suggestion! It didn’t work though…
I saw there is an time-passes option which might give me more insights into what’s happing in the pass manager. Do you know how can pass the option to ExecutionEngine?