Project proposition: llvm based dsp engine

I have been looking for a while to LLVM as a possible technology to build a DSP execution engine, providing the runtime flexibility needed by real-time interactive DSP applications (like patcher languages), or by plug-in based processors, and in the same moment
the powerful link time optimizations that such a system can provide.

Such a task is daunting for a single developer project like mine, but it may become feasible if such engine could be useful for multiple projects, so to become a community initiative (i am sending this
mail to the Linux Audio Developers mailing list and to the LLVM mailing list, feel free to send it elsewhere if you find it useful).

So, I'll explain in a few lines exactly what i thinking of, and i propose that if anybody is interested in such initiative she/he can contact me directly, and we will see if there is enough interest
for actually start something. Of course, i may also miss an existing project on the same field, in this case, let me know :->.

The idea is the following: build a tool kit, based on LLVM, to dynamically build the execution chain of DSP applications; almost all DSP applications using plug-ins have something of this kind.
This toolkit will be based on code generation and JIT, and should have a flexible architecture so to accomodate multiple plugin API/ABI, to be able to call, from the same execution chain, DSP operation written
for different contexts (for example, LADSPA plug-ins, PD objects, and so on). The code generation would allow this integration for a very low run time cost.

But the most interesting part is actually providing a definition of native, LLVM compiled, plug-ins, and to use the LLVM link time optimizations for doing things like inlining, loop fusions etc, between different DSP operation; this can provide quite a speed improvement for those chains composed of simple operators.

The execution chain should be flexible enough to provide support for multi-rate synchronous data flows, and should include support for multiple threads execution (probably different chains for different threads).

To complete the project, let's add a library of DSP operators, a reasonable high level API to use it, and documentation to help the DSP application designer (i find the LLVM learning curve quite steep :).

Contact maurizio@dececco.name if you are interest in setting up such a project; if enough developers with the right skills and enough interest are found I'll set up a project mailing list to further discuss the subject.

Cordially,
  Maurizio

My own gut feeling on this is that you'd be better off figuring out
how to do this in the context of Faust, which already does a important
subset of what you are describing, though notably without (I think)
the LLVM part.

Note also that hosts which run plugins at the level of LADSPA/LV2,
VST, AU, DSSI etc are unlikely to be easy candidates for any
cross-plugin optimization. I can say for sure that, for example,
Ardour3 has an object called a Processor whose ::run() method
encapsulates all DSP done within Ardour, but nevertheless it would be
more or less impossible to do any kind of optimization that looked
"across" all the Processors in a signal chain (eg. gain, pan, plugins,
etc, etc).

--p

Ciao Maurizio,

I have been looking for a while to LLVM as a possible technology to build a
DSP execution engine, providing the runtime flexibility needed by real-time
interactive DSP applications (like patcher languages), or by plug-in based
processors, and in the same moment
the powerful link time optimizations that such a system can provide.

Such a task is daunting for a single developer project like mine, but it may
become feasible if such engine could be useful for multiple projects, so to
become a community initiative (i am sending this
mail to the Linux Audio Developers mailing list and to the LLVM mailing
list, feel free to send it elsewhere if you find it useful).

It's some time that I actually have a somewhat related idea: when you
use physics-based modeling techniques a crucial point is proper
scheduling of operations, which has NOT to be done on a "processing
object" (i.e., plugin or whatever) basis, but by considering
dependencies among single input/output expressions even inside
"processing objects" (this may sound unclear, I know).

This means that in practice, if you deal with compiled code (a.k.a.
plugins), you just can't reasonably do it, as opposed to specialized
tools, and in particular special audio programming languages that can
either be interpreted or compiled to some sort of "understandable"
bytecode that preserves input/output time relationships (i.e., delays)
and then run in a virtual machine that solves the scheduling issue.

An example of such specialized tools is the audio DSP language I wrote
called Permafrost. Here, the scheduling issue is solved when compiling
the Permafrost source code to LV2 plugin source (C code and Turtle/RDF
metadata), but this is a one way operation - you can't use the output
plugin code to build a new plugin taking into account these issues,
whether at runtime or not.

My idea was to define an intermediate DSP bytecode, hopefully also
capable of including metadata, and a virtual machine that schedules,
optimizes and runs the whole chain/graph.

A stupid example of what can be achieved with such a thing is the
following: suppose you have a physics-based simulator of a tube
amplifier that allows you to "plug" a physics-based loudspeaker model
into it (if you are into this kind of stuff, suppose it is WDF-based)
- you would be able to change such loudspeaker model while the system
is running and have it all correctly scheduled, optimized and
perfectly working.

On the other hand, I believe you are aiming for something different,
hence I do suggest you (as Paul did) to contact FAUST developers,
since they're pretty much into this kind of stuff, especially when it
comes to LLVM.

BR,

Stefano

Hi, and thanks for the feedback.

My own gut feeling on this is that you'd be better off figuring out
how to do this in the context of Faust, which already does a important
subset of what you are describing, though notably without (I think)
the LLVM part.

Surely a contribution to the discussion by the Faust people would
be welcome; and even more than to the discussion :->. By the way, they
do something with LLVM, but i do not know exactly what (Faust is included
in the list of projects using LLVM).

I think the interest, if any, would be in having a *low* level
library generic enough to be used by multiple applications, so
to factor the work. It may be that an existing project have internally
a large part of the needed code, though.

Note also that hosts which run plugins at the level of LADSPA/LV2,
VST, AU, DSSI etc are unlikely to be easy candidates for any
cross-plugin optimization.

I was actually talking about two different context, reusing existing
binaries, and having cross plugin optimisations for native plugins.

But, if the plugins are compiled with a different compilation chain
(using the LLVM compiler producing LLVM byte code, so introducing a different
binary format for them), than cross plugin optimisation is possible also for
existing plugins like a LADSPA plugin.

Would that be worthwhile ? Difficult to know for sure, but it is an interesting
subject to explore.

Maurizio

Maurizio De Cecco wrote:

Surely a contribution to the discussion by the Faust people would
be welcome; and even more than to the discussion :->. By the way, they
do something with LLVM, but i do not know exactly what (Faust is included
in the list of projects using LLVM).

Thanks to Stephane Letz' work, Faust has had an LLVM backend for some
time. (You need to get the faust2 branch from the Faust git repo for
that, however.) Faust2 supports the creation of both LLVM assembler and
bitcode. The latter can be loaded and executed directly by other LLVM
clients (such as my own Pure, which also sports inlining of Faust
programs in Pure scripts now).

Albert