Email list just for front end developers?

Howdy!

I've noticed that requests for help by people like me who are
developing front ends are often lost in the noise in the llvm-dev
list. Although LLVM is a wonderful and very productive framework, I've
found that getting help on my front end has been very difficult, and
learning necessary detail about the APIs and the like is often very,
very challenging.

Perhaps it would be good to set up a mailing list just for front end
developers to provide mutual assistance to one another? That way we
would not have to bother people working on optimizers or the back end
with our stupid beginner questions, but there would still be a place
to go to get assistance.

It might also be nice to have a Wiki. I've learned a bunch of things
about how to use the APIs over the last weeks as I've been fighting
with the code, but I don't really have any good place to share the
information I've learned with other people, so others can't learn
work to figure out, and it would be nice to document them so other
people don't have to do the same thing.

Perry

I'm not sure if by front-end you mean like programming language side of
things, but have you seen cfe-dev?
http://lists.llvm.org/pipermail/cfe-dev/

> Howdy!
>
> I've noticed that requests for help by people like me who are
> developing front ends are often lost in the noise in the llvm-dev
> list. Although LLVM is a wonderful and very productive framework,
> I've found that getting help on my front end has been very
> difficult, and learning necessary detail about the APIs and the
> like is often very, very challenging.
>
> Perhaps it would be good to set up a mailing list just for front
> end developers to provide mutual assistance to one another?

I'm not sure if by front-end you mean like programming language
side of things,

A front end is a translator from some programming language
(everything from Julia to C++ to Haskell to Rust to dozens of others)
into the LLVM IR.

but have you seen cfe-dev?
http://lists.llvm.org/pipermail/cfe-dev/

That's specific to the clang front end.

There are a lot of people out there writing front ends for other
programming languages than C and C++ using LLVM (as I'm sure everyone
is well aware). I'm writing one myself right now as part of my
research.

People working on front ends typically don't really know everything
about the innards of LLVM, and thanks to LLVM's very nicely designed
architecture, we don't need to, we mostly need to know how to
generate IR and hook up to the rest of the system. We're more like
users than developers of LLVM itself, and yet our questions tend to
be quite technical since much of the API is not as well documented as
one might like.

Having a place dedicated to mutual aid among front end developers
(and of course assistance from anyone else who is willing to answer
questions) would be nice.

Perry

I referred you to that list in speaking from experience. cfe-dev may have
started life as a c-centric list, but that's where the experts are and
where I'd recommend you to direct your questions. (If you don't believe me
look at the list archive and you'll see similar questions from time to time)

> Having a place dedicated to mutual aid among front end developers
> (and of course assistance from anyone else who is willing to
> answer questions) would be nice.
>

I referred you to that list in speaking from experience. cfe-dev
may have started life as a c-centric list, but that's where the
experts are and where I'd recommend you to direct your questions.

I am a subscriber to that list and am familiar with it. Almost all
the traffic is Clang specific.

(If you don't believe me look at the list archive and you'll see
similar questions from time to time)

Yes, from time to time, just as from time to time people ask
questions here. It is somewhat difficult getting attention for
such questions amidst the other traffic. It would be nice to have a
list dedicated to the purpose. You need not subscribe if such a list
does not serve your needs. It would, however, likely serve _my_
needs, and I suspect it would serve other people's needs as well.

Perry

You don't seem to be listening, but one last try.

If you have a FE related question - Ask on cfe-dev and if you don't get a
response I'd be surprised. Additionally, if you don't get a response from
there maybe then follow-up with your idea of another list. What you're
describing would be extremely low traffic and I personally doubt you'd get
as much expertise interested in order to keep it viable.

Pardon the length of this reply, but it seemed useful to be explicit
about my concerns.

You don't seem to be listening, but one last try.

I listened. I respectfully disagree.

If you have a FE related question - Ask on cfe-dev and if you don't
get a response I'd be surprised.

I would not be surprised.

I'll be straightforward. It is very hard to get answers to questions
on any of the lists about simple front end stuff, especially stupid
newbie questions. It is also hard to get answers to simple questions
in the IRC chat room. (I've tried, believe me.)

I can understand why. It isn't anyone's fault, and it isn't
unexpected. These questions aren't very interesting to people who are
deeply involved in the Clang infrastructure or the LLVM optimization
infrastructure, they're probably quite repetitive, and I'm sure that
sometimes they're annoying because they're so obvious and basic to
people who live and breathe LLVM. That's entirely understandable.

However, there are many people out there, maybe even a lot more than
are developing Clang in fact (though it is hard to know), who are
developers using LLVM for other languages.

Yes, I agree there are more _users_ of Clang than there are of Rust
and Julia and Rubymotion and Crystal and Clasp and the dozens of
others out there. After all, Clang is Apple's compiler, so there are
huge numbers of users of Clang. However, in terms of people
_developing_ other front ends, I get the impression there are a very
large number of us, because, frankly, right now LLVM is the most
rational choice for any compiler developer, and the world has a lot of
compilers under development. (Indeed, it has more than it used to
partially because LLVM exists.)

It makes sense to have a place for us to talk to each other where
we're the dominant traffic, where our questions for each other aren't
concealed amidst a very large amount of traffic on Clang's internals
(or the rest of LLVM's), and where and our questions unrelated to local
concerns are not viewed as an annoyance.

Again, I suspect front end developers will generally be more
sympathetic to each other's problems and questions than people who are
worried much more about Clang's internals or the backend internals.

Additionally, if you don't get a response

It isn't a question of "a" response. I myself have a new question
almost every day. After one sees what happens when people ask
questions, one gets intimidated and does not wish to continue.

Here's an example. This post about a month ago from someone having a
great deal of trouble with very basic things got not a single answer
in response:

http://lists.llvm.org/pipermail/llvm-dev/2017-March/111526.html

I think that gentleman's situation is very typical. I've had questions
almost identical to his, too. I've been reduced to reading source code
instead of documentation, painfully, and making very slow progress
because of it.

Again, I don't blame anyone for this state of affairs. LLVM is open
source software, and no one has any duty to answer my questions, write
documentation for me, etc. I cannot reasonably ask that people who are
working hard on a new processor back end or new optimizations or what
have you take time out to help me. It isn't their job.

I would just like a place that makes self help among other people in
my position and that of the gentleman whose query I link to above more
likely to succeed.

from there maybe then follow-up with your idea of another list.

I understand that it may be too much trouble to ask the LLVM project
to set up another mailman mailing list.

If that is the case, given that I have my own mailman setup, perhaps
I'll just create the list on my own machine and advertise it, and
perhaps create a Wiki. It would be unofficial so fewer people would
find it, which might doom it, but as I've indicated, I would like to
have such a thing, and I suspect others would, too, so perhaps that's
the route to take.

I don't want to bother people associated with the project overly much,
as frankly I'd prefer that all of you spend your time making LLVM even
better, and if the project doesn't want to create such a list I won't
push, I'll just do it on my own if it continues to seem important to me.

What you're describing would be extremely low traffic

I very, very much doubt that will be the case.

I understand that this isn't obvious to those who are very familiar
with the way LLVM works, but again, it is challenging getting started
with LLVM once one is past the tutorial. One has endless questions
about details the tutorial doesn't cover, and the documentation sadly
doesn't help with them. Even when one is past that stage, one keeps
encountering issues one would like to ask about.

There are also a lot of people out there working on LLVM based front
end projects, far more even than LLVM web site indicates. There are
also whole classes of undergrads using LLVM in compiler courses and I
suspect they have questions too.

And, once again, most of us learn very quickly that simple newbie
questions generally don't get answered by busy people who have other
interests. (And again, this is understandable.)

That's fine, I understand why it is the case. I don't want to bother
others who have other concerns. I just want a place to ask
questions. If such a list existed, I would be able to contribute quite
constructively -- I've learned the answers to many of my own questions
"the hard way" by having to read the sources, and I could help others
to avoid having to do the same thing with things I understand. I would
also probably feel less put off asking simple questions on such a
list. I bet one of the developers of Rust or the JavaScript JIT or the
Common Lisp front end or Crystal or Julia or any of the literally
dozens of others would probably be happy to answer my questions,
having had the same steep learning curve themselves.

and I personally doubt you'd get as much expertise interested in
order to keep it viable.

Then it will fail. That's a risk with any new mailing list. Luckily,
the cost of mailing lists is very low, so the only risk is that the
list gets no traffic. If that happens, well, so be it.

Perry

Hi,

It makes sense to have a place for us to talk to each other where
we're the dominant traffic, where our questions for each other aren't
concealed amidst a very large amount of traffic on Clang's internals
(or the rest of LLVM's), and where and our questions unrelated to local
concerns are not viewed as an annoyance.

Why not just filter traffic on llvm-dev with "emails that haven't been
replied to within a week"? Right now llvm-dev is, say, ~220 emails a
week, so after that filter you should a small enough set of threads
left which you can manually inspect.

I don't think people will be annoyed by frontend specific questions on
llvm-dev unless there is a very high number of them (unlikely), at
which point we can //maybe// consider splitting out a new mailing list
(but see below).

I would just like a place that makes self help among other people in
my position and that of the gentleman whose query I link to above more
likely to succeed.

I agree that we do not do the best possible job in answering beginner
questions, but is there something preventing you from changing that
trend on llvm-dev, and making llvm-dev be that place where beginner
questions are reliably answered?

and I personally doubt you'd get as much expertise interested in
order to keep it viable.

Then it will fail. That's a risk with any new mailing list. Luckily,
the cost of mailing lists is very low, so the only risk is that the
list gets no traffic. If that happens, well, so be it.

There are other risks too - fragmentation (I have to sign up for yet
another mailing list), discoverability (at this point many people
already know about llvm-dev, how do they find out about the
llvm-frontend-dev?).

-- Sanjoy

Hi,

> It makes sense to have a place for us to talk to each other where
> we're the dominant traffic, where our questions for each other
> aren't concealed amidst a very large amount of traffic on Clang's
> internals (or the rest of LLVM's), and where and our questions
> unrelated to local concerns are not viewed as an annoyance.

Why not just filter traffic on llvm-dev with "emails that haven't
been replied to within a week"? Right now llvm-dev is, say, ~220
emails a week, so after that filter you should a small enough set
of threads left which you can manually inspect.

I'm not sure that will work well, because people won't usually wait a
week to get their questions before moving on. If you're actively
working full time on a project, you probably want answers in much less
time.

Usually a question is inspired because you're trying to do something
immediately (say, in my own case, when I was working on real constants
and realized I couldn't find much documentation for what formats
APFloat understands when you pass the constructor a string), and
laying tools down for that long isn't an option. If no one answers in
less than a day, you probably have to go and slog through the
implementation on your own, which is of course sub-optimal.

I don't think people will be annoyed by frontend specific questions
on llvm-dev unless there is a very high number of them (unlikely),
at which point we can //maybe// consider splitting out a new
mailing list (but see below).

I doubt that there ever would be a lot of them as it stands, because I
think people find that they don't get answered and that they feel
weird about asking more questions of the same form. After a while a
culture gets entrenched.

Note that it isn't even something weird or inexplicable, I understand
why this sort of thing happens. llvm-dev serves a vital need already,
which is letting the llvm developers communicate with each other on
technical matters. Unusual traffic that isn't really any one person's
area gets less mental space.

So you get a dual problem either here or on the cfe lists, which is
that the normal traffic isn't traffic you understand (or care about)
and that the traffic that you are interested in isn't of much interest
to the overwhelming bulk of the list.

> I would just like a place that makes self help among other people
> in my position and that of the gentleman whose query I link to
> above more likely to succeed.

I agree that we do not do the best possible job in answering
beginner questions, but is there something preventing you from
changing that trend on llvm-dev, and making llvm-dev be that place
where beginner questions are reliably answered?

Well, partially, I think, that it is hard to change a culture,
partially that llvm-dev isn't geared towards it (I myself don't
understand much of the traffic since I don't know anything past
IRBuilder well and am unlikely to dive into that soon, partially
because LLVM does such a good job at separating concerns that I don't
_have_ to understand the middle and back end work.)

However, if people think that just making a more determined effort to
answer newbie questions here is the best way to go, we can try, and
I'll certainly help, since I've been learning what many of them are
the hard way.

Again, though, I didn't even notice that message I posted asking for
help until long after that guy probably didn't care any more. It
probably remains hard answer such questions in an environment where
there's an impedance mismatch between the topics. As I noted, most of
the traffic here isn't even about stuff I understand well, I'm here
(and on the Clang lists) only because they're the closest thing to
what I actually _do_ need.

And again, the impedance mismatch is unlikely to be malice against
newcomers or any such thing. I think it's because, de facto, this is
where people discuss middle and back end development, and it is the
only good channel for that, and there's apparently a lot of that to
discuss.

>> and I personally doubt you'd get as much expertise interested in
>> order to keep it viable.
>
> Then it will fail. That's a risk with any new mailing list.
> Luckily, the cost of mailing lists is very low, so the only risk
> is that the list gets no traffic. If that happens, well, so be
> it.

There are other risks too - fragmentation (I have to sign up for yet
another mailing list), discoverability (at this point many people
already know about llvm-dev, how do they find out about the
llvm-frontend-dev?).

Fragmentation is a real problem, especially since the people who are
often best equipped to answer the questions are people working on the
middle and back end. That could absolutely make this fail
hard. However, as it stands, such people often don't have much time to
answer anyway, or so it seems. To be sure, though, such an experiment
is most likely to fail for that reason.

I'm less worried about discoverability. One good post to the
announcements list and bit on LLVM weekly, and having it documented in
the normal places (lists.llvm.org, the various resources web pages)
should be enough.

And, if it fails, one can always turn it off, merge it back in, and
try something else. The cost is low.

But again, that said, if people really think that this is the right
place for such things and not a new list, that can probably be given a
good faith effort. My perception was that it probably wouldn't work
well because this list is de facto for something else, but I'm willing
to try if that's a strong consensus.

*That* said, I still think a Wiki would also be nice. When one finds a
hard-fought answer to a question, putting that answer somewhere beats
having the information only in people's heads. Yes, for sure, the
"right thing" is to submit a documentation patch, but there's a high
barrier involved for an outsider to do that, and putting the
information somewhere is better than having it nowhere. The only real
problem with a Wiki is it requires policing, they acquire spam at a
prodigious rate.

Perry

Hi Perry,

Hi,

> It makes sense to have a place for us to talk to each other where
> we're the dominant traffic, where our questions for each other
> aren't concealed amidst a very large amount of traffic on Clang's
> internals (or the rest of LLVM's), and where and our questions
> unrelated to local concerns are not viewed as an annoyance.

Why not just filter traffic on llvm-dev with "emails that haven't
been replied to within a week"? Right now llvm-dev is, say, ~220
emails a week, so after that filter you should a small enough set
of threads left which you can manually inspect.

I'm not sure that will work well, because people won't usually wait a
week to get their questions before moving on. If you're actively
working full time on a project, you probably want answers in much less
time.

Sure s/week/day/ then. :slight_smile: You're also welcome to be more active on
the existing IRC channel.

I don't think people will be annoyed by frontend specific questions
on llvm-dev unless there is a very high number of them (unlikely),
at which point we can //maybe// consider splitting out a new
mailing list (but see below).

So you get a dual problem either here or on the cfe lists, which is
that the normal traffic isn't traffic you understand (or care about)
and that the traffic that you are interested in isn't of much interest
to the overwhelming bulk of the list.

I don't think the latter is a problem. Despite being an active user
of LLVM, I also don't care about a non-trivial fraction of the threads
on llvm-dev. A few more won't hurt.

But again, that said, if people really think that this is the right
place for such things and not a new list, that can probably be given a
good faith effort. My perception was that it probably wouldn't work
well because this list is de facto for something else, but I'm willing
to try if that's a strong consensus.

Let me put it this way -- I'm pretty sure if you take it on yourself
to answer beginner questions on llvm-dev promptly, nobody will
*complain*. In fact, you're going to make some folks happy. :slight_smile:

*That* said, I still think a Wiki would also be nice. When one finds a
hard-fought answer to a question, putting that answer somewhere beats
having the information only in people's heads. Yes, for sure, the
"right thing" is to submit a documentation patch, but there's a high
barrier involved for an outsider to do that, and putting the
information somewhere is better than having it nowhere. The only real
problem with a Wiki is it requires policing, they acquire spam at a
prodigious rate.

I think a policed wiki is going to be more cumbersome than checked in
documentation. Moreover, we already have precedent here:
http://llvm.org/docs/Frontend/PerformanceTips.html.

-- Sanjoy

> I'm not sure that will work well, because people won't usually
> wait a week to get their questions before moving on. If you're
> actively working full time on a project, you probably want
> answers in much less time.

Sure s/week/day/ then. :slight_smile: You're also welcome to be more active on
the existing IRC channel.

I've tried being active on (and asking questions on) the IRC channel.
There doesn't seem to be much of a culture of answering questions
there either, though. :frowning:

> But again, that said, if people really think that this is the
> right place for such things and not a new list, that can probably
> be given a good faith effort. My perception was that it probably
> wouldn't work well because this list is de facto for something
> else, but I'm willing to try if that's a strong consensus.

Let me put it this way -- I'm pretty sure if you take it on yourself
to answer beginner questions on llvm-dev promptly, nobody will
*complain*. In fact, you're going to make some folks happy. :slight_smile:

Sure, but who will answer mine? Part of the reason I asked about
this was selfish, as I'd like access to other people in the same
situation as me who might know answers. (I don't think the people here
now will want to answer more questions, and many of my questions
doubtless border on inanely stupid but for the fact that it is hard
to find answers in the docs.)

That said, I'm willing to give it a try, as an experiment. How long
should we attempt it for? If that's reasonably agreed to, I'll
personally give it a good faith effort myself during the period. (I'm
not sure I will be able to help with more than a fraction of the
questions, being very new at this myself, but I'll do my best.)

If we are to try this for real, we also might want to make some sort
of effort to encourage people to ask more questions here, or it is
likely that the current habit of not asking them will persist.

Perry

Hi Perry,

Let me put it this way -- I'm pretty sure if you take it on yourself
to answer beginner questions on llvm-dev promptly, nobody will
*complain*. In fact, you're going to make some folks happy. :slight_smile:

Sure, but who will answer mine? Part of the reason I asked about
this was selfish, as I'd like access to other people in the same
situation as me who might know answers. (I don't think the people here
now will want to answer more questions, and many of my questions
doubtless border on inanely stupid but for the fact that it is hard
to find answers in the docs.)

I don't see how people who don't have time to answer on llvm-dev will
find time to answer on some other mailing list.

That said, I'm willing to give it a try, as an experiment. How long
should we attempt it for? If that's reasonably agreed to, I'll

Depends on how much time you have -- that's the limiting resource here.

However, I liked your idea of maintaining a wiki-like thing. Both
http://llvm.org/docs/Frontend/PerformanceTips.html and
http://llvm.org/docs/tutorial/ will be good additions there.

personally give it a good faith effort myself during the period. (I'm
not sure I will be able to help with more than a fraction of the
questions, being very new at this myself, but I'll do my best.)

Thanks!
-- Sanjoy

Hi Perry,

>> Let me put it this way -- I'm pretty sure if you take it on
>> yourself to answer beginner questions on llvm-dev promptly,
>> nobody will *complain*. In fact, you're going to make some
>> folks happy. :slight_smile:
>
> Sure, but who will answer mine? Part of the reason I asked about
> this was selfish, as I'd like access to other people in the same
> situation as me who might know answers. (I don't think the people
> here now will want to answer more questions, and many of my
> questions doubtless border on inanely stupid but for the fact
> that it is hard to find answers in the docs.)

I don't see how people who don't have time to answer on llvm-dev
will find time to answer on some other mailing list.

My belief was they would be different populations. That is why there
would be a separate list. The other list would attract mostly front
end devs. I would expect front end devs would be both more interested
in a list that discusses mostly things they care about (including IR
changes they need to be aware of) and more interested in mutual
assistance.

But never mind, it seems that I'm the only person who likes that idea,
so we're ignoring that possibility for the moment.

> That said, I'm willing to give it a try, as an experiment. How
> long should we attempt it for? If that's reasonably agreed to,
> I'll

Depends on how much time you have -- that's the limiting resource
here.

By time I meant the question of how long one tries a strategy before
deciding to look for new strategies. :slight_smile: Given that few questions about
front end things arrive on the -dev list, trying to help answering
them, for the moment, will only require that I attend more closely to
the postings, and that other people make a bit of an effort to answer
"dumb questions" as well.

Hopefully that will simply work. My skepticism comes from the feeling
that if it was really going to work, it would have already, but
there's no reason not to give it a good faith try.

On the wider issue:

LLVM has been transformative. There are now a lot more people
experimenting with compiling languages than ever before (including
me!) because LLVM removes a lot of the pain from constructing
everything past the IR.

However, until I started working with it personally, I didn't
understand how much work it was to learn how to use it, and much of
that work is because there is a very big gap between the programmers
guide + the tutorial (which makes it all look like you can learn the
whole thing in an afternoon) and what you actually need to figure out
to get things done. Much of what you need you have to puzzle out
painstakingly on your own, trying to make sense of the doxygen docs
and reading code, because there's less written down than one would
hope. Even the IR language reference, which is seemingly complete at
first glance, has some big gaps in it.

Given good enough resources, though, even more people could be using
LLVM to create compilers, which would be a great thing. And,
selfishly, given better resources, my own work would be easier. :slight_smile:
Thus my overall interest.

However, I liked your idea of maintaining a wiki-like thing. Both
http://llvm.org/docs/Frontend/PerformanceTips.html and
http://llvm.org/docs/tutorial/ will be good additions there.

Parts of the programmer's manual
http://llvm.org/docs/ProgrammersManual.html are also essential, though
for a front end developer, especially one using the C API, not
everything is something you want to know, and it isn't initially
obvious when you need to read closely.

Anyway, given that you would prefer for it not to be an actual wiki,
what would be lower barrier to entry and Wiki-ish?

(A big issue with the current workflow, I think, is that for outsiders
submitting documentation updates the process is very heavyweight. Even
the use of svn vs. git increases the amount of work for casual users.
It's just not worth it as it stands for someone to say "hey, please
fix just this one sentence", especially if you've never contributed to
LLVM before. However, often even single sentence edits improve
comprehension. Having a low barrier for small fixes is one of the
reasons Wikipedia grew so fast. Of course, it is also the reason that
policing Wikipedia takes huge effort, so there's a tradeoff.)

Perry

2nd try to sent this email. ^^

FWIW, I think the “modern” way of doing any of this is with any one of the following services (that don’t need to be owned/maintained by the LLVM Foundation):

  • Stack Overflow (maybe if it gets large enough, get a “vertical” for just LLVM usage)
  • Reddit (if there isn’t an /r/llvm yet, that might be a good “community” place for less llvm-dev worthy material)
  • Slack (if there isn’t one yet, maybe an llvm-users slack instance for more persistent discussions)

I suspect none of this has to get the blessing of the llvm-dev@ members nor the foundation, and if there indeed are a lot of people that might benefit from these resources, then you would have started that community for the users.

Hope this helps.

Cheers

> Hi Perry,
>
> >> Let me put it this way -- I'm pretty sure if you take it on
> >> yourself to answer beginner questions on llvm-dev promptly,
> >> nobody will *complain*. In fact, you're going to make some
> >> folks happy. :slight_smile:
> >
> > Sure, but who will answer mine? Part of the reason I asked about
> > this was selfish, as I'd like access to other people in the same
> > situation as me who might know answers. (I don't think the people
> > here now will want to answer more questions, and many of my
> > questions doubtless border on inanely stupid but for the fact
> > that it is hard to find answers in the docs.)
>
> I don't see how people who don't have time to answer on llvm-dev
> will find time to answer on some other mailing list.

My belief was they would be different populations. That is why there
would be a separate list. The other list would attract mostly front
end devs. I would expect front end devs would be both more interested
in a list that discusses mostly things they care about (including IR
changes they need to be aware of) and more interested in mutual
assistance.

But never mind, it seems that I'm the only person who likes that idea,
so we're ignoring that possibility for the moment.

> > That said, I'm willing to give it a try, as an experiment. How
> > long should we attempt it for? If that's reasonably agreed to,
> > I'll
>
> Depends on how much time you have -- that's the limiting resource
> here.

By time I meant the question of how long one tries a strategy before
deciding to look for new strategies. :slight_smile: Given that few questions about
front end things arrive on the -dev list, trying to help answering
them, for the moment, will only require that I attend more closely to
the postings, and that other people make a bit of an effort to answer
"dumb questions" as well.

Hopefully that will simply work. My skepticism comes from the feeling
that if it was really going to work, it would have already, but
there's no reason not to give it a good faith try.

On the wider issue:

LLVM has been transformative. There are now a lot more people
experimenting with compiling languages than ever before (including
me!) because LLVM removes a lot of the pain from constructing
everything past the IR.

However, until I started working with it personally, I didn't
understand how much work it was to learn how to use it, and much of
that work is because there is a very big gap between the programmers
guide + the tutorial (which makes it all look like you can learn the
whole thing in an afternoon) and what you actually need to figure out
to get things done. Much of what you need you have to puzzle out
painstakingly on your own, trying to make sense of the doxygen docs
and reading code, because there's less written down than one would
hope. Even the IR language reference, which is seemingly complete at
first glance, has some big gaps in it.

Given good enough resources, though, even more people could be using
LLVM to create compilers, which would be a great thing. And,
selfishly, given better resources, my own work would be easier. :slight_smile:
Thus my overall interest.

I'm still unclear about what sorts of questions/discussion this audience is
interested in. Can you provide a few examples of questions that might be
asked, or links to discussions/blog posts on the net that exemplify the
type of thing that you would want to happen in the "frontend authors using
LLVM" medium you are thinking of?

For example, looking at the post you linked (
http://lists.llvm.org/pipermail/llvm-dev/2017-March/111526.html the author
of which is in this thread I think, hi!), there's not much LLVM-specific
about it. In order to generate LLVM IR, it is necessary to understand its
semantics. The semantics are mostly C-like, and so the answer to questions
like those in the post:

- How to assign a text to a string variable?
- How to access for example the 4th element in a heap-allocated array?
- How to create local variables which are not part of the global scope?
- How to create complex structures like classes and objects?

is basically "how would you do it in C"? For example, to assign text to a
"string variable", you first have to have a struct that represents you
string type, then you do some C-level code to manipulate that data
structure. For a class with inheritance, you would put a vptr in your
structs and emit vtables. Etc. Then it's just a matter of writing some C
code and running it through Clang to see how Clang lowers it to LLVM IR. It
shouldn't be too hard to then trace back from the textual IR to the API's
(at worst, requiring a small amount of trial and error, at worst looking at
small parts of Clang's source code after grepping for AllocaInst or
whatever).

Something like
https://f0rki.gitbooks.io/mapping-high-level-constructs-to-llvm-ir/content/
might be useful too, though you could get mostly the same effect by looking
at Clang's output.

For some of the specifics of LLVM IR's SSA form (like phi nodes) working
through the Kaleidoscope tutorial should give a feel for how to do it. (for
an intro, a video like https://www.youtube.com/watch?v=haQ2cijhvhE by our
own Chandler Carruth might be useful).

However, from your other posts, you say something like:
"People working on front ends typically don't really know everything
about the innards of LLVM, and thanks to LLVM's very nicely designed
architecture, we don't need to, we mostly need to know how to
generate IR and hook up to the rest of the system."

So it sounds like you're more interested in API usage like IRBuilder,
setting up codegen, setting up MC, configuring pass pipelines, etc.? I.e.
you have a clear idea about the "big picture" and what you are trying to
do, but want to know how that is done in LLVM (e.g. "how do I create a phi
node?"). This is a very different kind of question from the ones above.

On the other hand, "hook it up to the rest of the system" might be
referring to general toolchain knowledge (which is again not LLVM specifc).
For example object file semantics (i.e. the semantics of what comes out the
LLVM backends; there's a lot more to it than just a single blob of machine
code) and how that interacts with the rest of the system (e.g. how a malloc
declaration in LLVM IR gets actually resolved and called at runtime).

I'd really like to hear your feedback. I think that historically, Clang has
been the dominant frontend and so there hasn't been much impetus for
providing certain kinds of documention, but as you mention upthread, there
is now a quite large tail of other frontends that are developed outside of
the LLVM community, so that might have to change.

-- Sean Silva

So in writing this, I want to emphasize, in advance, that I'm grateful
to the LLVM team for having created LLVM, I do not feel I am in any
way entitled to anyone else's help, etc. The purpose of my original
post was just to suggest that there be a way for people who are
struggling with some parts of the APIs to find each other and answer
each other's questions. If any of the following sounds like I'm
demanding anything, that's not my intent. This may be longer than it
should be, too.

I'm still unclear about what sorts of questions/discussion this
audience is interested in. Can you provide a few examples of
questions that might be asked, or links to discussions/blog posts
on the net that exemplify the type of thing that you would want to
happen in the "frontend authors using LLVM" medium you are thinking
of?

For example, looking at the post you linked (
http://lists.llvm.org/pipermail/llvm-dev/2017-March/111526.html the
author of which is in this thread I think, hi!), there's not much
LLVM-specific about it. In order to generate LLVM IR, it is
necessary to understand its semantics. The semantics are mostly
C-like, and so the answer to questions like those in the post:

- How to assign a text to a string variable?
- How to access for example the 4th element in a heap-allocated
array?
- How to create local variables which are not part of the global
scope?
- How to create complex structures like classes and objects?

is basically "how would you do it in C"?

I think you're missing the import of his questions. This isn't "how
you would do it in C". He's asking effectively how to use the
IRBuilder APIs to do these things, not what in principle you want the
machine to do or what you might write into a file of text form IR if
you were bypassing the IRBuilder.

The "how do I construct a string constant in the API" one cost me
hours to figure out until someone on IRC took pity on me, which was
unusual, and steered me to the right call (CreateGlobalString).

Now I know what you're thinking. "Why didn't you just look for
something like CreateGlobalString to begin with?" And the answer is,
because I had no idea what I was looking for, what form it would take,
or what it was called, and googling at random wasn't helping. I knew
what sort of IR output I wanted but I didn't know how to generate it
in the API.

And yes, a naive newcomer can literally spend hours trying to figure
out that just one call they wanted was CreateGlobalString. Maybe I
missed the part of the extant documentation or the tutorial that
pointed there, but that's kind of the point, isn't it? If it was
there, I didn't find it, and I needed to ask for help.

It would have been nice to have a place where I could have
dropped the question and no one would have thought twice about just
answering.

And thus, my desire for a place for mutual aid for people who are
figuring out front end stuff or who have already figured it out and
are willing to help others.

For example, to assign text to a "string variable", you first have
to have a struct that represents you string type, then you do some
C-level code to manipulate that data structure.

That's not what he's asking, or at least I don't think he is. He's
asking about how to generate the right IR. He probably was looking for
CreateGlobalString and how to put it together with other stuff. His
other messages made that seem clear to someone else who had been
struggling recently with the same issues,.

For some of the specifics of LLVM IR's SSA form (like phi nodes)
working through the Kaleidoscope tutorial should give a feel for
how to do it.

A feel isn't enough to write code that compiles and does the right
stuff. It gives you a tantalizing hint of what you want, but you need
more than that. The tutorial makes it all look really easy, but it
skips even a lot of details about what it is doing, and then you start
to ask questions that go slightly past the edge of the tutorial and
the answers _really_ aren't obvious.

Here's just one a of a million examples: in a real compiler you don't
want to just parse a number into a machine int or double or what have
you, you want to use arbitrary precision stuff that's machine
independent, and of course, LLVM has that because it needs it.

Unluckily, even once you know it is there, it takes a while to figure
out what you need to use it, and there's very little documentation at
all. There was a point where I wanted to just punt and let the APFloat
API parse a string from my front end's scanner (after all, why
duplicate that code and less well?), only there's no documentation on
what sorts of float formats it parses. If someone could have answered
that, or if there had been a paragraph or two in the doxygen output,
it would have saved me lots of time reading code. As it was, the best
someone had to suggest to me was read the tests, so I went off and
slogged through. (And yah, it turns out that it handles a variety of
standard C-like formats including hex floats, which was something I
wanted to figure out but couldn't just read in the doxygen docs
because it isn't there.)

So yah, you get a feel for what you want from the tutorial, but a feel
isn't enough to write a compiler for something non-trivial. The
programmer's guide helps a bit more, but you're quickly off the road
and feeling stuff out.

Note again that I'm not in any way suggesting that anyone should feel
obligated to help others. No one is! I have no claim on anyone else's
time, and again, thank all of you for creating LLVM in the first
place. All I was asking for was just a place for those who are willing
to help each other over the hump to contact each other. Puzzling
things out by reading the test cases for a piece of compiler
middle-end machinery isn't the most fun way to learn an API, and it is
slow, too.

However, from your other posts, you say something like:
"People working on front ends typically don't really know everything
about the innards of LLVM, and thanks to LLVM's very nicely designed
architecture, we don't need to, we mostly need to know how to
generate IR and hook up to the rest of the system."

So it sounds like you're more interested in API usage like
IRBuilder, setting up codegen, setting up MC, configuring pass
pipelines, etc.? I.e. you have a clear idea about the "big picture"
and what you are trying to do, but want to know how that is done in
LLVM (e.g. "how do I create a phi node?"). This is a very different
kind of question from the ones above.

The questions that guy was asking seem to be part and parcel, though I
could be wrong.

There's a big IR Builder API, the C bindings for all this stuff are
poorly documented, the C++ bindings are only slightly better
documented, and dealing with the important bits is not trivial. The
language reference superficially looks quite nice, but then you
realize the only way to actually figure out what IR you need for many
constructs is to write C code and get Clang to dump out the IR that
results. (It took me a while to learn that trick, even though, yes, it
should have been obvious, just like finding CreateGlobalString should
have been obvious only I didn't know I was looking for it.)

However, even once having seen what Clang generates, you still need to
figure out what to put into IRBuilder to get the same results. Sure,
that's probably somewhere in the innards of Clang, but try finding
it. It's not trivial, and without much documentation it takes a while
to figure out. I really would have like to have just asked someone
about a bunch of stuff instead of deciphering things.

Also take his second question about accessing parts of
arrays. getelementptr is sufficiently weird that there actually _is_ a
decent bit of documentation on it, but it isn't obvious that you're
looking for that unless someone tells you. (Luckily I stumbled on it.)

I get that this all seems really easy if you know it already. To
someone who is coming at this completely naively, without much
understanding of where to dig in, there's a very, very steep learning
curve and little documentation once you're done with the tutorial. A
lot has to be found in blog postings, bits of sample code you happen
to find, reading the implementation, etc.

And I note that even though I'm still at it, I don't yet have my front
end doing very much. There's still a lot to learn. And yah, maybe I'm
not the best at this. Maybe if I was smarter I'd just pick it up from
reading Clang code or something, but I'm slow and middle aged and yet
I have a job to get done anyway. So I keep beating my head against
stuff until I figure it out. But I still wish there were people to ask
questions.

I'd really like to hear your feedback. I think that historically,
Clang has been the dominant frontend and so there hasn't been much
impetus for providing certain kinds of documention, but as you
mention upthread, there is now a quite large tail of other
frontends that are developed outside of the LLVM community, so that
might have to change.

If I had a magic wand, I'd have everything well documented, but that
seems like it isn't going to happen. As a more practical second best,
all I really want is to be in contact with other people who work on
front ends who are sympathetic and remember learning the same thing.

Perry

FWIW, I think the "modern" way of doing any of this is with any one
of the following services (that don't need to be owned/maintained
by the LLVM Foundation):

  - Stack Overflow (maybe if it gets large enough, get a "vertical"
for just LLVM usage)

Unfortunately, stack overflow works best for questions where there
are a lot of people who know the answers and people are willing
to actively log in day after day to answer questions. It works great
for "how do I use SQL to do something" but less well for small
niches. The LLVM questions I've seen posed there seem to usually have
zero or one answers rather than the large number common things get.

  - Reddit (if there isn't an /r/llvm yet, that might be a good
"community" place for less llvm-dev worthy material)

Maybe. I like mailing lists because you don't have to poll them.
Having to log in constantly to a service to see if there is
discussion waiting is a bit of a pain. (I have to admit I've never
been much of a reddit fan in general.)

  - Slack (if there isn't one yet, maybe an llvm-users slack
instance for more persistent discussions)

There is an IRC channel already, though my experience with it as a
help mechanism has been mixed as it seems more geared towards the
middle and back end. And even if people decided this was a good idea,
without the LLVM foundation documenting it somewhere, no one will
find it.

All that said, I might be wrong and these might be the better path.

Perry

I agree. I think the main problem is that there is no useful documentation on the IR builder api and the new passmanager and all the examples don’t use the IR builder api but instead some older api. The only example is clang itself. I finally got the answers i needed from reading and understanding the clang source code.
There are still many areas i am not sure about.
E.g. when using the IR builder, do you need a builder module context per function or per BC file (group of functions).

It also works best for answers that are correct for a long time. Look at StackOverflow’s C++ section and you’ll find a load of questions where the flagged-correct answer is pre-C++11 and there’s a much better answer if you use a current standard. In comparison to LLVM, C++ evolves very slowly and it’s not that uncommon for the correct and best answer now to be entirely wrong and dangerous advice in six months time.

I would be very happy (as someone who has been working on LLVM front ends since 2008) to participate in a front-end developers’ mailing list.

David