Obsolete PTX is NOT completely removed in 3.2 release

Brooks,

Hi Pawel,

PTX already be replaced with NVPTX. However, PTX
subdirectory still sit in lib/Target in 3.2 release. Do
you think update the release tarball is a good idea? Also
could you remove it from the trunk?

Please do not, under no circumstances, change the 3.2
release tarballs at this point. They are mirrored around
the world now with cryptographic hashes and signatures.
Changing them will break things for many people, especially
for an extremely minor thing like an empty directory.

I'm not sure if Pawel's tarball change should be reverted
now as it already caused uproar, so changing it back might
only make matters worse.

The tarballs were changed?

r172208

I finally updated the FreeBSD ports yesterday and today a user
complained about distfile changes. IMO, this revision should
be reverted or all the other BSDs will have to chase checksums
as well.

If you really want to remove the directory, ship a 3.2.1
tarball rather than screwing all the downstream consumers who's
infrastructure exists to detect trojan'd tarballs.

Tarball is signed, it is not trjoan. Your infrastructure should
be able to deal with it?

The FreeBSD ports collection maintains a set of sha256 hashes for
each distfile. The system can deal with them changing, but it's
an inconvenience to port maintainers and users. Even if we did
have infrastructure to verify the signatures we'd still have to
check the contents and would not just trust the signature since
there's always a risk of keys being compromised.

-- Brooks

At first your reaction has puzzled me, but despite a few fire
breathing dragons getting loose and after carefully re-reading
these posts over the weekend I think I understand the underlying
issue.

Simply put we both take this release seriously.

The problem is the release process is not entirely
defined. Which is reflected in question from David Blaikie
(the other post I am quoting here).

Which release process document are you referring to that indicates
that?

Simple answer is the one on the llvm.org website.
But the real answer is, it all depends on how you read it.
Apparently, we read it differently which is not that
surprising as even our well defined compilers have
a bit of a problem with undefined behavior.

So what can we expect from a release process document?
My hope is that we have reached a sort of sequence
point and the ambiguity of the release process
can be resolved despite all the side effects.

David also stated this:

Generally, so far as I know, this kind of post-release change is
unprecedented.

I agree with this conclusion but even more unprecedented
is to a expect timely and problem free release with zero and
I'll repeat again zero involvement from the wider community that
depends on the clang+llvm!
This is how the 3.2 release has happened, just compare
the list of 3.2 to 3.1 binaries and BTW not a single
3.2 binary has been produced with the help of a respective
project, distribution or who ever.

I hear the cries of your wounded infrastructures
and I understand that in the days of forged root certificates
my signature on the tarballs may not guarantee much of a
security.

But why didn't you get involved in the 3.2 release?

We could have resolved all this in the 2 months since
the 3.2 release got rolling. Everybody knew about the
changes for the 3.2 release, simple e-mail to the new
LLVMRM would go a long way to resolve any security or
other release related issues not to mention a bit
of a courtesy.

I am doing this as service to the LLVM community
but frankly I'd rather be chasing my dragons of
probability now.

Paweł

Anton,

Pawel,

We all understand that you're pretty new to release process, etc., but
I think you should understand the implications of your actions.

You just created a lot of harm for really huge pile of users - the
ones who downloads the tarball via some automated build system and
rely on the known good checksum. This includes, but not limited to to
the users of FreeBSD, Gentoo, etc.

Even worse, you did this silently. Without any announcement, with any
e-mail to llvm-dev, etc. And all this at the same time when people do
wide announcement for e.g. 30 mins of restart of buildbot master ro 10
minute restart of web server.

What you did it definitely inacceptable for release manager of such
big project as LLVM. So, may I kindly ask you to revert the tarball
back within next 24 hours an write and entry to New section on the
website. If you want to include the changes name them 3.2.1 or 3.2a
and write an entry on the website. This is the only way how you can
fix all the weird stuff you done in a moment.

All I can say is if you did not get involved in the release but you
depend on it don't cry wolf now.

Before we get any more personal read carefully my reply to Brooks
message.

http://lists.cs.uiuc.edu/pipermail/llvmdev/2013-January/058088.html

Paweł

Brooks,

Hi Pawel,

PTX already be replaced with NVPTX. However, PTX
subdirectory still sit in lib/Target in 3.2 release. Do
you think update the release tarball is a good idea? Also
could you remove it from the trunk?

Please do not, under no circumstances, change the 3.2
release tarballs at this point. They are mirrored around
the world now with cryptographic hashes and signatures.
Changing them will break things for many people, especially
for an extremely minor thing like an empty directory.

I'm not sure if Pawel's tarball change should be reverted
now as it already caused uproar, so changing it back might
only make matters worse.

The tarballs were changed?

r172208

I finally updated the FreeBSD ports yesterday and today a user
complained about distfile changes. IMO, this revision should
be reverted or all the other BSDs will have to chase checksums
as well.

If you really want to remove the directory, ship a 3.2.1
tarball rather than screwing all the downstream consumers who's
infrastructure exists to detect trojan'd tarballs.

Tarball is signed, it is not trjoan. Your infrastructure should
be able to deal with it?

The FreeBSD ports collection maintains a set of sha256 hashes for
each distfile. The system can deal with them changing, but it's
an inconvenience to port maintainers and users. Even if we did
have infrastructure to verify the signatures we'd still have to
check the contents and would not just trust the signature since
there's always a risk of keys being compromised.

-- Brooks

At first your reaction has puzzled me, but despite a few fire
breathing dragons getting loose and after carefully re-reading
these posts over the weekend I think I understand the underlying
issue.

Simply put we both take this release seriously.

The problem is the release process is not entirely
defined. Which is reflected in question from David Blaikie
(the other post I am quoting here).

Which release process document are you referring to that indicates
that?

Simple answer is the one on the llvm.org website.

I'm looking for something a bit more specific. Something that explains
(even ambiguously) the post-release process that might
discuss/imply/support the situation we've ended up in (with a
post-release change to the original tarballs well after the release
announcement, etc).

So far as I can tell the release documentation ends at the point where
the release is finalized/announced/etc.

But the real answer is, it all depends on how you read it.
Apparently, we read it differently which is not that
surprising as even our well defined compilers have
a bit of a problem with undefined behavior.

So what can we expect from a release process document?
My hope is that we have reached a sort of sequence
point and the ambiguity of the release process
can be resolved despite all the side effects.

While that does need to be addressed (& I did bring it up), there's
also the immediate issue at hand to deal with as well.

David also stated this:

Generally, so far as I know, this kind of post-release change is
unprecedented.

I agree with this conclusion but even more unprecedented
is to a expect timely and problem free release with zero and
I'll repeat again zero involvement from the wider community that
depends on the clang+llvm!
This is how the 3.2 release has happened, just compare
the list of 3.2 to 3.1 binaries and BTW not a single
3.2 binary has been produced with the help of a respective
project, distribution or who ever.

I hear the cries of your wounded infrastructures
and I understand that in the days of forged root certificates
my signature on the tarballs may not guarantee much of a
security.

But why didn't you get involved in the 3.2 release?

Presumably: because the process has worked in the past. There would be
no reason to dictate how the release works when experience (&
documentation) seems to support the desired model.

We could have resolved all this in the 2 months since
the 3.2 release got rolling.

Not exactly - it wasn't (& isn't) an unreasonable expectation that
releases are static. That's sort of the definition of a release,
really.

Everybody knew about the changes for the 3.2 release,

What changes are you referring to? I don't know of any changes to the
/release process/ in 3.2, other than the change in RM (you rather than
Bill Wendling). Obviously with any such change there may be
miscommunications/misunderstandings, and that seems like all this is -
and couldn't really have been anticipated by 3rd party consumers of
LLVM.

Pawel,

First, all your help with the 3.2 release is greatly appreciated. I do not think anyone is saying otherwise.

I apologize for the lack of documentation regarding this issue. I do ask that you consult with previous release manager (myself or Bill) to determine what the best course of action is. There is a lot of room to improve our release process, but its a collaborative effort.

You are correct that we do not do "dot" releases. There has long been debate on this and we just don't have the man power to accomplish such a task. This is why we have a relatively short release process.

However, we do not change the tarballs after the release has been "shipped". I remember once we did have a critical issue that caused us to a quick "reship" of a tarball, but we labeled it 3.Xa" to denote the new tarball.

So I ask that the changes be reverted and then hopefully we can just move forward from this misunderstanding.

Thank you again for your hard work here,

Tanya

Tanya,

Pawel,

First, all your help with the 3.2 release is greatly appreciated. I do not think anyone is saying otherwise.

Nothing was said so nothing to worry about.

I apologize for the lack of documentation regarding this issue. I do ask that you consult with previous release manager (myself or Bill) to determine what the best course of action is. There is a lot of room to improve our release process, but its a collaborative effort.

No need to apologize for anything, the were a lot of changes for the
3.2 release and not everything can be captured in process documentation.
Bill spent a lot time giving me pointers on the release which I really
appreciate. However, if I am managing the release then I take full
responsibility for the decisions. At the same time one can reasonably
expect that anybody who is depending on the release would at least
contact release manager before the release happens!

Which brings us to the collaborative effort. I was working with this
assumption but it turned out to be not what really happened.
Frankly there was no collaborative effort!

Not a single, I'll repeat again not a single project or OS that
supposedly critically depends on the clang+llvm distribution
has helped with the release, either testing or building
binaries or just letting me know they exist!

I think this is the real problem.

You are correct that we do not do "dot" releases. There has long been debate on this and we just don't have the man power to accomplish such a task. This is why we have a relatively short release process.

However, we do not change the tarballs after the release has been "shipped". I remember once we did have a critical issue that caused us to a quick "reship" of a tarball, but we labeled it 3.Xa" to denote the new tarball.

So I ask that the changes be reverted and then hopefully we can just move forward from this misunderstanding.

It was not a misunderstanding, I have made a decision to re-ship
based on the information I had at the time. What is interesting
is that the timing of my commit was such that it has got almost
immediately out before I had a chance to send the announcement.

Anyway, both versions of the tarball are available from the the SVN.
I do not see a technical reason of not being able to pull one or the
other. At this point reverting the commit might cause even more "harm"
then good so perhaps we should consult this with wider LLVM community.

Considering that my role as LLVMRM has effectively ended on Dec 21st
I will be happy to "finalize" the release once we reach a consensus.

Thank you again for your hard work here,

Tanya

Paweł

Hi Pawel,

  Sorry for the trouble. At first I think maybe we can upload a new
release tarball not replacing it, sorry I didn't say it in the previous
mail. IMHO, if you have to do something new after the post-release, make
a "dot" release would be better. Perhaps you can write down this
experience to benifit other LLVMRM in the future. :slight_smile:

Regards,
chenwj

Brooks,

>>>>
>>>>
>>>>>
>>>>>
>>>>>> Hi Pawel,
>>>>>>
>>>>>> PTX already be replaced with NVPTX. However, PTX
>>>>>> subdirectory still sit in lib/Target in 3.2 release. Do
>>>>>> you think update the release tarball is a good idea? Also
>>>>>> could you remove it from the trunk?
>>>>>
>>>>> Please do not, under no circumstances, change the 3.2
>>>>> release tarballs at this point. They are mirrored around
>>>>> the world now with cryptographic hashes and signatures.
>>>>> Changing them will break things for many people, especially
>>>>> for an extremely minor thing like an empty directory.
>>>>>
>>>>> I'm not sure if Pawel's tarball change should be reverted
>>>>> now as it already caused uproar, so changing it back might
>>>>> only make matters worse.
>>>>>
>>>>> The tarballs were changed?
>>>>
>>>> r172208
>>>
>>> I finally updated the FreeBSD ports yesterday and today a user
>>> complained about distfile changes. IMO, this revision should
>>> be reverted or all the other BSDs will have to chase checksums
>>> as well.
>>>
>>> If you really want to remove the directory, ship a 3.2.1
>>> tarball rather than screwing all the downstream consumers who's
>>> infrastructure exists to detect trojan'd tarballs.
>>
>> Tarball is signed, it is not trjoan. Your infrastructure should
>> be able to deal with it?
>
> The FreeBSD ports collection maintains a set of sha256 hashes for
> each distfile. The system can deal with them changing, but it's
> an inconvenience to port maintainers and users. Even if we did
> have infrastructure to verify the signatures we'd still have to
> check the contents and would not just trust the signature since
> there's always a risk of keys being compromised.
>
> -- Brooks
>

At first your reaction has puzzled me, but despite a few fire
breathing dragons getting loose and after carefully re-reading
these posts over the weekend I think I understand the underlying
issue.

Simply put we both take this release seriously.

The problem is the release process is not entirely
defined. Which is reflected in question from David Blaikie
(the other post I am quoting here).

> Which release process document are you referring to that indicates
> that?
>

Simple answer is the one on the llvm.org website.
But the real answer is, it all depends on how you read it.
Apparently, we read it differently which is not that
surprising as even our well defined compilers have
a bit of a problem with undefined behavior.

So what can we expect from a release process document?
My hope is that we have reached a sort of sequence
point and the ambiguity of the release process
can be resolved despite all the side effects.

David also stated this:

> Generally, so far as I know, this kind of post-release change is
> unprecedented.

I agree with this conclusion but even more unprecedented
is to a expect timely and problem free release with zero and
I'll repeat again zero involvement from the wider community that
depends on the clang+llvm!
This is how the 3.2 release has happened, just compare
the list of 3.2 to 3.1 binaries and BTW not a single
3.2 binary has been produced with the help of a respective
project, distribution or who ever.

I hear the cries of your wounded infrastructures
and I understand that in the days of forged root certificates
my signature on the tarballs may not guarantee much of a
security.

But why didn't you get involved in the 3.2 release?

We could have resolved all this in the 2 months since
the 3.2 release got rolling. Everybody knew about the
changes for the 3.2 release, simple e-mail to the new
LLVMRM would go a long way to resolve any security or
other release related issues not to mention a bit
of a courtesy.

I don't understand how my lack of involvement with this release has
anything to do with your decision to fix what appears to me to be
a non-problem in the release tarball well after the release and in
the process do something (replacing release tarballs) that is quite
simply NOT DONE to by projects who wish to treat their downstreams
with respect. This is something many projects end up learning due to
the painful way so it's not something you should feel bad about as
long as the project learns the right lesson(s).

If anything I'd be more annoyed if I'd actually gotten the port updated
sooner since then it would have worked for more than a day. As it
is, I could really use a final decision on which tarball will be at
http://www.llvm.org/releases/3.2/llvm-3.2.src.tar.gz in perpetuity so I
can change the port or not as required and avoid churn.

-- Brooks

Brooks,

Hi Pawel,

PTX already be replaced with NVPTX. However, PTX
subdirectory still sit in lib/Target in 3.2 release. Do
you think update the release tarball is a good idea? Also
could you remove it from the trunk?

Please do not, under no circumstances, change the 3.2
release tarballs at this point. They are mirrored around
the world now with cryptographic hashes and signatures.
Changing them will break things for many people, especially
for an extremely minor thing like an empty directory.

I'm not sure if Pawel's tarball change should be reverted
now as it already caused uproar, so changing it back might
only make matters worse.

The tarballs were changed?

r172208

I finally updated the FreeBSD ports yesterday and today a user
complained about distfile changes. IMO, this revision should
be reverted or all the other BSDs will have to chase checksums
as well.

If you really want to remove the directory, ship a 3.2.1
tarball rather than screwing all the downstream consumers who's
infrastructure exists to detect trojan'd tarballs.

Tarball is signed, it is not trjoan. Your infrastructure should
be able to deal with it?

The FreeBSD ports collection maintains a set of sha256 hashes for
each distfile. The system can deal with them changing, but it's
an inconvenience to port maintainers and users. Even if we did
have infrastructure to verify the signatures we'd still have to
check the contents and would not just trust the signature since
there's always a risk of keys being compromised.

-- Brooks

At first your reaction has puzzled me, but despite a few fire
breathing dragons getting loose and after carefully re-reading
these posts over the weekend I think I understand the underlying
issue.

Simply put we both take this release seriously.

The problem is the release process is not entirely
defined. Which is reflected in question from David Blaikie
(the other post I am quoting here).

Which release process document are you referring to that indicates
that?

Simple answer is the one on the llvm.org website.
But the real answer is, it all depends on how you read it.
Apparently, we read it differently which is not that
surprising as even our well defined compilers have
a bit of a problem with undefined behavior.

So what can we expect from a release process document?
My hope is that we have reached a sort of sequence
point and the ambiguity of the release process
can be resolved despite all the side effects.

David also stated this:

Generally, so far as I know, this kind of post-release change is
unprecedented.

I agree with this conclusion but even more unprecedented
is to a expect timely and problem free release with zero and
I'll repeat again zero involvement from the wider community that
depends on the clang+llvm!
This is how the 3.2 release has happened, just compare
the list of 3.2 to 3.1 binaries and BTW not a single
3.2 binary has been produced with the help of a respective
project, distribution or who ever.

I hear the cries of your wounded infrastructures
and I understand that in the days of forged root certificates
my signature on the tarballs may not guarantee much of a
security.

But why didn't you get involved in the 3.2 release?

We could have resolved all this in the 2 months since
the 3.2 release got rolling. Everybody knew about the
changes for the 3.2 release, simple e-mail to the new
LLVMRM would go a long way to resolve any security or
other release related issues not to mention a bit
of a courtesy.

I don't understand how my lack of involvement with this release has
anything to do with your decision to fix what appears to me to be
a non-problem in the release tarball well after the release and in
the process do something (replacing release tarballs) that is quite
simply NOT DONE to by projects who wish to treat their downstreams
with respect. This is something many projects end up learning due to
the painful way so it's not something you should feel bad about as
long as the project learns the right lesson(s).

Yessir! - lesson learned. Right one? Depends on a point of view.

Seems that downstream might also learn a thing or two about
participating in the release, and it does not have that painful.

If anything I'd be more annoyed if I'd actually gotten the port updated
sooner since then it would have worked for more than a day. As it
is, I could really use a final decision on which tarball will be at
http://www.llvm.org/releases/3.2/llvm-3.2.src.tar.gz in perpetuity so I
can change the port or not as required and avoid churn.

If you are in a hurry check with Tanya or Bill.

-- Brooks

I am doing this as service to the LLVM community
but frankly I'd rather be chasing my dragons of
probability now.

Paweł

If you are in a hurry check with Tanya or Bill.

Tanya explicitly asked you to revert everything. I'm going to do this
for you asap.

Stay tuned

On the FreeBSD ports side, I certainly could have done better, but
was unable to keep LLVM interactions high enough on my priority list.
Unfortunately, the release cycle seems to be well synchronized with the
schedule of meetings that require me to focus almost exclusively on
hacking demos. The version in the base system saw considerable testing
as it was integrated into FreeBSD at a couple points.

Thanks for all your work on the 3.2 release. When all is said and done
I'm quite happy with it.

-- Brooks

Anton,

If you are in a hurry check with Tanya or Bill.

Tanya explicitly asked you to revert everything. I'm going to do this
for you asap.

Thank you for taking the initiative to restore proper balance
in the LLVM universe.

Stay tuned

Are tuning to a local TV channel to watch some
"Nu, pogodi!" perhaps?

--
With best regards, Anton Korobeynikov
Faculty of Mathematics and Mechanics, Saint Petersburg State University

Paweł

Hi,

Hi Pawel,

  Sorry for the trouble. At first I think maybe we can upload a new
release tarball not replacing it, sorry I didn't say it in the previous
mail. IMHO, if you have to do something new after the post-release, make
a "dot" release would be better. Perhaps you can write down this
experience to benifit other LLVMRM in the future. :slight_smile:

Do not worry about this :slight_smile: You have spotted a release issue
that I have judged to be a problem.

The lesson from this is that the community needs to get
involved in the release at the early stage. I think this
the real challenge for the future LLVMRM.

Seems that too much has been written down already so
I am not sure if can contribute any more insight.
BTW your web page has a lot of good info!

Regards,
chenwj

Paweł

Brooks,

Hi Pawel,

PTX already be replaced with NVPTX. However, PTX
subdirectory still sit in lib/Target in 3.2
release. Do you think update the release tarball
is a good idea? Also could you remove it from the
trunk?

Please do not, under no circumstances, change the
3.2 release tarballs at this point. They are
mirrored around the world now with cryptographic
hashes and signatures. Changing them will break
things for many people, especially for an extremely
minor thing like an empty directory.

I'm not sure if Pawel's tarball change should be
reverted now as it already caused uproar, so
changing it back might only make matters worse.

The tarballs were changed?

r172208

I finally updated the FreeBSD ports yesterday and today
a user complained about distfile changes. IMO, this
revision should be reverted or all the other BSDs will
have to chase checksums as well.

If you really want to remove the directory, ship a
3.2.1 tarball rather than screwing all the downstream
consumers who's infrastructure exists to detect
trojan'd tarballs.

Tarball is signed, it is not trjoan. Your infrastructure
should be able to deal with it?

The FreeBSD ports collection maintains a set of sha256
hashes for each distfile. The system can deal with them
changing, but it's an inconvenience to port maintainers and
users. Even if we did have infrastructure to verify the
signatures we'd still have to check the contents and would
not just trust the signature since there's always a risk of
keys being compromised.

-- Brooks

At first your reaction has puzzled me, but despite a few
fire breathing dragons getting loose and after carefully
re-reading these posts over the weekend I think I understand
the underlying issue.

Simply put we both take this release seriously.

The problem is the release process is not entirely defined.
Which is reflected in question from David Blaikie (the other
post I am quoting here).

Which release process document are you referring to that
indicates that?

Simple answer is the one on the llvm.org website. But the
real answer is, it all depends on how you read it.
Apparently, we read it differently which is not that
surprising as even our well defined compilers have a bit of a
problem with undefined behavior.

So what can we expect from a release process document? My
hope is that we have reached a sort of sequence point and the
ambiguity of the release process can be resolved despite all
the side effects.

David also stated this:

Generally, so far as I know, this kind of post-release
change is unprecedented.

I agree with this conclusion but even more unprecedented is
to a expect timely and problem free release with zero and
I'll repeat again zero involvement from the wider community
that depends on the clang+llvm! This is how the 3.2 release
has happened, just compare the list of 3.2 to 3.1 binaries
and BTW not a single 3.2 binary has been produced with the
help of a respective project, distribution or who ever.

I hear the cries of your wounded infrastructures and I
understand that in the days of forged root certificates my
signature on the tarballs may not guarantee much of a
security.

But why didn't you get involved in the 3.2 release?

We could have resolved all this in the 2 months since the 3.2
release got rolling. Everybody knew about the changes for the
3.2 release, simple e-mail to the new LLVMRM would go a long
way to resolve any security or other release related issues
not to mention a bit of a courtesy.

I don't understand how my lack of involvement with this release
has anything to do with your decision to fix what appears to me
to be a non-problem in the release tarball well after the
release and in the process do something (replacing release
tarballs) that is quite simply NOT DONE to by projects who wish
to treat their downstreams with respect. This is something
many projects end up learning due to the painful way so it's
not something you should feel bad about as long as the project
learns the right lesson(s).

Yessir! - lesson learned. Right one? Depends on a point of view.

Seems that downstream might also learn a thing or two about
participating in the release, and it does not have that painful.

On the FreeBSD ports side, I certainly could have done better, but
was unable to keep LLVM interactions high enough on my priority
list. Unfortunately, the release cycle seems to be well
synchronized with the schedule of meetings that require me to focus
almost exclusively on hacking demos. The version in the base
system saw considerable testing as it was integrated into FreeBSD
at a couple points.

What I meant is that it is good to know who gets the final product
and also to get some feedback on the release before it goes out.
Staring at the web page with some links as the final product
at least for me does not seem to carry the same weight.

Thanks for all your work on the 3.2 release. When all is said and
done I'm quite happy with it.

Glad to hear this!

-- Brooks

Paweł