Weird miscompilation? Please help, i'm lost.

Hi.

I have accidentally wrote some code that results in rather strange symptoms.
I do not have a reduced testcase. I do not know how to reduce more than one
sourcefile in an automatic way.
*Please* bear with me, i'll try to write all the details i know.

The program is compiled with *NO* optimizations whatsoever, without any
threading, with clang version 6.0.0-svn310042-1~exp1;
however i observe the same with 3.9/4.0/5.0, and at least with -O1.

0. The main symptom is:
The following assertion fails:
https://github.com/LebedevRI/rawspeed/blob/
4901a4b3b979440be22c052b5dab962b84491b01/src/librawspeed/decompressors/
PentaxDecompressor.cpp#L212
How can it possibly fail, given all that code, no threading, and the fact that
they are integers?
1. Undefined Behaviour sanitizer (including interger sanitizer) do not say
anything about that code.
2. Address Sanitizer is silent too.
3. Memory Sanitizer is also silent (!)
4. This is probably *not* un-initialized read because the results are
deterministic and stable regardless of MALLOC_PERTURB_ and asan's
malloc_fill_byte)

Reproduction:
0. $ git clone https://github.com/LebedevRI/rawspeed.git
1. $ cd rawspeed
2. $ git checkout origin/llvm-broken
3. $ mkdir build && cd build
4. build it, with one single external dep - googletest:
$ CC=clang-6.0 CXX=clang++-6.0 cmake -DCMAKE_BUILD_TYPE=Debug -
DWITH_OPENMP=OFF -DWITH_PTHREADS=OFF -DWITH_PUGIXML=OFF -DWITH_JPEG=OFF -
DWITH_ZLIB=OFF -DUSE_XMLLINT=OFF -DBUILD_TESTING=ON -
DALLOW_DOWNLOADING_GOOGLEBENCHMARK=ON ../
5. $ make
6. get the file that uses that codepath (given raw sample is under CC0)
$ wget https://raw.pixls.us/getfile.php/867/raw/PK1_0762.PEF
7. $ touch PK1_0762.PEF.hash
8. actually hopefully reproduce:
$ src/utilities/rstest/rstest PK1_0762.PEF

This has to be either miscompilation/bug, or missing feature in UBSan, right?

I'm so sorry, i do not have a minimized testcase :frowning:
I'm lost. Not sure how else i can help with this. Please do tell.
But now, here is some (a lot of) gdb output.

$ gdb src/utilities/rstest/rstest
(gdb) r PK1_0762.PEF
Starting program: /home/lebedevri/rawspeed/build/src/utilities/rstest/rstest
PK1_0762.PEF
PK1_0762.PEF: starting decoding ...
rstest: /home/lebedevri/rawspeed/src/librawspeed/decompressors/
PentaxDecompressor.cpp:212: void
rawspeed::PentaxDecompressor::decompressInternal(int, int) const [TM =
rawspeed::PentaxDecompressor::ThreadingModel::SlaveThread]: Assertion `dest[x]
== pred[0]' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

^ the main symptom is that assertion failure.

This is not threading issue:
(gdb) thread
[Current thread is 1 (process 20673)]
(gdb) info threads
  Id Target Id Frame
* 1 process 20673 "rstest" __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/
sysv/linux/raise.c:51

(gdb) f 4
#4 0x000000000056976f in
rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>
(this=0x7fffffffcf30, row_start=0, row_end=4950) at /home/lebedevri/rawspeed/
src/librawspeed/decompressors/PentaxDecompressor.cpp:212
212 assert(dest[x] == pred[0]);
(gdb) bt full
...
#4 0x000000000056976f in
rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>
(this=0x7fffffffcf30, row_start=0, row_end=4950) at /home/lebedevri/rawspeed/
src/librawspeed/decompressors/PentaxDecompressor.cpp:212
        diff = {65535, 0}
        x = 2
        dest = 0x7fffefacd010
        pred = {65662, 126}
        y = 0
        bs = {<rawspeed::ByteStream> = {<rawspeed::DataBuffer> =
{<rawspeed::Buffer> = {data = 0x0, size = 0, isOwner = false}, endianness =
rawspeed::Endianness::little}, pos = 0}, cache =
{<rawspeed::BitStreamCacheBase> = {cache = 0, fillLevel = 0, static Size = 64,
              static MaxGetBits = 32, static MaxProcessBytes = 8}, <No data

}}

        draw = 0x7fffefacd010 "\177"
        pUp = {{0, 0}, {0, 0}}
...
(gdb) f 4
(gdb) p pred[0]
$1 = 65662
(gdb) p dest[x]
$2 = 126
^ so at least the assert did not lie.

Now, let's look at the disassembly. I have a slight hope someone might spot
something strange here? This is long, i'm pasting it all as-is.

(gdb) disassemble /m
Dump of assembler code for function
rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const:
137 void PentaxDecompressor::decompressInternal(int row_start, int
row_end) const {
   0x00000000005693b0 <+0>: push %rbp
   0x00000000005693b1 <+1>: mov %rsp,%rbp
   0x00000000005693b4 <+4>: sub $0x100,%rsp
   0x00000000005693bb <+11>: mov %rdi,-0x8(%rbp)
   0x00000000005693bf <+15>: mov %esi,-0xc(%rbp)
   0x00000000005693c2 <+18>: mov %edx,-0x10(%rbp)
   0x00000000005693c5 <+21>: mov -0x8(%rbp),%rdi
   0x00000000005693c9 <+25>: lea -0x38(%rbp),%rax

138 //#ifndef HAVE_PTHREAD
139 // static_assert(TM == ThreadingModel::NoThreading, "if there are
no
140 // pthreads, "
141 // "then serial
optimized "
142 // "codepath
should be
143 // called");
144 //#endif
145
146 // assert(!((disableThreading || getThreadCount() <= 1) ^
147 // (TM == ThreadingModel::NoThreading)) &&
148 // "either there is just one thread, and serial optimized
codepath is
149 // " "called; or there is more than one thread, and paralell
optimized
150 // " "codepaths are called");
151
152 BitPumpMSB bs;
   0x00000000005693cd <+29>: mov %rdi,-0x90(%rbp)
   0x00000000005693d4 <+36>: mov %rax,%rdi
   0x00000000005693d7 <+39>: callq 0x569910
<rawspeed::BitStream<rawspeed::MSBBitPumpTag,
rawspeed::BitStreamCacheRightInLeftOut>::BitStream()>

153
154 if (TM != ThreadingModel::SlaveThread) {
155 assert(row_start == 0);
156 assert(row_end > row_start);
157 assert(mRaw->dim.y == row_end);
158
159 bs = BitPumpMSB(input);
160 }
161
162 uchar8* draw = mRaw->getData();
   0x00000000005693dc <+44>: mov -0x90(%rbp),%rax
   0x00000000005693e3 <+51>: add $0x8,%rax
   0x00000000005693e7 <+55>: mov %rax,%rdi
   0x00000000005693ea <+58>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x00000000005693ef <+63>: mov %rax,-0x98(%rbp)
   0x00000000005693f6 <+70>: jmpq 0x5693fb
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+75>
   0x00000000005693fb <+75>: mov -0x98(%rbp),%rdi
   0x0000000000569402 <+82>: callq 0x4d4ad0
<rawspeed::RawImageData::getData() const>
   0x0000000000569407 <+87>: mov %rax,-0xa0(%rbp)
   0x000000000056940e <+94>: jmpq 0x569413
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+99>
   0x0000000000569413 <+99>: mov -0xa0(%rbp),%rax
   0x000000000056941a <+106>: mov %rax,-0x40(%rbp)

163
164 assert(mRaw->dim.y > 0);
   0x000000000056941e <+110>: mov -0x90(%rbp),%rcx
   0x0000000000569425 <+117>: add $0x8,%rcx
   0x0000000000569429 <+121>: mov %rcx,%rdi
   0x000000000056942c <+124>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x0000000000569431 <+129>: mov %rax,-0xa8(%rbp)
   0x0000000000569438 <+136>: jmpq 0x56943d
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+141>
   0x000000000056943d <+141>: mov -0xa8(%rbp),%rax
   0x0000000000569444 <+148>: cmpl $0x0,0x2c(%rax)
   0x0000000000569448 <+152>: jle 0x569453
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+163>
   0x000000000056944e <+158>: jmpq 0x569489
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+217>
   0x0000000000569453 <+163>: lea 0x18731(%rip),%rdi # 0x581b8b
   0x000000000056945a <+170>: lea 0x21c84(%rip),%rsi # 0x58b0e5
   0x0000000000569461 <+177>: mov $0xa4,%edx
   0x0000000000569466 <+182>: lea 0x2207e(%rip),%rcx # 0x58b4eb
   0x000000000056946d <+189>: callq 0x4c5d20 <__assert_fail@plt>
   0x0000000000569472 <+194>: lea -0x38(%rbp),%rdi

165 assert(mRaw->dim.x > 0);
   0x0000000000569489 <+217>: mov -0x90(%rbp),%rax
   0x0000000000569490 <+224>: add $0x8,%rax
   0x0000000000569494 <+228>: mov %rax,%rdi
   0x0000000000569497 <+231>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x000000000056949c <+236>: mov %rax,-0xb0(%rbp)
   0x00000000005694a3 <+243>: jmpq 0x5694a8
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+248>
   0x00000000005694a8 <+248>: mov -0xb0(%rbp),%rax
   0x00000000005694af <+255>: cmpl $0x0,0x28(%rax)
   0x00000000005694b3 <+259>: jle 0x5694be
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+270>
   0x00000000005694b9 <+265>: jmpq 0x5694dd
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+301>
   0x00000000005694be <+270>: lea 0x186b6(%rip),%rdi # 0x581b7b
   0x00000000005694c5 <+277>: lea 0x21c19(%rip),%rsi # 0x58b0e5
   0x00000000005694cc <+284>: mov $0xa5,%edx
   0x00000000005694d1 <+289>: lea 0x22013(%rip),%rcx # 0x58b4eb
   0x00000000005694d8 <+296>: callq 0x4c5d20 <__assert_fail@plt>

166 assert(mRaw->dim.x % 2 == 0);
   0x00000000005694dd <+301>: mov -0x90(%rbp),%rax
   0x00000000005694e4 <+308>: add $0x8,%rax
   0x00000000005694e8 <+312>: mov %rax,%rdi
   0x00000000005694eb <+315>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x00000000005694f0 <+320>: mov %rax,-0xb8(%rbp)
   0x00000000005694f7 <+327>: jmpq 0x5694fc
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+332>
   0x00000000005694fc <+332>: mov $0x2,%eax
   0x0000000000569501 <+337>: mov -0xb8(%rbp),%rcx
   0x0000000000569508 <+344>: mov 0x28(%rcx),%edx
   0x000000000056950b <+347>: mov %eax,-0xbc(%rbp)
   0x0000000000569511 <+353>: mov %edx,%eax
   0x0000000000569513 <+355>: cltd
   0x0000000000569514 <+356>: mov -0xbc(%rbp),%esi
   0x000000000056951a <+362>: idiv %esi
   0x000000000056951c <+364>: cmp $0x0,%edx
   0x000000000056951f <+367>: jne 0x56952a
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+378>
   0x0000000000569525 <+373>: jmpq 0x569549
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+409>
   0x000000000056952a <+378>: lea 0x21f8e(%rip),%rdi # 0x58b4bf
   0x0000000000569531 <+385>: lea 0x21bad(%rip),%rsi # 0x58b0e5
   0x0000000000569538 <+392>: mov $0xa6,%edx
   0x000000000056953d <+397>: lea 0x21fa7(%rip),%rcx # 0x58b4eb
   0x0000000000569544 <+404>: callq 0x4c5d20 <__assert_fail@plt>

167 assert(row_end <= mRaw->dim.y);
   0x0000000000569549 <+409>: mov -0x10(%rbp),%eax
   0x000000000056954c <+412>: mov -0x90(%rbp),%rcx
   0x0000000000569553 <+419>: add $0x8,%rcx
   0x0000000000569557 <+423>: mov %rcx,%rdi
   0x000000000056955a <+426>: mov %eax,-0xc0(%rbp)
   0x0000000000569560 <+432>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x0000000000569565 <+437>: mov %rax,-0xc8(%rbp)
   0x000000000056956c <+444>: jmpq 0x569571
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+449>
   0x0000000000569571 <+449>: mov -0xc0(%rbp),%eax
   0x0000000000569577 <+455>: mov -0xc8(%rbp),%rcx
   0x000000000056957e <+462>: cmp 0x2c(%rcx),%eax
   0x0000000000569581 <+465>: jg 0x56958c
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+476>
   0x0000000000569587 <+471>: jmpq 0x5695ab
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+507>
   0x000000000056958c <+476>: lea 0x21f41(%rip),%rdi # 0x58b4d4
   0x0000000000569593 <+483>: lea 0x21b4b(%rip),%rsi # 0x58b0e5
   0x000000000056959a <+490>: mov $0xa7,%edx
   0x000000000056959f <+495>: lea 0x21f45(%rip),%rcx # 0x58b4eb
   0x00000000005695a6 <+502>: callq 0x4c5d20 <__assert_fail@plt>
   0x00000000005695ab <+507>: xor %esi,%esi
   0x00000000005695ad <+509>: mov $0x10,%eax
   0x00000000005695b2 <+514>: mov %eax,%edx
   0x00000000005695b4 <+516>: lea -0x60(%rbp),%rcx

168
169 int pUp[2][2] = {{0, 0}, {0, 0}};
   0x00000000005695b8 <+520>: mov %rcx,%rdi
   0x00000000005695bb <+523>: callq 0x4c60f0 <memset@plt>

170
171 for (int y = row_start; y < row_end && mRaw->dim.x >= 2; y++) {
   0x00000000005695c0 <+528>: mov -0xc(%rbp),%eax
   0x00000000005695c3 <+531>: mov %eax,-0x64(%rbp)
   0x00000000005695c6 <+534>: xor %eax,%eax
   0x00000000005695c8 <+536>: mov %al,%cl
   0x00000000005695ca <+538>: mov -0x64(%rbp),%eax
   0x00000000005695cd <+541>: cmp -0x10(%rbp),%eax
   0x00000000005695d0 <+544>: mov %cl,-0xc9(%rbp)
   0x00000000005695d6 <+550>: jge 0x56960f
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+607>
   0x00000000005695dc <+556>: mov -0x90(%rbp),%rax
   0x00000000005695e3 <+563>: add $0x8,%rax
   0x00000000005695e7 <+567>: mov %rax,%rdi
   0x00000000005695ea <+570>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x00000000005695ef <+575>: mov %rax,-0xd8(%rbp)
   0x00000000005695f6 <+582>: jmpq 0x5695fb
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+587>
   0x00000000005695fb <+587>: mov -0xd8(%rbp),%rax
   0x0000000000569602 <+594>: cmpl $0x2,0x28(%rax)
   0x0000000000569606 <+598>: setge %cl
   0x0000000000569609 <+601>: mov %cl,-0xc9(%rbp)
   0x000000000056960f <+607>: mov -0xc9(%rbp),%al
   0x0000000000569615 <+613>: test $0x1,%al
   0x0000000000569617 <+615>: jne 0x569622
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+626>
   0x000000000056961d <+621>: jmpq 0x569883
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1235>
   0x0000000000569875 <+1221>: mov -0x64(%rbp),%eax
   0x0000000000569878 <+1224>: add $0x1,%eax
   0x000000000056987b <+1227>: mov %eax,-0x64(%rbp)
   0x000000000056987e <+1230>: jmpq 0x5695c6
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+534>
   0x0000000000569883 <+1235>: lea -0x38(%rbp),%rdi

172 auto* dest = reinterpret_cast<ushort16*>(&draw[y * mRaw->pitch]);
   0x0000000000569622 <+626>: mov -0x40(%rbp),%rax
   0x0000000000569626 <+630>: mov -0x64(%rbp),%ecx
   0x0000000000569629 <+633>: mov -0x90(%rbp),%rdx
   0x0000000000569630 <+640>: add $0x8,%rdx
   0x0000000000569634 <+644>: mov %rdx,%rdi
   0x0000000000569637 <+647>: mov %rax,-0xe0(%rbp)
   0x000000000056963e <+654>: mov %ecx,-0xe4(%rbp)
   0x0000000000569644 <+660>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x0000000000569649 <+665>: mov %rax,-0xf0(%rbp)
   0x0000000000569650 <+672>: jmpq 0x569655
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+677>
   0x0000000000569655 <+677>: mov -0xe4(%rbp),%eax
   0x000000000056965b <+683>: mov -0xf0(%rbp),%rcx
   0x0000000000569662 <+690>: imul 0x30(%rcx),%eax
   0x0000000000569666 <+694>: mov %eax,%eax
   0x0000000000569668 <+696>: mov %eax,%edx
   0x000000000056966a <+698>: mov -0xe0(%rbp),%rsi
   0x0000000000569671 <+705>: add %rdx,%rsi
   0x0000000000569674 <+708>: mov %rsi,-0x70(%rbp)

173
174 int pred[2];
175
176 // first two pixels of the row
177 if (TM != ThreadingModel::SlaveThread) {
178 // decode differences, do prediction, store predicted values
179 dest[0] = pred[0] = pUp[0][y & 1] += ht.decodeNext(bs);
180 dest[1] = pred[1] = pUp[1][y & 1] += ht.decodeNext(bs);
181 } else {
182 // already predicted, just read initial predictor values.
183 pred[0] = dest[0];
   0x0000000000569678 <+712>: mov -0x70(%rbp),%rdx
   0x000000000056967c <+716>: movzwl (%rdx),%eax
   0x000000000056967f <+719>: mov %eax,-0x78(%rbp)

184 pred[1] = dest[1];
   0x0000000000569682 <+722>: mov -0x70(%rbp),%rdx
   0x0000000000569686 <+726>: movzwl 0x2(%rdx),%eax
   0x000000000056968a <+730>: mov %eax,-0x74(%rbp)

185 }
186
187 for (int x = 2; x < mRaw->dim.x; x += 2) {
   0x000000000056968d <+733>: movl $0x2,-0x7c(%rbp)
   0x0000000000569694 <+740>: mov -0x7c(%rbp),%eax
   0x0000000000569697 <+743>: mov -0x90(%rbp),%rcx
   0x000000000056969e <+750>: add $0x8,%rcx
   0x00000000005696a2 <+754>: mov %rcx,%rdi
   0x00000000005696a5 <+757>: mov %eax,-0xf4(%rbp)
   0x00000000005696ab <+763>: callq 0x4ca770 <rawspeed::RawImage::operator-

() const>

   0x00000000005696b0 <+768>: mov %rax,-0x100(%rbp)
   0x00000000005696b7 <+775>: jmpq 0x5696bc
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+780>
   0x00000000005696bc <+780>: mov -0xf4(%rbp),%eax
   0x00000000005696c2 <+786>: mov -0x100(%rbp),%rcx
   0x00000000005696c9 <+793>: cmp 0x28(%rcx),%eax
   0x00000000005696cc <+796>: jge 0x569870
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1216>
   0x0000000000569862 <+1202>: mov -0x7c(%rbp),%eax
   0x0000000000569865 <+1205>: add $0x2,%eax
   0x0000000000569868 <+1208>: mov %eax,-0x7c(%rbp)
   0x000000000056986b <+1211>: jmpq 0x569694
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+740>

188 int diff[2];
189
190 if (TM != ThreadingModel::SlaveThread) {
191 // if this is not the prediction slave threads, decode next
difference
192 diff[0] = ht.decodeNext(bs);
193 diff[1] = ht.decodeNext(bs);
194 } else {
195 // else, read next difference from the the output
196 diff[0] = dest[x];
   0x00000000005696d2 <+802>: mov -0x70(%rbp),%rax
   0x00000000005696d6 <+806>: movslq -0x7c(%rbp),%rcx
   0x00000000005696da <+810>: movzwl (%rax,%rcx,2),%edx
   0x00000000005696de <+814>: mov %edx,-0x84(%rbp)

197 diff[1] = dest[x + 1];
   0x00000000005696e4 <+820>: mov -0x70(%rbp),%rax
   0x00000000005696e8 <+824>: mov -0x7c(%rbp),%edx
   0x00000000005696eb <+827>: add $0x1,%edx
   0x00000000005696ee <+830>: movslq %edx,%rcx
   0x00000000005696f1 <+833>: movzwl (%rax,%rcx,2),%edx
   0x00000000005696f5 <+837>: mov %edx,-0x80(%rbp)

198 }
199
200 if (TM == ThreadingModel::MainThread) {
201 // if this is the main thread, just store the differences
202 dest[x] = diff[0];
203 dest[x + 1] = diff[1];
204 } else {
205 // else, do prediction, store predicted values
206 pred[0] += diff[0];
   0x00000000005696f8 <+840>: mov -0x84(%rbp),%edx
   0x00000000005696fe <+846>: add -0x78(%rbp),%edx
   0x0000000000569701 <+849>: mov %edx,-0x78(%rbp)

207 pred[1] += diff[1];
   0x0000000000569704 <+852>: mov -0x80(%rbp),%edx
   0x0000000000569707 <+855>: add -0x74(%rbp),%edx
   0x000000000056970a <+858>: mov %edx,-0x74(%rbp)

208
209 dest[x] = pred[0];
   0x000000000056970d <+861>: mov -0x78(%rbp),%edx
   0x0000000000569710 <+864>: mov %dx,%si
   0x0000000000569713 <+867>: mov -0x70(%rbp),%rax
   0x0000000000569717 <+871>: movslq -0x7c(%rbp),%rcx
   0x000000000056971b <+875>: mov %si,(%rax,%rcx,2)

210 dest[x + 1] = pred[1];
   0x000000000056971f <+879>: mov -0x74(%rbp),%edx
   0x0000000000569722 <+882>: mov %dx,%si
   0x0000000000569725 <+885>: mov -0x70(%rbp),%rax
   0x0000000000569729 <+889>: mov -0x7c(%rbp),%edx
   0x000000000056972c <+892>: add $0x1,%edx
   0x000000000056972f <+895>: movslq %edx,%rcx
   0x0000000000569732 <+898>: mov %si,(%rax,%rcx,2)

211
212 assert(dest[x] == pred[0]);
   0x0000000000569736 <+902>: mov -0x70(%rbp),%rax
   0x000000000056973a <+906>: movslq -0x7c(%rbp),%rcx
   0x000000000056973e <+910>: movzwl (%rax,%rcx,2),%edx
   0x0000000000569742 <+914>: cmp -0x78(%rbp),%edx
   0x0000000000569745 <+917>: jne 0x569750
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+928>
   0x000000000056974b <+923>: jmpq 0x56976f
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+959>
   0x0000000000569750 <+928>: lea 0x21e1b(%rip),%rdi # 0x58b572
   0x0000000000569757 <+935>: lea 0x21987(%rip),%rsi # 0x58b0e5
   0x000000000056975e <+942>: mov $0xd4,%edx
   0x0000000000569763 <+947>: lea 0x21d81(%rip),%rcx # 0x58b4eb
   0x000000000056976a <+954>: callq 0x4c5d20 <__assert_fail@plt>

213 assert(dest[x + 1] == pred[1]);
=> 0x000000000056976f <+959>: mov -0x70(%rbp),%rax
   0x0000000000569773 <+963>: mov -0x7c(%rbp),%ecx
   0x0000000000569776 <+966>: add $0x1,%ecx
   0x0000000000569779 <+969>: movslq %ecx,%rdx
   0x000000000056977c <+972>: movzwl (%rax,%rdx,2),%ecx
   0x0000000000569780 <+976>: cmp -0x74(%rbp),%ecx
   0x0000000000569783 <+979>: jne 0x56978e
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+990>
   0x0000000000569789 <+985>: jmpq 0x5697ad
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1021>
   0x000000000056978e <+990>: lea 0x21df0(%rip),%rdi # 0x58b585
   0x0000000000569795 <+997>: lea 0x21949(%rip),%rsi # 0x58b0e5
   0x000000000056979c <+1004>: mov $0xd5,%edx
   0x00000000005697a1 <+1009>: lea 0x21d43(%rip),%rcx # 0x58b4eb
   0x00000000005697a8 <+1016>: callq 0x4c5d20 <__assert_fail@plt>

214
215 if (dest[x] < 0 || dest[x] > 65535)
   0x00000000005697ad <+1021>: mov -0x70(%rbp),%rax
   0x00000000005697b1 <+1025>: movslq -0x7c(%rbp),%rcx
   0x00000000005697b5 <+1029>: movzwl (%rax,%rcx,2),%edx
   0x00000000005697b9 <+1033>: cmp $0x0,%edx
   0x00000000005697bc <+1036>: jl 0x5697da
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1066>
   0x00000000005697c2 <+1042>: mov -0x70(%rbp),%rax
   0x00000000005697c6 <+1046>: movslq -0x7c(%rbp),%rcx
   0x00000000005697ca <+1050>: movzwl (%rax,%rcx,2),%edx
   0x00000000005697ce <+1054>: cmp $0xffff,%edx
   0x00000000005697d4 <+1060>: jle 0x569800
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1104>

216 ThrowRDE("decoded value out of bounds at (%d, %d)", x, y);
   0x00000000005697da <+1066>: mov -0x7c(%rbp),%edx
   0x00000000005697dd <+1069>: mov -0x64(%rbp),%ecx
   0x00000000005697e0 <+1072>: lea 0x21db5(%rip),%rdi # 0x58b59c
   0x00000000005697e7 <+1079>: lea 0x21cfd(%rip),%rsi # 0x58b4eb
   0x00000000005697ee <+1086>: xor %eax,%eax
   0x00000000005697f0 <+1088>: mov %al,%r8b
   0x00000000005697f3 <+1091>: mov %r8b,%al
   0x00000000005697f6 <+1094>: callq 0x4d6550
<rawspeed::ThrowException<rawspeed::RawDecoderException>(char const*, ...)>
   0x00000000005697fb <+1099>: jmpq 0x569800
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1104>

217 if (dest[x + 1] < 0 || dest[x + 1] > 65535)
   0x0000000000569800 <+1104>: mov -0x70(%rbp),%rax
   0x0000000000569804 <+1108>: mov -0x7c(%rbp),%ecx
   0x0000000000569807 <+1111>: add $0x1,%ecx
   0x000000000056980a <+1114>: movslq %ecx,%rdx
   0x000000000056980d <+1117>: movzwl (%rax,%rdx,2),%ecx
   0x0000000000569811 <+1121>: cmp $0x0,%ecx
   0x0000000000569814 <+1124>: jl 0x569837
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1159>
   0x000000000056981a <+1130>: mov -0x70(%rbp),%rax
   0x000000000056981e <+1134>: mov -0x7c(%rbp),%ecx
   0x0000000000569821 <+1137>: add $0x1,%ecx
   0x0000000000569824 <+1140>: movslq %ecx,%rdx
   0x0000000000569827 <+1143>: movzwl (%rax,%rdx,2),%ecx
   0x000000000056982b <+1147>: cmp $0xffff,%ecx
   0x0000000000569831 <+1153>: jle 0x56985d
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1197>

218 ThrowRDE("decoded value out of bounds at (%d, %d)", x, y);
   0x0000000000569837 <+1159>: mov -0x7c(%rbp),%edx
   0x000000000056983a <+1162>: mov -0x64(%rbp),%ecx
   0x000000000056983d <+1165>: lea 0x21dd7(%rip),%rdi # 0x58b61b
   0x0000000000569844 <+1172>: lea 0x21ca0(%rip),%rsi # 0x58b4eb
   0x000000000056984b <+1179>: xor %eax,%eax
   0x000000000056984d <+1181>: mov %al,%r8b
   0x0000000000569850 <+1184>: mov %r8b,%al
   0x0000000000569853 <+1187>: callq 0x4d6550
<rawspeed::ThrowException<rawspeed::RawDecoderException>(char const*, ...)>
   0x0000000000569858 <+1192>: jmpq 0x56985d
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1197>

219 }
220 }
   0x000000000056985d <+1197>: jmpq 0x569862
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1202>

221 }
   0x0000000000569870 <+1216>: jmpq 0x569875
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1221>

222 }
   0x0000000000569476 <+198>: mov %edx,%ecx
   0x0000000000569478 <+200>: mov %rax,-0x48(%rbp)
   0x000000000056947c <+204>: mov %ecx,-0x4c(%rbp)
   0x000000000056947f <+207>: callq 0x4f9290
<rawspeed::BitStream<rawspeed::MSBBitPumpTag,
rawspeed::BitStreamCacheRightInLeftOut>::~BitStream()>
   0x0000000000569484 <+212>: jmpq 0x569895
<rawspeed::PentaxDecompressor::decompressInternal<(rawspeed::PentaxDecompressor::ThreadingModel)2>(int,
int) const+1253>
   0x0000000000569887 <+1239>: callq 0x4f9290
<rawspeed::BitStream<rawspeed::MSBBitPumpTag,
rawspeed::BitStreamCacheRightInLeftOut>::~BitStream()>
   0x000000000056988c <+1244>: add $0x100,%rsp
   0x0000000000569893 <+1251>: pop %rbp
   0x0000000000569894 <+1252>: retq
   0x0000000000569895 <+1253>: mov -0x48(%rbp),%rdi
   0x0000000000569899 <+1257>: callq 0x4c5c40 <_Unwind_Resume@plt>

End of assembler dump.

Roman.

Hi all.

Thankfully, it turned out to be a simple integer truncation in my code. :slight_smile:

Would it be *too* much out of the scope of UBSan,
since there already is a not-so-UB -fsanitize=integer?

Sadly, i have -Wconversion diagnostics disabled, though it would
not have helped much here: it seems gcc's -Wtype-limits needs
to be implemented, https://bugs.llvm.org/show_bug.cgi?id=34147

Sorry for the noise :slight_smile:

Roman.

Hi.

I have accidentally wrote some code that results in rather strange symptoms.
I do not have a reduced testcase. I do not know how to reduce more than one
sourcefile in an automatic way.
*Please* bear with me, i'll try to write all the details i know.

...