OpenJDK8 failed to work after compiled by LLVM 8 for X86

Hi all,

OpenJDK8 jdk8u-dev[1] is just able to work after compiled with LLVM 3.9.1 for X86:

$ ./build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/bin/java -version
openjdk version "1.8.0-internal-debug"
OpenJDK Runtime Environment (build 1.8.0-internal-debug-xiangzhai_2018_09_09_21_08-b00)
OpenJDK 64-Bit Server VM (build 25.71-b00-debug, mixed mode)

$ strings ./build/linux-x86_64-normal-server-slowdebug/images/j2sdk-image/bin/java

grep clang

clang version 3.9.1 (tags/RELEASE_391/final)

But it failed to work after compiled with LLVM 8 for X86:

$ ./build/linux-x86_64-normal-server-fastdebug/images/j2sdk-image/bin/java -version
Segmentation fault

$ gdb --ex run --args ./build/linux-x86_64-normal-server-fastdebug/images/j2sdk-image/bin/java -version
GNU gdb (GDB) Fedora 7.12.1-48.fc25
...
Starting program: /home/xiangzhai/project/jdk8u-llvm/build/linux-x86_64-normal-server-fastdebug/images/j2sdk-image/bin/java -version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
[ Legend: Modified register | Code | Heap | Stack | String ]
───────────────────────────────────────────────────────────────────────────────────────────────[ registers ]────
$rax : 0x0
$rbx : 0x0
$rcx : 0x7ffff7315eb0 → 0x00007ffff6ae30a0 → <NativeCallStack::print_on(outputStream*)+0> xor edx, edx
$rdx : 0x0
$rsp : 0x7fffffff9328 → 0x00007ffff6a76822 → <MemTracker::init_tracking_level()+178> mov edi, ebx
$rbp : 0x7fffffff93c0 → 0x00007fffffff94f0 → 0x00007fffffff9510 → 0x0000000000000002
$rsi : 0x0
$rdi : 0x7ffff7315e70 → 0x00007ffff7315eb0 → 0x00007ffff6ae30a0 → <NativeCallStack::print_on(outputStream*)+0> xor edx, edx
$rip : 0x7ffff6ae2f5b → <NativeCallStack::NativeCallStack(int,+0> mov QWORD PTR [rdi], rcx
$r8 : 0x1
$r9 : 0xf
$r10 : 0x64
$r11 : 0x0
$r12 : 0x7fffffffdc88 → 0x00007fffffffe04c → "/home/xiangzhai/project/jdk8u-llvm/build/linux-x86[...]"
$r13 : 0x7fffffffdca0 → 0x00007fffffffe0bf → "XDG_VTNR=1"
$r14 : 0x7fffffff9330 → "NMT_LEVEL_10535"
$r15 : 0x6
$eflags: [carry parity adjust zero sign trap INTERRUPT direction overflow RESUME virtualx86 identification]
$ss: 0x002b $ds: 0x0000 $cs: 0x0033 $es: 0x0000 $gs: 0x0000 $fs: 0x0000
───────────────────────────────────────────────────────────────────────────────────────────────────[ stack ]────
0x00007fffffff9328│+0x00: 0x00007ffff6a76822 → <MemTracker::init_tracking_level()+178> mov edi, ebx ← $rsp
0x00007fffffff9330│+0x08: "NMT_LEVEL_10535" ← $r14
0x00007fffffff9338│+0x10: 0x0035333530315f4c ("L_10535"?)
0x00007fffffff9340│+0x18: 0x0000000000000001
0x00007fffffff9348│+0x20: 0x0000000000602190 → 0x00007ffff5eb7000 → 0x00010102464c457f
0x00007fffffff9350│+0x28: 0x0000000000000004
0x00007fffffff9358│+0x30: 0x000000066474e551
0x00007fffffff9360│+0x38: 0x0000000000000000
────────────────────────────────────────────────────────────────────────────────────────[ code:i386:x86-64 ]────
0x7ffff6ae2f4b <print_owned_locks_on_error(outputStream*)+187> add eax, 0x1f0f00
0x7ffff6ae2f50 <NativeCallStack::NativeCallStack(int,+0> mov rcx, QWORD PTR [rip+0x842781] # 0x7ffff73256d8
0x7ffff6ae2f57 <NativeCallStack::NativeCallStack(int,+0> add rcx, 0x10
→ 0x7ffff6ae2f5b <NativeCallStack::NativeCallStack(int,+0> mov QWORD PTR [rdi], rcx
0x7ffff6ae2f5e <NativeCallStack::NativeCallStack(int,+0> mov DWORD PTR [rdi+0x28], 0x0
0x7ffff6ae2f65 <NativeCallStack::NativeCallStack(int,+0> add rdi, 0x8
0x7ffff6ae2f69 <NativeCallStack::NativeCallStack(int,+0> test edx, edx
0x7ffff6ae2f6b <NativeCallStack::NativeCallStack(int,+0> je 0x7ffff6ae2f7b <NativeCallStack::NativeCallStack(int, bool)+43>
0x7ffff6ae2f6d <NativeCallStack::NativeCallStack(int,+0> mov eax, esi
─────────[ source:/home/xiangzhai/project/jdk8u-llvm/hotspot/src/share/vm/utilities/nativeCallStack.cpp+33 ]────
28 #include "utilities/nativeCallStack.hpp"
29
30 const NativeCallStack NativeCallStack::EMPTY_STACK(0, false);
31
32 NativeCallStack::NativeCallStack(int toSkip, bool fillStack) :
→ 33 _hash_value(0) {
34
35 #if !PLATFORM_NATIVE_STACK_WALKING_SUPPORTED
36 fillStack = false;
37 #endif
38
─────────────────────────────────────────────────────────────────────────────────────────────────[ threads ]────
[#0] Id 1, Name: "java", stopped, reason: SIGSEGV
───────────────────────────────────────────────────────────────────────────────────────────────────[ trace ]────
[#0] 0x7ffff6ae2f5b → Name: NativeCallStack::NativeCallStack(this=0x7ffff7315e70 <NativeCallStack::EMPTY_STACK>, toSkip=0x0, fillStack=0x0)
[#1] 0x7ffff6a76822 → Name: MemTracker::init_tracking_level()
[#2] 0x7ffff62eff49 → Name: MemTracker::tracking_level()
[#3] 0x7ffff62eff49 → Name: ResourceObj::operator new(size=0x20, type=ResourceObj::STACK_OR_EMBEDDED, flags=<optimized out>)
[#4] 0x7ffff61369ec → Name: _GLOBAL__sub_I_c1_Greedy.cpp()
[#5] 0x7ffff7de7d9a → Name: call_init(l=<optimized out>, argc=0x2, argv=0x7fffffffdc88, env=0x7fffffffdca0)
[#6] 0x7ffff7de7eab → Name: call_init(env=0x7fffffffdca0, argv=0x7fffffffdc88, argc=0x2, l=<optimized out>)
[#7] 0x7ffff7de7eab → Name: _dl_init(main_map=0x602190, argc=0x2, argv=0x7fffffffdc88, env=0x7fffffffdca0)
[#8] 0x7ffff7dece46 → Name: dl_open_worker(a=0x7fffffff97e0)
[#9] 0x7ffff7de7c44 → Name: _dl_catch_error(objname=0x7fffffff97d0, errstring=0x7fffffff97d8, mallocedp=0x7fffffff97cf, operate=0x7ffff7decae0 <dl_open_worker>, args=0x7fffffff97e0)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────
0x00007ffff6ae2f5b in NativeCallStack::NativeCallStack (this=0x7ffff7315e70 <NativeCallStack::EMPTY_STACK>, toSkip=0x0, fillStack=0x0) at /home/xiangzhai/project/jdk8u-llvm/hotspot/src/share/vm/utilities/nativeCallStack.cpp:33
33 _hash_value(0) {
gef➤ c
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
gef➤

----- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---

I started to use and contribute clang static analyzer[2] just for finding the bugs for open source software from LLVM 3.x, perhaps it is a bug of LLVM 8, because jdk8u is just able to print out '-version' after compiled with llvm-3.9.1, I will rebuild the LLVM 8 again, then rebuild jdk8u for triple check.

1. Workaround-compile-with-llvm.patch https://raw.githubusercontent.com/xiangzhai/jdk8u-dev/master/Workaround-compile-with-llvm.patch

2. https://github.com/llvm-mirror/clang/commit/a530e823ed2793ac149de7a984014e242a787682#diff-ff9160d4628cb9b6186559837c2c8668

3. The build option of LLVM 8 (bootstrap with LLVM 3.9.1):

cmake .. -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_C_FLAGS="-fPIC" \
-DCMAKE_CXX_FLAGS="-std=c++11 -fPIC" \
-DLLVM_BUILD_LLVM_DYLIB=ON \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_INSTALL_UTILS=ON \
-DLLVM_ENABLE_RTTI=ON \
-DLLVM_ENABLE_FFI=ON \
-DLLVM_ENABLE_EH=ON \
-DLLVM_BUILD_TESTS=ON \
-DLLVM_BUILD_DOCS=OFF \
-DLLVM_ENABLE_SPHINX=OFF \
-DLLVM_ENABLE_DOXYGEN=OFF \
-DLLDB_DISABLE_LIBEDIT=1 \
-DSPHINX_WARNINGS_AS_ERRORS=OFF \
-DFFI_INCLUDE_DIR=$(pkg-config --variable=includedir libffi) \
-DFFI_LIBRARY_DIR:PATH="$(pkg-config --variable=libdir libffi)" \
-DLLVM_BINUTILS_INCDIR=/usr/include \
-DLLVM_LIBDIR_SUFFIX=64 \
-DLIBCXX_ENABLE_ABI_LINKER_SCRIPT=ON \
-DLIBUNWIND_ENABLE_SHARED=ON \
-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-DLLDB_TEST_C_COMPILER=clang \
-DLLDB_TEST_CXX_COMPILER=clang++ \
-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-redhat-linux" \
-DCLANG_VENDOR="LLVM China"

4. $ clang -v
LLVM China clang version 8.0.0 (git@github.com:llvm-mirror/clang.git 81ef98628ebf5186d746c0986dcbf5073e842043) (git@github.com:llvm-mirror/llvm.git e1aac9723d55497e74d83d216329f08d9842e494) (based on LLVM 8.0.0svn)
Target: x86_64-redhat-linux
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-redhat-linux/6.4.1
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/6.4.1
Found candidate GCC installation: /usr/lib/gcc/i686-redhat-linux/6.4.1
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/6.4.1
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/6.4.1
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64

Hi Leslie,

This is likely the same problem as was reported in https://bugs.freebsd.org/225054#c8, and fixed by the following patch:

https://svnweb.freebsd.org/ports/head/java/openjdk8/files/patch-hotspot_src_share_vm_services_memTracker.cpp?view=markup&pathrev=459368

Can you please try that out, and see if it fixes it for you too?

-Dimitry

Hi Dimitry,

Thanks for your kind response!

Thanks for the commit message of Jung's patch, I found that the bug had been fixed in OpenJDK 12 by Zhengyu https://bugs.openjdk.java.net/browse/JDK-8205965 But only backported to 11. So Jung could backport it for OpenJDK 8, thanks a lot!

But I argue that the root cause might be in the compiler side, why clang-3.9.1, gcc-6.4.1 couldn't reproduce the bug? And it might be work for clang-4.0 https://bugs.openjdk.java.net/browse/JDK-8208494 I will test LLVM 5 to check out whether or not reproducible.

Hi Leslie,

The problem really lies in the OpenJDK code, as it is attempting to
write to a const object. If this seems to work with certain compiler(s)
and optimization settings, it is just luck. :slight_smile:

Here is a reduced example, which shows the issue:

Hi Dimitry,

Thank you so much for the reduced testcase! It is able to reproduce after compiled with clang-8 optimized for X86:

$ clang++ -O3 -S -c JDK-8205969.cpp -o JDK-8205969-opt-8.0.s
$ clang++ -O3 -c JDK-8205969.cpp -o JDK-8205969-opt-8.0.o
$ clang++ -o JDK-8205969-opt-8.0.out JDK-8205969-opt-8.0.o

$ ./JDK-8205969-opt-8.0.out
Segmentation fault

$ cat JDK-8205969-opt-8.0.s
.text
.file "JDK-8205969.cpp"
.globl main # -- Begin function main
.p2align 4, 0x90
.type main,@function
main: # @main
.cfi_startproc
# %bb.0:
xorps %xmm0, %xmm0
movups %xmm0, _ZN15NativeCallStack11EMPTY_STACKE+16(%rip)
movups %xmm0, _ZN15NativeCallStack11EMPTY_STACKE(%rip)
xorl %eax, %eax
retq
.Lfunc_end0:
.size main, .Lfunc_end0-main
.cfi_endproc
# -- End function
.type _ZN15NativeCallStack11EMPTY_STACKE,@object # @_ZN15NativeCallStack11EMPTY_STACKE
.section .rodata,"a",@progbits

                          ^\-\-\- READ\-ONLY segment

     \.globl  \_ZN15NativeCallStack11EMPTY\_STACKE
     \.p2align        3

_ZN15NativeCallStack11EMPTY_STACKE:
.zero 32
.size _ZN15NativeCallStack11EMPTY_STACKE, 32

     \.ident  &quot;LLVM China clang version 8\.0\.0 \(git@github\.com:llvm\-mirror/clang\.git 81ef98628ebf5186d746c0986dcbf5073e842043\) \(git@github\.com:llvm\-mirror/llvm\.git e1aac9723d55497e74d83d216329f08d9842e494\) \(based on LLVM 8\.0\.0svn\)&quot;
     \.section        &quot;\.note\.GNU\-stack&quot;,&quot;&quot;,@progbits
     \.addrsig

But it is *not* able to reproduce with clang-8 not optimized for X86:

$ clang++ -O0 -S -c JDK-8205969.cpp -o JDK-8205969-unopt-8.0.s
$ clang++ -O0 -c JDK-8205969.cpp -o JDK-8205969-unopt-8.0.o
$ clang++ -o JDK-8205969-unopt-8.0.out JDK-8205969-unopt-8.0.o

$ ./JDK-8205969-unopt-8.0.out

No segfault

$ cat JDK-8205969-unopt-8.0.s
.text
.file "JDK-8205969.cpp"
.section .text.startup,"ax",@progbits
.p2align 4, 0x90 # -- Begin function __cxx_global_var_init
.type __cxx_global_var_init,@function
__cxx_global_var_init: # @__cxx_global_var_init
.cfi_startproc
# %bb.0:
pushq %rbp
.cfi_def_cfa_offset 16
.cfi_offset %rbp, -16
movq %rsp, %rbp
.cfi_def_cfa_register %rbp
movabsq $_ZN15NativeCallStack11EMPTY_STACKE, %rdi
callq _ZN15NativeCallStackC2Ev
popq %rbp
.cfi_def_cfa %rsp, 8
retq
.Lfunc_end0:
.size __cxx_global_var_init, .Lfunc_end0-__cxx_global_var_init
.cfi_endproc
# -- End function
.section .text._ZN15NativeCallStackC2Ev,"axG",@progbits,_ZN15NativeCallStackC2Ev,comdat
.weak _ZN15NativeCallStackC2Ev # -- Begin function _ZN15NativeCallStackC2Ev
.p2align 4, 0x90
.type _ZN15NativeCallStackC2Ev,@function
_ZN15NativeCallStackC2Ev: # @_ZN15NativeCallStackC2Ev
.cfi_startproc
# %bb.0:
pushq %rbp
.cfi_def_cfa_offset 16
.cfi_offset %rbp, -16
movq %rsp, %rbp
.cfi_def_cfa_register %rbp
movq %rdi, -8(%rbp)
movq -8(%rbp), %rdi
movl $0, -12(%rbp)
movq %rdi, -24(%rbp) # 8-byte Spill
.LBB1_1: # =>This Inner Loop Header: Depth=1
cmpl $4, -12(%rbp)
jge .LBB1_4
# %bb.2: # in Loop: Header=BB1_1 Depth=1
movslq -12(%rbp), %rax
movq -24(%rbp), %rcx # 8-byte Reload
movq $0, (%rcx,%rax,8)
# %bb.3: # in Loop: Header=BB1_1 Depth=1
movl -12(%rbp), %eax
addl $1, %eax
movl %eax, -12(%rbp)
jmp .LBB1_1
.LBB1_4:
popq %rbp
.cfi_def_cfa %rsp, 8
retq
.Lfunc_end1:
.size _ZN15NativeCallStackC2Ev, .Lfunc_end1-_ZN15NativeCallStackC2Ev
.cfi_endproc
# -- End function
.text
.globl main # -- Begin function main
.p2align 4, 0x90
.type main,@function
main: # @main
.cfi_startproc
# %bb.0:
pushq %rbp
.cfi_def_cfa_offset 16
.cfi_offset %rbp, -16
movq %rsp, %rbp
.cfi_def_cfa_register %rbp
subq $16, %rsp
movl $0, -4(%rbp)
movabsq $_ZN15NativeCallStack11EMPTY_STACKE, %rdi
callq _ZN15NativeCallStackC2Ev
xorl %eax, %eax
addq $16, %rsp
popq %rbp
.cfi_def_cfa %rsp, 8
retq
.Lfunc_end2:
.size main, .Lfunc_end2-main
.cfi_endproc
# -- End function
.section .text.startup,"ax",@progbits
.p2align 4, 0x90 # -- Begin function _GLOBAL__sub_I_JDK_8205969.cpp
.type _GLOBAL__sub_I_JDK_8205969.cpp,@function
_GLOBAL__sub_I_JDK_8205969.cpp: # @_GLOBAL__sub_I_JDK_8205969.cpp
.cfi_startproc
# %bb.0:
pushq %rbp
.cfi_def_cfa_offset 16
.cfi_offset %rbp, -16
movq %rsp, %rbp
.cfi_def_cfa_register %rbp
callq __cxx_global_var_init
popq %rbp
.cfi_def_cfa %rsp, 8
retq
.Lfunc_end3:
.size _GLOBAL__sub_I_JDK_8205969.cpp, .Lfunc_end3-_GLOBAL__sub_I_JDK_8205969.cpp
.cfi_endproc
# -- End function
.type _ZN15NativeCallStack11EMPTY_STACKE,@object # @_ZN15NativeCallStack11EMPTY_STACKE
.bss

     ^\-\-\- R/W segment

     \.globl  \_ZN15NativeCallStack11EMPTY\_STACKE
     \.p2align        3

_ZN15NativeCallStack11EMPTY_STACKE:
.zero 32
.size _ZN15NativeCallStack11EMPTY_STACKE, 32

     \.section        \.init\_array,&quot;aw&quot;,@init\_array
     \.p2align        3
     \.quad   \_GLOBAL\_\_sub\_I\_JDK\_8205969\.cpp

     \.ident  &quot;LLVM China clang version 8\.0\.0 \(git@github\.com:llvm\-mirror/clang\.git 81ef98628ebf5186d746c0986dcbf5073e842043\) \(git@github\.com:llvm\-mirror/llvm\.git e1aac9723d55497e74d83d216329f08d9842e494\) \(based on LLVM 8\.0\.0svn\)&quot;
     \.section        &quot;\.note\.GNU\-stack&quot;,&quot;&quot;,@progbits
     \.addrsig
     \.addrsig\_sym \_\_cxx\_global\_var\_init
     \.addrsig\_sym \_GLOBAL\_\_sub\_I\_JDK\_8205969\.cpp
     \.addrsig\_sym \_ZN15NativeCallStack11EMPTY\_STACKE

----- 8< -------- 8< -------- 8< -------- 8< -------- 8< -------- 8< ---

Furthermore clang-3.9.1 *optimized* for X86 is *not* able to reproduce the issue neither. It behaviors like gcc-6.4.1 and gcc-8. But I will also check out clang-8 -O3/-O0 for mips64el.

And I am building LLVM 5.0.2 for X86, if still *not* able to reproduce, it might be reduced the issue to [6, +∞) version range. I will dig deep-into which commit/review bring in the issue.

https://openjdk.markmail.org/thread/rwfcd6df6vhzli5m

Reported https://bugs.llvm.org/show_bug.cgi?id=38911

Thanks,

Leslie Zhai

I tend to think that clang's optimization was valid, and if placing
data into .rodata has any value at all (why else does .rodata exist?)
then the optimization is valuable.

Hi Martin,

Thanks for your response!

I tend to think that clang's optimization was valid, and if placing
data into .rodata has any value at all (why else does .rodata exist?)

placing data into .rodata is *not* wrong! But it needs to double check the setter of isReadOnly() if condition in the llvm/lib/Analysis/MemoryDependenceAnalysis.cpp and llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp
Thanks,
Leslie Zhai

Hi Leslie,

Hi Martin,

Thanks for your response!

I tend to think that clang's optimization was valid, and if placing
data into .rodata has any value at all (why else does .rodata exist?)

placing data into .rodata is *not* wrong! But it needs to double check the
setter of isReadOnly() if condition in the
llvm/lib/Analysis/MemoryDependenceAnalysis.cpp and
llvm/lib/CodeGen/TargetLoweringObjectFileImpl.cpp

I'm not a compiler engineer but .... The bug report should make it
clear why the new clang behavior is wrong. I want const globals to be
put into .rodata whenever possible - I WANT to get a segmentation
fault when placement new is used with the address of a const global.

The code in question has an undefined behavior per C++ standard. And
the transformation is certainly valid.