xxchan
April 9, 2023, 9:55am
1
segfault backtrace:
* thread #10, name = 'risingwave-main', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x00000001944f6770 libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseFDEInstructions(libunwind::LocalAddressSpace&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info const&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info const&, unsigned long, int, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::PrologInfo*) + 204
libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseFDEInstructions:
-> 0x1944f6770 <+204>: ldrb w8, [x28], #0x1
0x1944f6774 <+208>: stur x28, [x29, #-0x98]
0x1944f6778 <+212>: cmp w8, #0x2f
0x1944f677c <+216>: b.hi 0x1944f72c4 ; <+3104>
Target 0: (risingwave) stopped.
(lldb) bt
* thread #10, name = 'risingwave-main', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
* frame #0: 0x00000001944f6770 libunwind.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseFDEInstructions(libunwind::LocalAddressSpace&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info const&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info const&, unsigned long, int, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::PrologInfo*) + 204
frame #1: 0x00000001944f6624 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::getInfoFromFdeCie(libunwind::CFI_Parser<libunwind::LocalAddressSpace>::FDE_Info const&, libunwind::CFI_Parser<libunwind::LocalAddressSpace>::CIE_Info const&, unsigned long, unsigned long) + 100
frame #2: 0x00000001944f62fc libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::getInfoFromDwarfSection(unsigned long, libunwind::UnwindInfoSections const&, unsigned int) + 184
frame #3: 0x00000001944f6220 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::setInfoBasedOnIPRegister(bool) + 1228
frame #4: 0x00000001944f86b0 libunwind.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step() + 696
frame #5: 0x00000001944fb0f0 libunwind.dylib`_Unwind_Backtrace + 348
frame #6: 0x00000001090822cc risingwave`std::backtrace::Backtrace::create::h908375f7f84cb508 [inlined] std::backtrace_rs::backtrace::libunwind::trace::h471a59e08ff9e5dc at mod.rs:66:5 [opt]
frame #7: 0x00000001090822bc risingwave`std::backtrace::Backtrace::create::h908375f7f84cb508 [inlined] std::backtrace_rs::backtrace::trace_unsynchronized::h4e694232d85e2708 at mod.rs:66:5 [opt]
frame #8: 0x00000001090822b0 risingwave`std::backtrace::Backtrace::create::h908375f7f84cb508 at backtrace.rs:333:13 [opt]
frame #9: 0x00000001056e7b5c risingwave`_$LT$risingwave_meta..error..MetaError$u20$as$u20$core..convert..From$LT$risingwave_meta..error..MetaErrorInner$GT$$GT$::from::hb4b62fbc8685e728(inner=<unavailable>) at error.rs:69:18
frame #10: 0x0000000105e86adc risingwave`_$LT$T$u20$as$u20$core..convert..Into$LT$U$GT$$GT$::into::h7837bc8fb77e8181(self=<unavailable>) at mod.rs:726:9
frame #11: 0x00000001056e7fe4 risingwave`risingwave_meta::error::MetaError::permission_denied::h6592a4f64415a283(s=<unavailable>) at error.rs:103:9
More context here https://github.com/rust-lang/rust/issues/104388
xxchan
April 10, 2023, 10:15am
2
I compiled libunwind from source (llvm-project/libunwind at 01025ddd6264737ebf617bd754173969af34243f · llvm/llvm-project · GitHub ), and here’s the backtrace:
* thread #4, name = 'risingwave-main', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x000000011c86eed4 libunwind.1.0.dylib`libunwind::LocalAddressSpace::get8(this=0x000000011c87c9b1, addr=0) at AddressSpace.hpp:165:5
162 typedef intptr_t sint_t;
163 uint8_t get8(pint_t addr) {
164 uint8_t val;
-> 165 memcpy(&val, (void *)addr, sizeof(val));
166 return val;
167 }
168 uint16_t get16(pint_t addr) {
Target 0: (risingwave) stopped.
(lldb) bt
* thread #4, name = 'risingwave-main', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
* frame #0: 0x000000011c86eed4 libunwind.1.0.dylib`libunwind::LocalAddressSpace::get8(this=0x000000011c87c9b1, addr=0) at AddressSpace.hpp:165:5
frame #1: 0x000000011c86d054 libunwind.1.0.dylib`libunwind::CFI_Parser<libunwind::LocalAddressSpace>::parseFDEInstructions(addressSpace=0x000000011c87c9b1, fdeInfo=0x0000000170407788, cieInfo=0x0000000170407750, upToPC=4395568915, arch=4, results=0x00000001704070c8) at DwarfParser.hpp:459:37
frame #2: 0x000000011c872440 libunwind.1.0.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::getInfoFromFdeCie(this=0x0000000170407c50, fdeInfo=0x0000000170407788, cieInfo=0x0000000170407750, pc=4395568915, dso_base=4294967296) at UnwindCursor.hpp:1614:7
frame #3: 0x000000011c8721a0 libunwind.1.0.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::getInfoFromDwarfSection(this=0x0000000170407c50, pc=4395568915, sects=0x00000001704078e0, fdeSectionOffsetHint=4675328) at UnwindCursor.hpp:1674:9
frame #4: 0x000000011c86babc libunwind.1.0.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::setInfoBasedOnIPRegister(this=0x0000000170407c50, isReturnAddress=true) at UnwindCursor.hpp:2549:21
frame #5: 0x000000011c86b89c libunwind.1.0.dylib`libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_arm64>::step(this=0x0000000170407c50, stage2=false) at UnwindCursor.hpp:2848:11
frame #6: 0x000000011c869d44 libunwind.1.0.dylib`::__unw_step(cursor=0x0000000170407c50) at libunwind.cpp:182:14
frame #7: 0x000000011c875158 libunwind.1.0.dylib`_Unwind_Backtrace(callback=(risingwave`std::backtrace_rs::backtrace::libunwind::trace::trace_fn::h77a2b5d5763fd5d7 at libunwind.rs:95), ref=0x0000000170408150) at UnwindLevel1-gcc-ext.c:156:9
frame #8: 0x00000001090852cc risingwave`std::backtrace::Backtrace::create::h908375f7f84cb508 [inlined] std::backtrace_rs::backtrace::libunwind::trace::h471a59e08ff9e5dc at mod.rs:66:5 [opt]
frame #9: 0x00000001090852bc risingwave`std::backtrace::Backtrace::create::h908375f7f84cb508 [inlined] std::backtrace_rs::backtrace::trace_unsynchronized::h4e694232d85e2708 at mod.rs:66:5 [opt]
frame #10: 0x00000001090852b0 risingwave`std::backtrace::Backtrace::create::h908375f7f84cb508 at backtrace.rs:333:13 [opt]
I’m not sure whether it’s rustc or lld or libunwind (or any other stuff)'s problem. Any suggestion about how to diagnose such issues would be appreciated.
xxchan
April 10, 2023, 12:27pm
3
(in parseFDEInstructions)
(ParseInfo[2]) parseInfoArray = {
[0] = (instructions = 0, instructionsEnd = 4467765419, pcoffset = 18446744073709551615)
[1] = (instructions = 4468701376, instructionsEnd = 7423526062, pcoffset = 4206506293)
}
This instructions = 0 leads to the segfault
It is cieInfo.cieInstructions
cieInfo = 0x0000000170407750: {
cieStart = 4467765419
cieLength = 0
cieInstructions = 0
pointerEncoding = '\0'
lsdaEncoding = '\xff'
personalityEncoding = '\0'
personalityOffsetInCIE = '\0'
personality = 0
codeAlignFactor = 0
dataAlignFactor = 0
isSignalFrame = false
fdesHaveAugmentationData = false
returnAddressRegister = '\0'
addressesSignedWithBKey = false
mteTaggedFrame = false
}
xxchan
April 10, 2023, 1:09pm
4
BTW, my setup to reproduce the issue:
git clone https://github.com/risingwavelabs/risingwave && cd risingwave
git checkout db6691b
./risedev p
# In a different terminal
psql -h localhost -p 4566 -d dev -U root
dev=> create table t(x int);create view v as select * from t;drop table t;
Also since this happens randomly, my environment is mac M1
❯ cc --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.3.0
Alternatively, the risingwave commit 7e0c548 SEGFAULTS immediately due to the same problem. In this version, we used homebrew’s LLVM (15/16)'s lld. (Because we had suspected the issue is caused by apple’s llvm 14’s lld’s issue, but it turned out not the case.)
[target.aarch64-apple-darwin]
rustflags = [
"-Clink-arg=-fuse-ld=/opt/homebrew/opt/llvm/bin/ld64.lld"
]
