Various strange errors while debugging PostgreSQL

Hello

Here is what I do. In one terminal I connect to PostgreSQL using psql
command. In another terminal I determine pid of a new PostgreSQL
process (backend process that handles a new connection) and run:

lldb-3.7.1 -p (pid)

Then:

b heap_open
c

In first terminal:

create temporary table tt1(x int);

In second terminal:

   1290	{
   1291		Relation	r;
   1292	
-> 1293		r = relation_open(relationId, lockmode);
   1294	
   1295		if (r->rd_rel->relkind == RELKIND_INDEX)
   1296			ereport(ERROR,
(lldb) p r
(Relation) $0 = 0x00000002018a70c8
(lldb) p r->rd_rel
error: Couldn't apply expression side effects : Couldn't dematerialize
   a result variable: couldn't read its memory

This is a first issue. The second issue is:

(lldb) process save-core /tmp/1.core
error: Failed to save core file for process: no ObjectFile plugins were
able to save a core for this process

LLDB was compiled manually from RELEASE_371/final tag using clang 3.6:

cmake ../llvm -G Ninja -DLLDB_DISABLE_CURSES:BOOL=TRUE \
  -DCMAKE_C_COMPILER=/usr/bin/clang-3.6 \
  -DCMAKE_CXX_COMPILER=/usr/bin/clang++-3.6 \
  -DCMAKE_BUILD_TYPE=Release
ninja -j2
sudo ninja install

It's Ubuntu Linux 14.04 LTS:

$ uname -a

Linux pgsql-dev-ubuntu 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11
00:25:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Same issues with lldb-3.6 installed using apt-get.

`process save-core` doesn't work in lldb-3.8 compiled using ports on
FreeBSD 10.2 as well. Unfortunately I can't tell if first bug reproduces
on FreeBSD because of another bug I reported previously:

https://llvm.org/bugs/show_bug.cgi?id=26924#c3

Anyway are these some sort of well-know issues that could be bypassed
somehow or brand new bugs? I can also provide any other debug
information you might need.

(lldb) p r->rd_rel
error: Couldn't apply expression side effects : Couldn't dematerialize
   a result variable: couldn't read its memory

OK, this issue is fixed in 3.8.0. Two others are not.

Hello

Here is what I do. In one terminal I connect to PostgreSQL using psql
command. In another terminal I determine pid of a new PostgreSQL
process (backend process that handles a new connection) and run:

lldb-3.7.1 -p (pid)

Then:

b heap_open
c

In first terminal:

create temporary table tt1(x int);

In second terminal:

  1290	{
  1291		Relation	r;
  1292	
-> 1293		r = relation_open(relationId, lockmode);
  1294	
  1295		if (r->rd_rel->relkind == RELKIND_INDEX)
  1296			ereport(ERROR,
(lldb) p r
(Relation) $0 = 0x00000002018a70c8
(lldb) p r->rd_rel
error: Couldn't apply expression side effects : Couldn't dematerialize
  a result variable: couldn't read its memory

You might not realize that the value in "r" is random junk. You haven't stepped over the line that actually assigns a value to "r" yet. So yes, your expression will probably fail. Your expression turns out to be really easy: take a pointer and deref a value, so this is emulated by evaluating llvm IR and when we try to deref "r + offset of rd_rel", it fails.

This is a first issue. The second issue is:

(lldb) process save-core /tmp/1.core
error: Failed to save core file for process: no ObjectFile plugins were
able to save a core for this process

What architecture is this? I believe people added code to save core files for ELF based processes, but not all architectures are supported. Looking in:

lldb/source/Plugins/Process/elf-core

I see the following files:

CMakeLists.txt
ProcessElfCore.cpp
ProcessElfCore.h
RegisterContextPOSIXCore_arm.cpp
RegisterContextPOSIXCore_arm.h
RegisterContextPOSIXCore_arm64.cpp
RegisterContextPOSIXCore_arm64.h
RegisterContextPOSIXCore_mips64.cpp
RegisterContextPOSIXCore_mips64.h
RegisterContextPOSIXCore_powerpc.cpp
RegisterContextPOSIXCore_powerpc.h
RegisterContextPOSIXCore_x86_64.cpp
RegisterContextPOSIXCore_x86_64.h
ThreadElfCore.cpp
ThreadElfCore.h

So I am guessing these are the only architectures that are supported. I am guessing you might be on an i386 system?

So I am guessing these are the only architectures that are supported.
I am guessing you might be on an i386 system?

No I'm afraid it's x86_64. File RegisterContextPOSIXCore_x86_64.cpp is
present in source tree I used to build lldb.

This bug is very easy to reproduce.

0. You need Ubuntu 14.04 LTS x64 with latest updates
1. Create a new LXC container to exclude any possible side effects from
other installed applications
2. Build LLDB using this script: Ubuntu Pastebin
3. Run `sudo ninja install`
4. Create t.c:

#include <stdio.h>

int main()
{
  printf("Hello!\n");
  return 0;
}

5. Run ` /usr/local/bin/clang-3.8 t.c -o t ` (or ... -g - it doesn't
matter)
6. Then:

$ ./t
Hello!
$ lldb ./t
(lldb) target create "./t"
Current executable set to './t' (x86_64).
(lldb) b main
Breakpoint 1: where = t`main, address = 0x0000000000400530
(lldb) r
Process 24974 launching
Process 24974 launched: './t' (x86_64)
Process 24974 stopped
* thread #1: tid = 24974, 0x0000000000400530 t`main, name = 't', stop
reason = breakpoint 1.1 frame #0: 0x0000000000400530 t`main
-> 0x400530 <main>: pushq %rbp
   0x400531 <main+1>: movq %rsp, %rbp
   0x400534 <main+4>: subq $0x10, %rsp
   0x400538 <main+8>: movabsq $0x4005e4, %rdi
(lldb) process save-core /tmp/1.core

error: Failed to save core file for process: no ObjectFile plugins were
able to save a core for this process

(lldb) ^D

The files I posted before are for reading core files, not generating them. The ObjectFile subclasses are what are responsible for creating a core file of a specific type.

ObjectFileELF doesn't support creating core files as we can see by the functions it registers with the plug-in manager:

ObjectFileELF::Initialize()
{
    PluginManager::RegisterPlugin(GetPluginNameStatic(),
                                  GetPluginDescriptionStatic(),
                                  CreateInstance,
                                  CreateMemoryInstance,
                                  GetModuleSpecifications);
}

Note that ObjectFileMachO does (see the extra SaveCore function it registers):

void
ObjectFileMachO::Initialize()
{
    PluginManager::RegisterPlugin (GetPluginNameStatic(),
                                   GetPluginDescriptionStatic(),
                                   CreateInstance,
                                   CreateMemoryInstance,
                                   GetModuleSpecifications,
                                   SaveCore);
}

So it looks like this is not supported. If you feel like implementing this, let me know as I did the ObjectFileMachO::SaveCore() implementation and am familiar with what goes into it.

Greg