Difficulty in using target.source-map to map source files

Hi All,

Below is a sample example, where target.source-map seems to have a limitation. The limitation seems to be because

  1. lldb does not have an equivalent command like directory in gdb
  2. target.source-map does not accept multiple mapping entries. I checked this only with freebsd.

(lldb) settings show target.source-map
target.source-map (path-map) =
[0] “/home/karnajitw/lldb_test_execs/test_source_line1” → “/u/test/test_source_line1”

  1. Haven’t checked in the code yet, but if we see the mappings of scenario 1, they all point to a single real path /home/karnajitw/lldb_test_execs/test_source_line1. But looks like the mapping logic only considers strings into account. But, at the same time, I am not claiming that they should be interpreted as path from a different machine during the mapping.

I want to check on this issue in-depth. But before that, want to confirm if this is real issue or there are other ways to deal these scenarios which I am not aware of?

I am referring below link for the lldb commands.
https://lldb.llvm.org/lldb-gdb.html

  1. First scenario: Different souce file path representation

[/home/karnajitw/lldb_test_execs/test_source_line1] $ clang -O0 -g /home/karnajitw/lldb_test_execs///test_source_line1/main.c /home/karnajitw/lldb_test_execs/…/lldb_test_execs/test_source_line1/a/ainc.c /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

Machine 1: /home/karnajitw/lldb_test_execs/test_source_line1 → Machine 2: /u/test/test_source_line1

test_source_line1

– a

– ainc.c
– ainc.h
-- b -- binc.c – binc.h
– a.out
`-- main.c

% ./lldb test_source_line1/a.out

(lldb) target create “test_source_line1/a.out”
Current executable set to ‘test_source_line1/a.out’ (x86_64).
(lldb) l main

File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/…/lldb_test_execs/test_source_line1/a/ainc.c
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

(lldb) settings set target.source-map /home/karnajitw/lldb_test_execs///test_source_line1 /u/test/test_source_line1

(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
1 #include “a/ainc.h”
2
3 int main()
4 {
5 afn();
6
7 bfn();
8
9 return 0;
10 }
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/…/lldb_test_execs/test_source_line1/a/ainc.c
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

(lldb) settings set target.source-map /home/karnajitw/lldb_test_execs/…/lldb_test_execs/test_source_line1 /u/test/test_source_line1

(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/…/lldb_test_execs/test_source_line1/a/ainc.c
1 #include <stdio.h>
2 #include “ainc.h”
3
4 void afn()
5 {
6 printf(“Hello this is afn…\n”);
7 }
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

(lldb) settings set target.source-map /home/karnajitw/lldb_test_execs/test_source_line1 /u/test/test_source_line1

(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/…/lldb_test_execs/test_source_line1/a/ainc.c
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
1 #include <stdio.h>
2 #include “binc.h”
3
4 void bfn()
5 {
6 printf(“Hello this is bfn…\n”);
7 }

  1. Scenario 2: Deep directory structure

/obj/a/b/c/d/e/app/sub/…/…/…/…/…/…/…/…/src/a/b/c/d/e/app/sub/file
/obj/a/b/c/d/e/app/…/…/…/…/…/…/…/src/a/b/c/d/e/app/file

  • If we carry the copy the source file to machine 2, we cannot easily map the source file without creating dummy /obj/a/b/c/d/e/f/g.
  1. Scenario 3: External libraries
  • I haven’t exactly tested this yet. But I belive in scenario too we might need to change the source-map.

Please look into this and guide me for the same.

Regards,
Karan

Hi All,

Below is a sample example, where target.source-map seems to have a limitation. The limitation seems to be because
1. lldb does not have an equivalent command like directory in gdb

The gdb "dir" command seemed always to be more annoying than useful in real situations. If your project had any complexity in the directory structure you ended up having to add dir commands for all the subdirectories, which got tedious quickly. Since you pretty much always move your sources around rigidly, "source maps" are a more natural way to specify this.

Note, we could add a recursive "dir" command, but you can't do the simpleminded recursive search or you'll mess up when the sources have same named files in different directories. Because of this, again the source maps seem more suited.

2. target.source-map does not accept multiple mapping entries. I checked this only with freebsd.

(lldb) settings show target.source-map
target.source-map (path-map) =
[0] "/home/karnajitw/lldb_test_execs/test_source_line1" -> "/u/test/test_source_line1"

That is not correct:

(lldb) settings set target.source-map /some/source/path /tmp /some/other/source/path /tmp
(lldb) settings show target.source-map
target.source-map (path-map) =
[0] "/some/source/path" -> "/tmp"
[1] "/some/other/source/path" -> "/tmp"

or

(lldb) set set target.source-map /some/source/path /tmp
(lldb) set append target.source-map /some/other/source/path /tmp
(lldb) set show target.source-map
target.source-map (path-map) =
[0] "/some/source/path" -> "/tmp"
[1] "/some/other/source/path" -> "/tmp"

3. Haven't checked in the code yet, but if we see the mappings of scenario 1, they all point to a single real path /home/karnajitw/lldb_test_execs/test_source_line1. But looks like the mapping logic only considers strings into account. But, at the same time, I am not claiming that they should be interpreted as path from a different machine during the mapping.

I'm not sure what you mean here. lldb does what it can to unwind the source path, but it doesn't assume that those paths actually exist locally, so the most it can do is remove redundant "/"-s and "/./"-s and unwind ".."-s. We do this before most source path comparisons (it's part of the FileSpec Equals method). We've been tinkering with this over time so make sure you are using a recent version of lldb. If there are places where we don't do this, then that's easily fixed.

I want to check on this issue in-depth. But before that, want to confirm if this is real issue or there are other ways to deal these scenarios which I am not aware of?

I am referring below link for the lldb commands.
https://lldb.llvm.org/lldb-gdb.html

1. First scenario: Different souce file path representation

[/home/karnajitw/lldb_test_execs/test_source_line1] $ clang -O0 -g /home/karnajitw/lldb_test_execs///test_source_line1/main.c /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

Machine 1: /home/karnajitw/lldb_test_execs/test_source_line1 -> Machine 2: /u/test/test_source_line1

test_source_line1
>-- a
> >-- ainc.c
> >-- ainc.h
> `-- b
> >-- binc.c
> `-- binc.h
>-- a.out
`-- main.c

% ./lldb test_source_line1/a.out

(lldb) target create "test_source_line1/a.out"
Current executable set to 'test_source_line1/a.out' (x86_64).
(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

(lldb) settings set target.source-map /home/karnajitw/lldb_test_execs///test_source_line1 /u/test/test_source_line1

(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
   1 #include "a/ainc.h"
   2
   3 int main()
   4 {
   5 afn();
   6
   7 bfn();
   8
   9 return 0;
   10 }
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

(lldb) settings set target.source-map /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1 /u/test/test_source_line1

(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
   1 #include <stdio.h>
   2 #include "ainc.h"
   3
   4 void afn()
   5 {
   6 printf("Hello this is afn...\n");
   7 }
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c

(lldb) settings set target.source-map /home/karnajitw/lldb_test_execs/test_source_line1 /u/test/test_source_line1

(lldb) l main
File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
(lldb) l afn
File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
(lldb) l bfn
File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
   1 #include <stdio.h>
   2 #include "binc.h"
   3
   4 void bfn()
   5 {
   6 printf("Hello this is bfn...\n");
   7 }

I can't test the "///" part of this, clang seems to always collapse these for me. But with a recent lldb, I see:

$ clang -g -O0 -c '/tmp/sources/build/../files/hello.c'
$ dwarfdump hello.o
...
0x0000000b: TAG_compile_unit [1] *
             AT_producer( "Apple LLVM version 9.0.0 (clang-900.0.31)" )
             AT_language( DW_LANG_C99 )
             AT_name( "/tmp/sources/build/../files/hello.c" )

So we did get a compile unit name with .. in it, but:

lldbrulez:/tmp/sources/build > clang -g -O0 -o hello hello.o
lldbrulez:/tmp/sources/build > lldb
(lldb) file hello
Current executable set to 'hello' (x86_64).
(lldb) source list -n main
File: /tmp/sources/build/../files/hello.c
   1 #include <stdio.h>
   2
   3 int
   4 main()
   5 {
   6 printf ("Hello there.\n");
   7 return 0;
   8 }

So we do unwind the ..'s in this case. This example doesn't have source-maps, but the same thing works if I move the sources and add just a <build-top> -> <debug-top> mapping. As I say, we have been working these ".." aware comparisons through all the file compares recently, so you may just need to get a newer lldb.

2. Scenario 2: Deep directory structure

<top>/obj/a/b/c/d/e/app/sub/../../../../../../../../src/a/b/c/d/e/app/sub/file
<top>/obj/a/b/c/d/e/app/../../../../../../../src/a/b/c/d/e/app/file

- If we carry the copy the source file to machine 2, we cannot easily map the source file without creating dummy <top>/obj/a/b/c/d/e/f/g.

You should only need to map <top> to <top>. The only reason you would have to start specifying subdirectories is you have ones specified with ".."-s, but that problem is properly handled by making the file comparisons aware of backup operators. So if the comparisons are working right (again check a more recent lldb) you should not need to deal with all these dots.

3. Scenario 3: External libraries
- I haven't exactly tested this yet. But I belive in scenario too we might need to change the source-map.

External libraries should be no different from your project. I don't see anything different needed here.

Jim

Hi Jim,

Thanks for the valuable reply. Please find my comments inline.

Hi Jim,

Thanks for the valuable reply. Please find my comments inline.

Regards,
Karan

>
> Hi All,
>
> Below is a sample example, where target.source-map seems to have a limitation. The limitation seems to be because
> 1. lldb does not have an equivalent command like directory in gdb

The gdb "dir" command seemed always to be more annoying than useful in real situations. If your project had any complexity in the directory structure you ended up having to add dir commands for all the subdirectories, which got tedious quickly. Since you pretty much always move your sources around rigidly, "source maps" are a more natural way to specify this.

Note, we could add a recursive "dir" command, but you can't do the simpleminded recursive search or you'll mess up when the sources have same named files in different directories. Because of this, again the source maps seem more suited.

-- Really a good info to know. Thanks!

> 2. target.source-map does not accept multiple mapping entries. I checked this only with freebsd.
>
> (lldb) settings show target.source-map
> target.source-map (path-map) =
> [0] "/home/karnajitw/lldb_test_execs/test_source_line1" -> "/u/test/test_source_line1"

That is not correct:

(lldb) settings set target.source-map /some/source/path /tmp /some/other/source/path /tmp
(lldb) settings show target.source-map
target.source-map (path-map) =
[0] "/some/source/path" -> "/tmp"
[1] "/some/other/source/path" -> "/tmp"

or

(lldb) set set target.source-map /some/source/path /tmp
(lldb) set append target.source-map /some/other/source/path /tmp
(lldb) set show target.source-map
target.source-map (path-map) =
[0] "/some/source/path" -> "/tmp"
[1] "/some/other/source/path" -> "/tmp"

- Thanks! This helped.

>
> 3. Haven't checked in the code yet, but if we see the mappings of scenario 1, they all point to a single real path /home/karnajitw/lldb_test_execs/test_source_line1. But looks like the mapping logic only considers strings into account. But, at the same time, I am not claiming that they should be interpreted as path from a different machine during the mapping.
>

I'm not sure what you mean here. lldb does what it can to unwind the source path, but it doesn't assume that those paths actually exist locally, so the most it can do is remove redundant "/"-s and "/./"-s and unwind ".."-s. We do this before most source path comparisons (it's part of the FileSpec Equals method). We've been tinkering with this over time so make sure you are using a recent version of lldb. If there are places where we don't do this, then that's easily fixed.

- Your answer is in line with what I wanted to know. My lldb should be quite recent. But I used it only for freebsd currently.

% ./lldb --version
lldb version 5.0.0 (http://llvm.org/svn/llvm-project/lldb/trunk revision 305778)
  clang revision 305546
  llvm revision 305548

> I want to check on this issue in-depth. But before that, want to confirm if this is real issue or there are other ways to deal these scenarios which I am not aware of?
>
> I am referring below link for the lldb commands.
> https://lldb.llvm.org/lldb-gdb.html
>
> 1. First scenario: Different souce file path representation
>
> [/home/karnajitw/lldb_test_execs/test_source_line1] $ clang -O0 -g /home/karnajitw/lldb_test_execs///test_source_line1/main.c /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
>
> Machine 1: /home/karnajitw/lldb_test_execs/test_source_line1 -> Machine 2: /u/test/test_source_line1
>
> test_source_line1
> >-- a
> > >-- ainc.c
> > >-- ainc.h
> > `-- b
> > >-- binc.c
> > `-- binc.h
> >-- a.out
> `-- main.c
>
> % ./lldb test_source_line1/a.out
>
> (lldb) target create "test_source_line1/a.out"
> Current executable set to 'test_source_line1/a.out' (x86_64).
> (lldb) l main
> File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
> (lldb) l afn
> File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
> (lldb) l bfn
> File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
>
> (lldb) settings set target.source-map /home/karnajitw/lldb_test_execs///test_source_line1 /u/test/test_source_line1
>
> (lldb) l main
> File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
> 1 #include "a/ainc.h"
> 2
> 3 int main()
> 4 {
> 5 afn();
> 6
> 7 bfn();
> 8
> 9 return 0;
> 10 }
> (lldb) l afn
> File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
> (lldb) l bfn
> File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
>
> (lldb) settings set target.source-map /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1 /u/test/test_source_line1
>
> (lldb) l main
> File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
> (lldb) l afn
> File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
> 1 #include <stdio.h>
> 2 #include "ainc.h"
> 3
> 4 void afn()
> 5 {
> 6 printf("Hello this is afn...\n");
> 7 }
> (lldb) l bfn
> File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
>
> (lldb) settings set target.source-map /home/karnajitw/lldb_test_execs/test_source_line1 /u/test/test_source_line1
>
> (lldb) l main
> File: /home/karnajitw/lldb_test_execs///test_source_line1/main.c
> (lldb) l afn
> File: /home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1/a/ainc.c
> (lldb) l bfn
> File: /home/karnajitw/lldb_test_execs/test_source_line1/a/b/binc.c
> 1 #include <stdio.h>
> 2 #include "binc.h"
> 3
> 4 void bfn()
> 5 {
> 6 printf("Hello this is bfn...\n");
> 7 }

I can't test the "///" part of this, clang seems to always collapse these for me. But with a recent lldb, I see:

$ clang -g -O0 -c '/tmp/sources/build/../files/hello.c'
$ dwarfdump hello.o
...
0x0000000b: TAG_compile_unit [1] *
             AT_producer( "Apple LLVM version 9.0.0 (clang-900.0.31)" )
             AT_language( DW_LANG_C99 )
             AT_name( "/tmp/sources/build/../files/hello.c" )

So we did get a compile unit name with .. in it, but:

lldbrulez:/tmp/sources/build > clang -g -O0 -o hello hello.o
lldbrulez:/tmp/sources/build > lldb
(lldb) file hello
Current executable set to 'hello' (x86_64).
(lldb) source list -n main
File: /tmp/sources/build/../files/hello.c
   1 #include <stdio.h>
   2
   3 int
   4 main()
   5 {
   6 printf ("Hello there.\n");
   7 return 0;
   8 }

So we do unwind the ..'s in this case. This example doesn't have source-maps, but the same thing works if I move the sources and add just a <build-top> -> <debug-top> mapping. As I say, we have been working these ".." aware comparisons through all the file compares recently, so you may just need to get a newer lldb.

- Lets say in my above example, the function is this way
main -> afn -> bfn

Now, I breakpoint on bfn, now to get source list on all the 3 frames, I will need this below mapping

(lldb)
target.source-map (path-map) =
[0] "/home/karnajitw/lldb_test_execs///test_source_line1" -> "/u/test/test_source_line1"
[1] "/home/karnajitw/lldb_test_execs/../lldb_test_execs/test_source_line1" -> "/u/test/test_source_line1"
[2] "/home/karnajitw/lldb_test_execs/test_source_line1" -> "/u/test/test_source_line1"

Like the previous discussion, All the LHS and RHS actually refers to the same path. So why we need 3 mappings here? But, as I have mentioned previously the problem could be just in the freebsd version also.

>
> 2. Scenario 2: Deep directory structure
>
> <top>/obj/a/b/c/d/e/app/sub/../../../../../../../../src/a/b/c/d/e/app/sub/file
> <top>/obj/a/b/c/d/e/app/../../../../../../../src/a/b/c/d/e/app/file
>
> - If we carry the copy the source file to machine 2, we cannot easily map the source file without creating dummy <top>/obj/a/b/c/d/e/f/g.
>
You should only need to map <top> to <top>. The only reason you would have to start specifying subdirectories is you have ones specified with ".."-s, but that problem is properly handled by making the file comparisons aware of backup operators. So if the comparisons are working right (again check a more recent lldb) you should not need to deal with all these dots.

- Checked with latest lldb build, its the same

(lldb) version
lldb version 6.0.0 (http://llvm.org/svn/llvm-project/lldb/trunk revision 311075)
  clang revision 311072
  llvm revision 311071

$ build/bin/lldb a.out
(lldb) target create "a.out"
Current executable set to 'a.out' (x86_64).
(lldb) settings set target.source-map /home/karnajitw/lldbtest/obj/a/b/c/d/e/app/../../../../../../../src/a/b/c/d/e/app /home/karnajitw/app
(lldb) l main
File: /home/karnajitw/lldbtest/obj/a/b/c/d/e/app/../../../../../../../src/a/b/c/d/e/app/main.c
   1 #include "sub/subfile.h"
   2
   3 int main()
   4 {
   5 subfn();
   6
   7 return 0;
   8 }
   9
(lldb) l subfn
File: /home/karnajitw/lldbtest/obj/a/b/c/d/e/app/sub/../../../../../../../../src/a/b/c/d/e/app/sub/subfile.c
(lldb) settings append target.source-map /home/karnajitw/lldbtest/obj/a/b/c/d/e/app/sub/../../../../../../../../src/a/b/c/d/e/app /home/karnajitw/app
(lldb) l subfn
File: /home/karnajitw/lldbtest/obj/a/b/c/d/e/app/sub/../../../../../../../../src/a/b/c/d/e/app/sub/subfile.c
   1 #include <stdio.h>
   2 #include "subfile.h"
   3
   4 void subfn()
   5 {
   6 printf("This is subfn...\n");
   7 }
   8

It still needs both the mappings
(lldb) settings show target.source-map
target.source-map (path-map) =
[0] "/home/karnajitw/lldbtest/obj/a/b/c/d/e/app/../../../../../../../src/a/b/c/d/e/app" -> "/home/karnajitw/app"
[1] "/home/karnajitw/lldbtest/obj/a/b/c/d/e/app/sub/../../../../../../../../src/a/b/c/d/e/app" -> "/home/karnajitw/app"

That does seem wrong. Please file a PR about this.

There's a method in FileSpec (GetNormalizedPath) that canonicalizes paths by doing the obvious text substitution (unwinding ..'s and removing /./ and /// -> /, etc.) Somewhere in the comparison of the source and build paths we aren't using this when we should do. There's also a FileSpec::Equal method that allows a canonicalized comparison. It might be that there's just some obvious place we should have been using that.

I'm in the middle of something else right now, but if you have some time to dig into this, it would probably be straightforward to track down where we aren't using the right comparison. But if there's a PR, then even if you don't have time, somebody else will get to the PR when they have a chance.

Thanks for chasing this down!

Jim

Thanks Jim for the help. I would definitely like to work on this PR soon.

Filed the PR here
https://bugs.llvm.org/show_bug.cgi?id=34341

Regards,
Karan

That will be great! Last time we were more aggressive about normalizing paths we caused a decent performance regression by normalizing all the paths every time we did a file-name compare for setting breakpoints. If you look at FileSpec::Equal we added some "is it worth doing this work" preparatory work. I don't know where you'll find the additional places this needed to be done, but it's worth keeping an eye out for not putting GetNormalizedPath on a hot path.

Jim

Thanks for the info. I will keep that in mind.