This appears to be a SWIG bug: https://github.com/swig/swig/issues/769
I'll do another snapshot maybe next week or the week after. You can
also ping me if you want it sooner or later.
We're kicking off the release process for 4.0.0 on Thursday. I don't
fully understand the problem here, but if there's some way to work
around it and get lldb into good shape for the 4.0.0 installer, that
would be great.
Thanks,
Hans
It sounds like the solution to the problem is to downgrade SWIG on the build machine. If it’s using version 3.0.9 or higher, just use whatever the last version before that is. 3.0.8, for example.
At least there’s a good workaround in the interim (i.e. setting PYTHONPATH)
I've downgraded my swig to 3.0.8 and built a new snapshot (r291454).
Please let me know if that works.
Yes, the new build works!
Great! Thanks for pushing this.
Sorry, just found another problem: the installed lldb crashes when given a script via the command line. For example, lldb -O "p 42" dies with exception 0xC0000409.
It didn’t happen with the one I’ve build locally, so I started digging… The difference seems to be that build_llvm_build_package.bat links CRT statically (-DLLVM_USE_CRT_RELEASE=MT), whereas default is the dynamic CRT.
The crash kinda makes sense, because python35.dll links CRT dynamically, so LLDB ends up with two copies of it in the process, which is known to cause all sorts of trouble.
Not sure what to do about this one, - you probably wanted static CRT to avoid having to install MSVC redistributable?
Is static CRT even still supported / recommended when using the new Universal CRT? My understanding is the new UCRT is considered a core windows component, so we don’t ahve to distribute redistributables anymore. Maybe I’m wrong about this.
That said, I think we want dynamic regardless, otherwise we’re back in the same boat of user having to compile Python, which is the exact thing 3.5+ is supposed to solve. We should just always use dynamic so the user doesn’t have to do anything and it just works.
The purpose of linking the CRT statically was to ensure that clang can run on systems that don’t have the CRT already installed from some other app or by VS itself. As long as that is preserved, feel free to do whatever you need.
I think we still want to link vcruntime140 statically and the C++ runtime statically, for example.
I was just reading up on this the other day. Statically linking to the Universal CRT (and related libraries) is supported though it’s not Microsoft’s top recommendation.
Initially, they said they weren’t going to support app-local deployment, but they backed off on that, and it, too, is now supported.
Some details here: https://blogs.msdn.microsoft.com/vcblog/2016/04/18/c-runtime-deployment-why-choosing-applocal/
If we do that though, we still get 2 copies of the CRT. One for python and one for LLDB. While I think there is a strong enough API boundary in how the application and Python access each others’ data structures that it doesn’t matter, I’m not 100% sure without further research. It seems safer to link dynamically unless there is a strong reason not to.
I agree that linking to them dynamically is safer, but then you get into the deployment problem. You do have to redistribute the DLLs for users using anything less than Windows 10.
There are several options for doing that: VCRedist, MSMs, MSUs, and app-local deployment. They each have drawbacks.
BTW, here’s the call stack at the point of failure. I don’t see anything Python-related in it, so maybe it’s some other CRT interaction.
liblldb!_invoke_watson(wchar_t * expression = 0x00000000 “”, wchar_t * function_name = 0x00000000 “”, wchar_t * file_name = 0x00000000 “”, unsigned int line_number = 0, unsigned int reserved = 0)+0xe
liblldb!_invalid_parameter(wchar_t * expression = , wchar_t * function_name = , wchar_t * file_name = , unsigned int line_number = , unsigned int reserved = )+0x7a
liblldb!_invalid_parameter_noinfo(void)+0xc
liblldb!_read(int fh = 0n0, void * buffer = 0x05afbb50, unsigned int buffer_size = 0x1000)+0x10a
liblldb!common_refill_and_read_nolock(class __crt_stdio_stream stream = class __crt_stdio_stream)+0x9f
liblldb!_fgetc_nolock(struct _iobuf * public_stream = 0x05af9c58)+0x2d
liblldb!__crt_char_traits::getc_nolock+0x8
liblldb!common_fgets(char * string = 0x019cf41c “b”, int count = 0n256, class __crt_stdio_stream stream = class __crt_stdio_stream)+0x81
liblldb!lldb_private::IOHandlerEditline::GetLine(class std::basic_string<char,std::char_traits,std::allocator > * line = 0x019cf548 “”, bool * interrupted = 0x019cf537)+0xd6
liblldb!lldb_private::IOHandlerEditline::Run(void)+0x11a
liblldb!lldb_private::Debugger::ExecuteIOHandlers(void)+0x35
liblldb!lldb_private::CommandInterpreter::RunCommandInterpreter(bool auto_handle_events = true, bool spawn_thread = false, class lldb_private::CommandInterpreterRunOptions * options = 0x05af9ca8)+0x7e
liblldb!lldb::SBDebugger::RunCommandInterpreter(bool auto_handle_events = true, bool spawn_thread = false, class lldb::SBCommandInterpreterRunOptions * options = 0x019cf604, int * num_errors = 0x019cf630, bool * quit_requested = 0x019cf603, bool * stopped_for_crash = 0x019cf60a)+0x43
lldb!Driver::MainLoop(void)+0x639
lldb!wmain(int argc = 0n3, wchar_t ** wargv = 0x01b28c90)+0x249
lldb!invoke_main+0x1d
lldb!__scrt_common_main_seh(void)+0xff
KERNEL32!BaseThreadInitThunk+0x24
ntdll_76f30000!__RtlUserThreadStart+0x2f
ntdll_76f30000!_RtlUserThreadStart+0x1b
Definitely seems CRT / Python related. The call to fgets indicates that it’s doing something involving a FILE*. It’s probably a FILE* that refers to a file created from within Python’s copy of the CRT. Another problem that dynamic linking would solve. Can you confirm that the problem does not occur when LLDB links dynamically?
Maybe in ScriptInterpreterPython somewhere? There’s a couple of places where we use FILE* in that file, but it’s not easy to tell by inspection whether the FILE*'s originate from something in Python.
More likely perhaps is to try putting a breakpoint in PythonDataObjects.cpp line 983 (called when try to create a wrapper around a Python File object so we can reference it from C++) and Line 1010 (called when we try to create a Python file object out of a native file descriptor). These are the two major API boundaries where we convert between native file descriptors and python file objects.
Okay, as far as I can see, the issue is not with Python, but that lldb.exe and liblldb.dll have their own private copies of the CRT.
When startup script is given on the command line, lldb.exe creates a pipe, writes the script into the write end, wraps a stdio file around the read end, and gives the resulting FILE* to SBDebugger::SetInputFileHandle(). Unfortunately, since SBDebugger lives in liblldb.dll, with its own copy of the CRT, that handle is not valid there. Later, it tries to read from that handle, and… boom.
Ahh, that would make sense as well, since LLDB links against liblldb as a dll. Don’t see a good solution to this short of forcing dynamic linking. liblldb has to be a dll because it needs to be visible to python as an extension module. And if it’s a dll and uses static linking, then we would have to require that lldb.exe not pass file handles across the dll boundary. If that’s easy then great, but I suspect it probably isn’t.
Honestly dynamic linking was created to solve exactly these kinds of problems. Yes there’s the redist issue if on Windows < 10, but it seems like a small price to pay for something that doesn’t have weird subtle bugs. And as time goes on, more and more people will be on windows 10 anyway.
Does the redistributable issue present a challenge for your use case?
There are actually two part of the MSVC runtime: the Universal C Runtime
(UCRT) and the compiler-specific, VCRUNTIME140. The former is widely
available (comes with Windows 10, and had been pushed to Vista, 7 and 8 via
Windows Update). The latter is tied to the compiler version and must still
be redistributed with programs. Ideally, LLDB would use dynamic UCRT +
static VCRUNTIMExx. Unfortunately this doesn't seem to be a supported
configuration (discussion of this issue in Python bug tracker
<Issue 24872: Add /NODEFAULTLIB:MSVCRT to _msvccompiler - Python tracker). Looks like Python folks opted for
shipping VCRUNTIME140.DLL in their install directory.
The compiler redistributable is also widely available:
https://www.microsoft.com/en-us/download/details.aspx?id=48145
The python 3.5 dll would still use the runtime dll; lldb should do the same. I’ve had crashes in the past because lldb and python used different runtimes.
[UCRT] had been pushed to Vista, 7 and 8 via Windows Update
I didn’t think that was true in general. One of the redist options is to include an MSU in your installer that tells Windows Update to add UCRT to the list of components it updates. So some Vista, 7, and 8 users may have received it via Windows Update, but probably not all.
That’s what all the deployment articles seem to suggest (assuming I’ve understood them correctly). I suppose they may have decided to ship UCRT to pre-Windows 10 users anyway, but I haven’t come across anything that says that.