Win32: Crash in DLL created by llvm that calls into the "putchar" function

Hello list,

I got a simple test case to reproduce a crash in code that has been
generated by llvm. The code calls into the "putchar" function from
LIBCMT.LIB. It works when creating an .exe file, but not when creating
a .dll file and calling into an exported function.

First, let me show how the code works as an .exe:

cat callPutchar.ll

; ModuleID = 'test'

declare i32 @putchar(i32)

define i32 @main() {
__entry__:
  %0 = call i32 @putchar(i32 79)
  ret i32 %0
}

lli callPutchar.ll ; Test, whether the code itself works. Output is as expected an 'O' (ASCII 79):

O

llc -filetype obj callPutchar.ll ; Create an .obj file and later link it with MSVC's link.exe:

link.exe /SUBSYSTEM:CONSOLE callPutchar.obj LIBCMT.LIB

Microsoft (R) Incremental Linker Version 11.00.60610.1
Copyright (C) Microsoft Corporation. All rights reserved.

callPutchar.exe ; Call the newly created .exe file. Output is as expected an 'O':

O

Now, when I do something very similar, but with a DLL instead of an
EXE file and call into the exported function, I get an access
violation in the generated function:

cat callPutchar.ll

; ModuleID = 'test'

declare i32 @putchar(i32)

define dllexport i32 @main() {
__entry__:
  %0 = call i32 @putchar(i32 79)
  ret i32 %0
}

llc -filetype obj callPutchar.ll

link .exe/DLL /NOENTRY /SUBSYSTEM:CONSOLE callPutchar.obj LIBCMT.LIB

Microsoft (R) Incremental Linker Version 11.00.60610.1
Copyright (C) Microsoft Corporation. All rights reserved.

   Bibliothek "callPutchar.lib" und Objekt "callPutchar.exp" werden erstellt.

demoDllCaller.exe ; >> The typical "demoDllCaller.exe stopped working..." Windows crash dialog appears.

demoDllCaller is a small application that I wrote that loads the dll
and calls the main function. Here's the code of demoDllCaller.exe:

#include <Windows.h>

int main ()
{
    HMODULE lib = LoadLibrary (L"callPutchar.dll");
    if (lib != nullptr) {
        typedef int __declspec(cdecl) (*main1_t) ();
        main1_t main1 = (main1_t) GetProcAddress (lib, "main");
        if (main1 != nullptr) {
            int result = (* main1) ();
            result = result;
        }
    }
}

I'm on Windows 8.1 using Visual Studio 2012 and 2013 (crash happens
with both). I compiled llvm with Visual Studio 2013 (exact version is
"12.0.21005.1 REL").
There is no usable stacktrace that I could show you.

What seems a little odd to me, too, is that the callPutchar.dll is
62MB in size. callPutchar.exe is 70MB. A bit much I guess :wink:

You can get all files (callPutchar.ll, .obj, .dll, .exe, .exp and .lib) here:
https://dl.dropboxusercontent.com/u/1913181/Temp/callPutchar.7z

Greetings,
Daniel Albuschat

Hey Jeremy,

(putting the discussion back to the list.)

If I run your demo exe it crashes, if I re-link the dll with VS 2010 it
errors gracefully;

runtime error R6030
- CRT not initialized

putchar is a c runtime function.
I'm betting that your exe doesn't initialise the runtime library?

You're right, this is surely the problem then. Thanks a buch for that pointer.
However, after quite much googling, I can't find a way to correctly
initialize the CRT in a DLL created from llvm.

From what I've learned, the system works as follows:

* The entry point for DLLs in __DllMainCRTStartup@12.
* The CRT libraries provide an implementation for this function in libcmt.lib.
* That implementation initializes the CRT (afaik using CRT_INIT) and
calls DllMain.

The question that remains is how to dllexport a function that is
defined in a separate library?

Greetings,
Daniel Albuschat

Hey Jeremy,

(putting the discussion back to the list.)

If I run your demo exe it crashes, if I re-link the dll with VS 2010 it
errors gracefully;

runtime error R6030
- CRT not initialized

putchar is a c runtime function.
I'm betting that your exe doesn't initialise the runtime library?

You're right, this is surely the problem then. Thanks a buch for that pointer.
However, after quite much googling, I can't find a way to correctly
initialize the CRT in a DLL created from llvm.
From what I've learned, the system works as follows:

* The entry point for DLLs in __DllMainCRTStartup@12.
* The CRT libraries provide an implementation for this function in libcmt.lib.
* That implementation initializes the CRT (afaik using CRT_INIT) and
calls DllMain.

The question that remains is how to dllexport a function that is
defined in a separate library?

Actually, I answered that question just a few seconds later. :wink:
I just forgot to pass LIBCMT.LIB to link.exe in my previous tries.

Here's the working .ll file that can be used to create a .dll that
calls into the putchar function:

cat callPutchar.ll

; ModuleID = 'test'

declare i32 @putchar(i32)
declare x86_stdcallcc i8 @DllMainCRTStartup(i32*, i32, i32*)

define dllexport i32 @main() {
__entry__:
  %0 = call i32 @putchar(i32 79)
  ret i32 %0
}

define dllexport x86_stdcallcc i8 @DllMain(i32*, i32, i32*) {
    ret i8 1
}

llc -filetype obj callPutchar.ll

link /DLL /SUBSYSTEM:CONSOLE callPutchar.obj LIBCMT.LIB

Microsoft (R) Incremental Linker Version 12.00.21005.1
Copyright (C) Microsoft Corporation. All rights reserved.

   Bibliothek "callPutchar.lib" und Objekt "callPutchar.exp" werden erstellt.

d:\callPutchar>demoDllCaller.exe
O

YAY!

Thanks again, Jeremy.

Daniel