RFC: Adding an option to llvm-link to allow it to get a list of input bitcode file names from a file

Project Infrastructure LLVM Dev List Archives
#1

I am proposing to add an option to llvm-link allow it to get a list of input bitcode file names from a file.

The reason is that there is a limitation for command line length which limits the number of input bitcode files that can be passed to llvm-link. By adding this option we can bypass such limitation.

The name of the option can be discussed. My initial proposal would be –input-file-list.

Each line of the list file is a path to an input bitcode file.

If this option is given, no input bitcode file names are allowed in the command line. If input bitcode file names are given in the command line, this option is not allowed.

The implementation is simple. Only llvm-link.cpp is affected.

Thanks.

Sam Liu

#2

Why is there a restriction on the usage between command line and input file? It seems that they both should be able to co-exist and the resulting linking is done as a set of all the bitcode files.

#3

My assumption is that a user will either specify input file names in the command line or passing them by a list file, but usually would not do both. Making these two different ways of specifying input files mutually exclusive may avoid confusion and misuse.

Sam

#4

What about the case where you have a list of bitcode files and you want to supplement that list with a few more files?

Take for example you have a common subset of bitcode files that you are attempting to link together along with a small number of device specific bitcode files, and you want to do this efficiently for multiple devices.

The common subset could be added to the file and then the device specific bitcodes can be specified on the command line. Otherwise you have to duplicate the bitcode file for each device.

Micah

#5

Liu, Yaxun (Sam) wrote:

I am proposing to add an option to llvm-link allow it to get a list of
input bitcode file names from a file.

The reason is that there is a limitation for command line length which
limits the number of input bitcode files that can be passed to
llvm-link. By adding this option we can bypass such limitation.

The name of the option can be discussed. My initial proposal would be
–input-file-list.

It's common for tools to accept "@file" as an argument that means "read options from file", where options may include both flags and positional arguments (ie., more input files). For instance, gcc, as and ld all support @file.

I realize this is more complicated than just reading in a list of input bitcode files, but I think the right fix is to teach llvm to support this. (It looks like clang already does? What are they doing?)

Nick

#6

It seems clang supports @file.

http://llvm.org/svn/llvm-project/cfe/trunk/tools/driver/driver.cpp

So maybe it is a better idea to let llvm-link support @file instead of adding a new option.

Sam

#7

Just found llvm-link (and other llvm commands) have already supported @file through cl::ParseCommandLineOptions, therefore no change is needed. Thanks.

Sam

#8

Well, although cl::ParseCommandLineOptions contains support for @file. By default it is disabled, which is the case for llvm-link.

To enable @file support in llvm-link, a small change is needed:

--- llvm-link.cpp.orig 2012-09-20 16:10:50.000000000 -0400
+++ llvm-link.cpp 2012-09-20 16:11:24.000000000 -0400
@@ -83,7 +83,7 @@

   LLVMContext &Context = getGlobalContext();
   llvm_shutdown_obj Y; // Call llvm_shutdown() on exit.
- cl::ParseCommandLineOptions(argc, argv, "llvm linker\n");
+ cl::ParseCommandLineOptions(argc, argv, "llvm linker\n", true);

   unsigned BaseArg = 0;
   std::string ErrorMessage;

Sam

#9

Could we enable ReadResponseFiles=true by default?

#10

I have no objection to that. Actually I am curious why it is disabled by default.

If we enable ReadResponseFiles=true by default, we will automatically get support of response file (@file) in all llvm tools. The only issue I can think of is that some users may have file names starting with "@".

Sam

#11

I have no objection to that. Actually I am curious why it is disabled by default.

If we enable ReadResponseFiles=true by default, we will automatically get support of response file (@file) in all llvm tools. The only issue I can think of is that some users may have file names starting with "@".

I don't think that is a likely problem. I agree with Nakamura, lets
change the default. Do you have write access or do you need me to
commit a patch?

Sam

Cheers,
Rafael

#12

I do not have write access. Please commit the patch for me. Thanks.

Sam

ResponseFile.patch (4.49 KB)

#13

I do not have write access. Please commit the patch for me. Thanks.

I have simplified the test and committed it as r165535.

Sam

Cheers,
Rafael