[CMake] External File Dependencies for Unit Tests

Hi,

I want to write unit tests for a project I am working on. I need to create object files, preferably I want to have yaml files in the source tree and have cmake generate object files using yaml2obj. Does anyone know what I would put in the CMakeLists.txt to get this behavior?

For a concrete example if I didn’t explain clearly:

In llvm/unittests/Object I would maybe have an Inputs directory, with a basic-obj.yaml. In llvm/unittests/Object I would want a test case which can open Inputs/basic-obj.o.

Thanks,
Alex

Hi Alex,

Not answering your question directly, but I’d highly recommend against using CMake to bake the test outputs at the cmake step for several different reasons:

  1. If yaml2obj changes, you’d have to rerun cmake to see whether the new version would cause the tests to fail. This is not the default, so could easily lead to build-bot breakages.
  2. It’s not clear from your statement whether you’re talking about lit tests or gtest unit tests. If the former, there’s no real precedence for doing this. If the latter, you’d need to have your C++ code able to detect the build output directory somehow, which feels like it would make the code at least unclear, and at worst unusable.
  3. In either case, if you wanted to change the test input, you’d need to rerun CMake again, rather than it just working. Runnning CMake is not cheap, especially on Windows for Visual Studio generators.

As Jordan has mentioned offline, and assuming you’re talking about gtest, you’re probably better off just calling the corresponding library functions to generate the object from YAML. Otherwise, just use yaml2obj directly.

If you want to explore your original suggestion further, and need help doing so, you probably need to give a bit more context as to what you’ve tried and what didn’t work about it.

Regards,

James

Thanks James!

Thanks for the help. I certainly see your point.

In either case, if you wanted to change the test input, you’d need to rerun CMake again
This is unfortunate, I would have hoped I could have CMake create $(wildcard *.yaml) type of output in the make files. But it sounds like this doesn’t exist on make alternatives, so this wont work.

There is actually a function to get the Inputs directory, http://llvm.org/doxygen/namespacellvm_1_1unittest.html. I think it is only used in one test, though. Would you recommend that I do what is done in unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string inside the source file, or use an Inputs directory? Both are not great, as you point out about using an Inputs directory.

Also, I’m not sure that I can create an object without invoking yaml2obj itself, I can’t seem to find a function to do this exposed by libObjectYAML. If I do need to invoke yaml2obj, would you happen to know how to tell CMake to build yaml2obj first, and also where would I find that executable? Thanks so much, James.

Hi Alex,

Would you recommend that I do what is done in unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string inside the source file, or use an Inputs directory? Both are not great, as you point out about using an Inputs directory

I’d be inclined to go with the string in the source file approach. It keeps the whole test localised to one place, making it easier to understand what the test is actually testing.

Also, I’m not sure that I can create an object without invoking yaml2obj itself, I can’t seem to find a function to do this exposed by libObjectYAML.

It looks to me like you’d need to move big chunks of code out of the yaml2obj tool into the ObjectYAML library. The Minidump version was added quite recently, and took the approach of doing all the layout in the library, whereas for most other formats, the work is done in the executable layer. I think it would be perfectly reasonable to do this code moving, and would then allow you to use the library in your test setup, similar to how minidump works.

If I do need to invoke yaml2obj, would you happen to know how to tell CMake to build yaml2obj first, and also where would I find that executable?

I’m not a CMake expert by any means, but I think you would need to create a dependency on yaml2obj from the test target, then add a step in the test’s CMakeLists.txt to run yaml2obj and generate your inputs. I’m still not convinced that it’s the right way to go though.

James

Hi Alex,

    Would you recommend that I do what is done in
    unittest/ObjectYAML/MinidumpYAMLTest.cpp which just uses a string
    inside the source file, or use an Inputs directory? Both are not
    great, as you point out about using an Inputs directory

I'd be inclined to go with the string in the source file approach. It keeps the whole test localised to one place, making it easier to understand what the test is actually testing.

    Also, I'm not sure that I can create an object without invoking
    yaml2obj itself, I can't seem to find a function to do this exposed
    by libObjectYAML.

It looks to me like you'd need to move big chunks of code out of the yaml2obj tool into the ObjectYAML library. The Minidump version was added quite recently, and took the approach of doing all the layout in the library, whereas for most other formats, the work is done in the executable layer. I think it would be perfectly reasonable to do this code moving, and would then allow you to use the library in your test setup, similar to how minidump works.

[Disclaimer: I'm the person who added the minidump code.]

I would very much like to see this happen. Lldb has some unit tests which need to binary files to operate. The way that's handled right now is by shelling out to yaml2obj from inside the unit test (so, not from cmake). However, it would be infinitely better if we could just invoke a library function to do this conversion for us.

    If I do need to invoke yaml2obj, would you happen to know how to
    tell CMake to build yaml2obj first, and also where would I find that
    executable?

I'm not a CMake expert by any means, but I think you would need to create a dependency on yaml2obj from the test target, then add a step in the test's CMakeLists.txt to run yaml2obj and generate your inputs. I'm still not convinced that it's the right way to go though.

Running yaml2obj from the unit test like lldb does is probably a bit better than doing it from cmake. You can search for ReadYAMLObjectFile to see how it's done there. However, I wouldn't really recommend emulating that... :slight_smile:

cheers,
pl

I actually added support to our unit tests for external file
dependencies a year or so ago. Search for
`add_llvm_unittest_with_input_files` in unittests/*.txt and that will
give you an example usage.

That said, why doesn't a lit test suffice for the OP's case of
creating an object file and doing something with it?