Race in TestLldbGdbServer and TestGdbRemote*

Hi Everybody,

Occasionally I see failures in TestLldbGdbServer and in TestGdbRemote* most likely caused by a race condition in the test runner hitting an edge case in these tests. As far as I understand the problem is that each test case compile the same source file (main.cpp) to main.o and then to a.out and when the test is finished it removes these files. The problem starts when multiple test case running at the same time trying to do it because it is possible that one of the test case just compiled it’s executable and then an other test case (running in a different process) immediately removes it causing a failure in process launch. This issue causing significant noise on our build bots so we should fix it as soon as possible.

One possible solution (what is used in several similar cases) is to use a different executable name for each test case what reduce the chance of the described scenario but it isn’t eliminates it completely because the name of the intermediate build file (main.o) can’t be changed easily from the test case.

A second possible solution is to merge all test case into a single file what will prevent the test runner (dosep.py) to execute them in parallel but this can have some negative effect on the speed of the tests.

A third option to move each test case into it’s own directory to eliminate the possibility for a source file conflict but I am not sure if it is possible without duplicating the source file of the inferior (quite complicated) into each folder.

I am looking for some advice about what approach you suggest to get rid of this race in the tests.



I have also investigated these failures for a while, and I am afraid
the solution is not that easy. dosep.py does not do paralelization
based on test files, but on folders. So, all test cases in a single
folder are running sequentially and should not interfere with one
another. Nevertheless, the file does get removed prematurely, so it
seems that someone else (a test in another folder?) is deleting it...
Unfortunately, I was not able to find the culprit....