warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)

ikitayama · December 5, 2021, 10:03am

Hi,
My offloading app runs correctly on host, but gets stuck (no progress)
on the device.

In a cuda gdb session I see:

(cuda-gdb) r

The program being debugged has been started already.

Start it from the beginning? (y or n) y

Starting program:
/p/project/cjinb33/kitayama1/opt/nest-clang-offload/bin/nest
hpc_benchmark.sli

warning: Cannot parse .gnu_debugdata section; LZMA support was
disabled at compile time

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/usr/lib64/libthread_db.so.1".

warning: File "/p/software/jurecadc/stages/2020/software/GCCcore/10.3.0/lib64/libstdc++.so.6.0.28-gdb.py"
auto-loading has been declined by your `auto-load safe-path' set to
"$debugdir:$datadir/auto-load".

warning: Cannot parse .gnu_debugdata section; LZMA support was
disabled at compile time

OMP: Warning #249: OMP_TARGET_OFFLOAD: syntax error, using DEFAULT.

warning: Cannot parse .gnu_debugdata section; LZMA support was
disabled at compile time

[Detaching after fork from child process 27044]

[New Thread 0x15554a92d700 (LWP 27050)]

[New Thread 0x15554a12c700 (LWP 27051)]

virtual void nest::SimulationManager::initialize() Map this pointer at
0x15552e1021a8

warning: Cuda API error detected: cuModuleGetGlobal_v2 returned (0x1f4)

Should I be concerned about the API error?

Shilei_Tian · December 11, 2021, 11:38pm

If the function call is to get the device environment variable from the image, that could return CUDA_ERROR_NOT_FOUND (500). However, it doesn’t affect the execution of the program. The hang could be something else.

ikitayama · December 11, 2021, 11:52pm

Shilei,

I tried to debug the hang with CUDA-gdb, though not sure if I was using it correctly. If there are other general approaches, would you let us know?

Shilei_Tian · December 12, 2021, 5:19pm

Probably in addition to printing things out, cuda-gdb would be the best tool here.

Regards,
Shilei