GDB my beloved (2026/10/05)

My internship more or less came to a close at the end of August. However, this does not mean that the work on ROCm stopped, rather, it still slowly marches forward towards the check point. Work definitely has slowed down, because I'm now looking for my next job + school term!

Which Library do I load?

On the surface this seems like a very simple question, simply load the library specific to your OS+Arch right? For most programs this is true, but not for rocr-runtime, no, no, no.

I'm going to give you a brief walk through of how hsa_init works + a small demo program. I will only explain up to what is working on FreeBSD, and what GDB has shown me whilst debugging. To start off, here is the program + script for adding things to PATH/LD stuff.

LD+PATH script

export PATH="/opt/rocm/bin:$PATH"
export LD_LIBRARY_PATH="/opt/rocm/lib:$LD_LIBRARY_PATH"

C Program

#include 
#include 

int main() {
            hsa_status_t status = hsa_init();
                if (status == HSA_STATUS_SUCCESS) {
                                printf("HSA initialized successfully.\n");
                                        hsa_shut_down();
                                                return 0;
                                                    }
                    printf("HSA initialization failed: 0x%x\n", status);
                        return 1;
}

Now that we're on the same page, let's start running it! First we will run it, then step into hsa_init. This leads us to a function which first initialises the API table LoadInitalHsaAPITable(). That function is basically just a wrapper to call another function called hsa_table_interface_init with some params attached. Which then just calls another one called hsa_api_table... which to be honest I can't even tell what it's doing.

Then it goes to another function called hsa_table_interface_init, which just sets variables, and binds them

Now we're moving onto the juicy part coreApiTable->hsa_init_fn() which is where we actually connect the api table stuff. This then calls a function called acquire, which creates the runtime singleton. It then acquires a lock on the singleton, makes sure it's the only one, if so make the runtime. Then it checks a bunch of other stuff. Eventually it hits the runtime_singleton_->Load() function, where we start loading stuff FINALLY.

Under this function, we first check the CpuID info, then we... CREATE THE ThunkLoader object which does actual stuff!!! Immediately after creating the object we call the function LoadThunkApiTable, which is a hell of a function to be honest... Reading, and understanding this function will teach you a lot about programming in general

The start of the function has a IsSharedLibraryLoaded check, which just returns the value of a boolean. But this boolean is set during the initialisation phase. If you remember my short discussion on the _is_dxg code from a few months ago, the is_loaded_ variable is kind a descendant from that block of code

The ThunkLoader constructor checks to see if there is a library loaded, then tries to seemingly load the library, if it fails to... I ran out of mental energy to fully comprehend how this ends up at the correct values. But the ThunkLoader::whoami function is incredibly important. As it tells you whether you're on windows or not, via some partial magic (I would recommend reading this function too, as it is weird, and non-trivial). Originally I thought I needed to return the libhsakmt.so library, but in retrospect (and after debugging) I was in fact supposed to return nothing LOL

Anyways, back to the if statement. If you are on windows, which is determined by whether the shared library is loaded or not, it. It lowkey took me like 3 hours of staring at all the involved functions to make all the connections, and build a graph. Partially because my brain is not wired to comprehend pre-processor macros fast, or efficiently

Then you just got regular weird library loading, I lowkey don't fully understand what it's doing, but I also don't want to invest the energy to fully understand it. One vivid memory I have of trying to debug this is literally stepping through the loading of each and every single function, and its linking, then checking the registers, took a solid 4-5 hours...

One thing that I did find weird is that it would seemingly only check for existence if it was on Windows, I think. I haven't delved too deep into the macros, and stuff to really know

So when it comes to the part of which library do I load, the ultimate answer is... None... then it loads in the libhsakmt library, then it also loads in the driver so it can bind to a bunch of amdgpu_* functions. I'm kinda confused as to why

Discovering, and Loading the WRONG driver!

You know, sometimes people make mistakes, even I make mistakes... One big one I made was uh, adding the XDNA driver as an enabled one in the list of drivers to discover and checkout. So there's this array called discover_driver_funcs the syntax is a work of art in all the wrong ways (lives under projects/rocr-runtime/runtime/hsa-runtime/core/runtime/amd_topology.cpp). The array lists all the drivers which need to be loaded, so that at runtime, another function (right below) called DiscoverDrivers, can well discover, hook into, and init the Driver interface.

It took me... maybe a day... or a few to figure out that this is why I was getting a seg fault. But after that was fixed, it was up and running pretty fast!

But now, Im doing something weird, it tries to load, discovers the drivers, then tries to do something with this weird thing called AgentDrivers.... but then it crashes. I haven't spent enough time investigating it

I forgot to load the driver

I spent like 30 minutes debugging an issue, tracing it through meta-programmed hell/heaven C++, just to realise I forgot to load the driver... Below are a bunch of fun screen shots I took + some random commentary I wrote out whilst debugging it

At the end of this post, there is a meme + screen shots + commentary on debugging this

How amdkfd, and AMD's gpu runtime work together, and potential reasons as to why

(Note: I met someone at BSDCan who wanted to know a bit more about amdkfd, and rocr-runtime's interactions, I kinda ghosted them for a while till I met them at X.org Dev Conference, here is the part of the email which you guys might find interesting)

I think part of the perceived complexity is because AMD's amdgpu and amdkfd driver have a lot of overlap in how they function, and the things they depend on each other for.

The amdgpu module manages the display, virtualization, power, SDMA, etc operations, these are all managed within amdgpu itself, and generally other parts of the driver modules (i.e amdxcp, amdkfd) generally do not directly touch them. Kind of... amdkfd at least exposes methods to interact with the sub-systems within amdgpu. I'm not sure about amdxcp.

In addition both the kernel driver, and user space runtime require doorbells to tell the GPU's hardware schedulers about new work. The kernel sets up the doorbell mappings, while the GPU hardware scheduler itself handles the multiplexing and scheduling between the user space code and kernel space code. Managing these doorbells, process contexts, and aperture mappings is likely where the perceived complexity comes from. Managing all of this did not look trivial when I was skimming the code.

Because of the kernel space, and user space both having their own doorbells, they thus have their own independent queues. This means that for essential subsystems like SDMA (more exist!), there exists both a kernel, and user space queue. Managing the mappings for these two queues also adds to the complexity.

The way in which amdkfd exposes a lot of these interfaces seems to be, with minimal abstraction, and trying to maximise the power which user space has over everything..? I believe the intention was to give the user space as much raw control as they could so that they could tune, and push it from the user-space side too.

This is at least what I've seen when I've been working on amdkfd, as it seems that they both share queues, and there is a lot of coordination between the two spaces.

I will note that this is my rough understanding of what is happening based on my experience with amdkfd, and is by no means perfect, there are probably mistakes, and I have definitely missed some things. I should also note that this is based on the code I've read from Linux kernel 6.12.

I will probably ask Alex Deucher a bit more about this a bit later next week to confirm. I can cc you on my email to them if you'd like.

On a side note, if you're an AMD person who can help answer this question, please reach out! I'd love to properly understand the design decisions