It's been a while since I last wrote a proper blog post, with semi decent story telling, and some good code examples, bugs, etc, and recently I hit a pretty big milestone on this project. So I decided it's probably time to write something which is a bit meatier, and interesting.
However, before we can go into all the weird technical things which happened, bugs, and random fixes along the way, we gotta first document everything
So during these past 4~ months, I've been well trying to to get ROCm compute platform working on FreeBSD, making I would say, decent amounts of progress. As of now, if you use my custom forks, and compile everything from source, you'll be able to compile ROCm code, the device driver will start, the /dev/kfd interface will start, it can semi communicate, the runtime runs, it just doesn't work since I haven't yet worked out all the kinks (I'm working on it lol).
About 2 weeks ago, I hit the point where I needed to run my code on real supported hardware (right now RDNA3 only), now that hardware was my laptop (780M integrated graphics, ik it's messy and weird), and so I needed to build all my code for zen 4 since that's what my laptop runs (the strix halo from the foundation I'd been using was zen5 or smth, so didn't want to risk incompat instructions or smth). This meant I had to recompile everything. Sure sounds good... is what I thought
So, funny thing, I've compiled llvm multiple times on my laptop, whether it be for AdaptiveC++, or for random llvm backend optimisation theories I had (they were all wrong btw). It's always taken less than 100gb, so I thought "Oh I have 120gb free, should be good", dam I was wrong, it needs like over 150gb to compile AMD's llvm fork for some reason, I have not looked into why
So I had to recompile from source on strix halo, then move to laptop, it was then I realized, my documentation on how to build everything has been terrible, it was wrong on many occasions, had fixed paths, things had been rewritten, the small bits like certain packages, etc had not been documented. It was overall terrible, so I had to figure it out again from the start. Took about two weeks to figure out a new set of instructions, I have infact recorded the steps this time though. The following are the steps required to install the ROCm tool chain on FreeBSD with some random commentary inbetween
First, we must install the kernel, my custom fork of the kernel with A LOT of linuxkpi changes, and additions, only partially upstreamed so far, a few changes have landed, others ready to land, others unreviewed, it's a slow process to bring everything up. I will likely make everything into a stacked PR soon tho. This kernel has not been rebased since June 24th, and will not rebased for a long time Stable hash; 200d41709521405fef0cbb12656901c5d930ce0d
git clone --depth 1 https://github.com/Yohello1/freebsd-src-rocm /usr/src
pkg install cmake cd /usr/src make sysent \ make -j31 buildworld make ALLOW_PKGBASE_INSTALLWORLD=yes installworld make -j32 buildkernel make ALLOW_PKGBASE_INSTALLKERNEL=yes installkernel
Now we need to install AMD's ROCm llvm fork, which all the required changes for FreeBSD have been upstreamed. Actually there's a PR I have open for nightly testing pending merge. Figuring out the build steps for this was genuinely hard, especially for comgr. I don't know why, but we need the Z3_SOLVER to be specifically disabled otherwise it failed for some reason. You also need to include the external projects in the initial clang and lld build otherwise it fails to envrion variable issues later, for unknown reasons. There was also something, where if you tried to build device-libs, and comgr with clang and lld at the same time, it would fail cause comgr depended upon device libs. I remember onetime I dealt with this by just running make -j32 like 20 times, until it finished building device-libs and stopped complaining. I have no idea how these issues happen tbh. Also install dir does matter a lot, I put it on opt so that it wouldn't destroy anything else LMAOO
cd ~/
git clone --depth 1 https://github.com/ROCm/llvm-project
cd llvm-project
cmake -S llvm -B build \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_EXTERNAL_AMD_DEVICE_LIBS_SOURCE_DIR="$PWD/amd/device-libs" \
-DLLVM_EXTERNAL_COMGR_SOURCE_DIR="$PWD/amd/comgr" \
-DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DCLANG_ANALYZER_ENABLE_Z3_SOLVER=OFF \
-DLLVM_BUILD_LLVM_DYLIB=ON \
-DLLVM_LINK_LLVM_DYLIB=ON
cmake --build build --parallel $(sysctl -n hw.ncpu)
cmake --install build
cmake -S amd/device-libs -B amd/device-libs/build \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR="/opt/rocm/lib/cmake/llvm" \
-DCMAKE_C_COMPILER="/opt/rocm/bin/clang" \
-DCMAKE_CXX_COMPILER="/opt/rocm/bin/clang++"
cmake --build amd/device-libs/build --parallel $(sysctl -n hw.ncpu)
cmake --install amd/device-libs/build
cmake -S amd/comgr -B amd/comgr/build \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR="/opt/rocm/lib/cmake/llvm" \
-DCMAKE_C_COMPILER="/opt/rocm/bin/clang" \
-DCMAKE_CXX_COMPILER="/opt/rocm/bin/clang++" \
-DCMAKE_PREFIX_PATH="/opt/rocm" \
-DCMAKE_SHARED_LINKER_FLAGS="-Wl,--undefined-version" \
-DCMAKE_CXX_FLAGS="-D_GNU_SOURCE"
cmake --build amd/comgr/build --parallel $(sysctl -n hw.ncpu)
cmake --install amd/comgr/build
Time to install my lib numa shim, someone suggested I use the FreeBSD ports shims, but I found that they were painful to install, so I'm just gonna install my own custom ones. Side Note: NUMA is purely for performance, nothing else, mostly
cd .. # Should be in home dir now git clone https://codeberg.org/yohwllo/FreeBSD_libnuma.git cd FreeBSD_libnuma mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Debug .. make make install cd ../.. # Home dir
Time for the middle ware ish part, rocr-runtime, and rocm-core. For some reason, if I tried to compile with a version greater than 6.4.0 it had issues, I did not investigate much, and I would like to ask someone else to investigate more tbh. The rocr-runtime code is like 75% upstreamed, waiting on some people who seem to be off right now. btw the ksh and bash sym links are required, otherwise, it just doesn't work, and the patches to make it work are NOT nice, or COOL, so although it's sketch, it's better.....
git clone --depth 1 https://github.com/Yohello1/rocm-systems.git # not fully upstreamed yet
cd rocm-systems/projects/rocm-core
mkdir build
cd build
cmake -S $PWD/../ -B . \
-DCMAKE_VERBOSE_MAKEFILE=1 \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DROCM_VERSION="6.4.0" \
..
make
make install
cd ..
cd ..
cd rocr-runtime
mkdir build
cd build
pkg install libdrm
cmake -DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DCMAKE_C_COMPILER=/opt/rocm/bin/amdclang \
-DCMAKE_CXX_COMPILER=/opt/rocm/bin/amdclang++ \
..
pkg install -y ksh93
ln -s /usr/local/bin/ksh93 /usr/local/bin/ksh
ln -s /usr/local/bin/bash /bin/bash
make
make install
cd ..
cd ..
cd ..
cd ..
Time to finally install the driver.... One day I will finish split up amdgpu, and amdkfd, here is the stable hash on sup-special branch
git clone http://github.com/Yohello1/drm-kmod-rocm.git cd drm-kmod-rocm git switch super-special cd amd/amdgpu make DEBUG_FLAGS=-g -j32 make DEBUG_FLAGS=-g install cd .. cd .. make DEBUG_FLAGS=-g -j32 make DEBUG_FLAGS=-g install pkg install gpu-firmware-kmod
Now reboot, then run
kldload amdgpu
Now /dev/kfd should show up, and u should be good to go!
On a related note, to compile code, you need something like this clang -I/opt/rocm/include -L/opt/rocm/lib -lhsa-runtime64 test_gpu.c -o test_gpu , and make sure to add it to the LD (don't ask me how, I forgot this part...)
I have no idea how I got this far
Now I can't stay serious for ever, we gotta share some of the funny stories and stuff that has happened
So there's this function called register_class in linux which takes lets you register a device class to expose a driver sub system via sysfs (under /sys/class). Now long long ago, before time had even been given a name, this function took in a mutable "struct class", which it would then fill in, and do whatever. This was in like kernel 6.5 or something, but at some point between then and kernel version 6.12, there was a massive refactor(?) which changed this class to read only. Such that it gets set/changed at compile time (I think?)
Because of this, the linuxkpi had a function called register_class which took in a mutable struct, and well, mutated it. Which is perfectly understandable, since it was made back when 6.9 or smth was current. However, I am pulling amdkfd from 6.12which assumed that register_class took in a const class, and treated it as such. i.e the struct was const in the code, but the function it was passed to assumed it was mutable
When I first ported the code, I thought all was well and fine, and didn't notice it. But I would find that it would panic whenever starting, so I spent like 5-6 hours trying to figure out why. I somehow went pretty deep into register_chrdev, and register_chrdev_p to find that it crashed at register_class. From that, I looked at the thing being passed in, function signature, and the type, and realized, OH SHIT, IT'S WRITING TO READ ONLY DATA
Now after discovering that, there are a few potential ways to fix this, one such way is to add in a new impl of this function for new linuxkpi versions. Another is to make the struct mutable..... I just made the struct mutable, it does the same thing anywayssssssssssss (I think)
Now if you thought we were done with register_chrdev, you thought wrong, very wrong.
register_chrdev is a very interesting function, first off because it's written chrdev and not chardev, purely to save one character in the function call. Beyond that, it's actually just a wrapper for __register_chrdev(..., 256); WHY IS IT A WRAPPER JUST LET ME CALL IT DIRECTLY
Anyways, that 256 at the end is actually to make 256 device files for some reason. But for some reason on linux it only makes /dev/kfd??? I actually don't understand this character device stuff. Understanding the ioctl, and kernel panics is easier than understanding this
Ok but now we're surely fine? NO THERE'S ANOTHER FUNCTION CALLED register_chrdev_p, which is what's actually used in freebsd's port of drivers for some reason. So I replace it with that, but same issue, creates /dev/kfd/0...256. I dive in, and it's the EXACT SAME THING, INCLUDING THE WRAPPER, except for one thing which is like cdev_add vs cdev_add_ext or something. I remember trying to get this function trying to work was too painful, so I transplanted its organs straight into its call location
Semi related, but right now the driver only works for sure on RDNA 3, since my gpu is RDNA 3, and I'm not fully sure whether RDNA 3.5 strix halo can run, since a code path which claims "Unsupported as of now" (pr_debug saying something similar to this in the code) as per the debug logs, runs, but it fully finishes, and seems to init correctly? Anyways, there are a few switch statements with a lot of cases commented out referencing to a bunch of functions for other generations of gpus and their initializations. To add support for them, you need to drag in their files, add them to make files, and uncomment their calls. Email me if you need more details, it's a bit wishy washy.
amdgpu, and amdkfd kernel modules should be built separately, and then linked together. But because I was being lazy, I merged everything into the amdgpu make file/linking stage. I do need to split them up before merging back with drm-kmod (to be honest this is a few months away), I will probably throw opencode+qwen3.8 27B at it, and tell it to continue until it works. I don't think spending my time on this is worth it.
The FreeBSD Foundation was the organization which hired me, paid me to work on this, and well gave me a lot of the hardware I used to develop this. As well as being able to ask some dumb questions time to time. I've pretty extensively abused their servers and computers to run many many builds, and vm's at times. I've also extensively used the uwaterloo csclub servers to test code lol.
So originally when I finished porting rocr-runtime back in like mid june or something I ran a massive git diff command against the clean upstream repo, and threw it into a file called qq.txt, and uploaded it to https://racha.ca/patches/qq.txt But I recently realized that there was a typo in there, and it was missing some ioctl stuff. I.e I misspelled advise as advice, and caused some substitution macro errors. Had to fix that, but was otherwise fine, was confused as to why nothing was building
One of my many patches to linuxkpi, has been adding a member called tgid to a struct in linuxkpi. But what I did not consider is that this breaks the KBI (Kernel Binary Interface) as Bjoern pointed out in the review . So I've patched that, and am pending review again lol