Fun project: ROCm and local LLMs

Back in May, I got the idea that while I was still employed and had money, I should buy upgrades for my computer. I did so. Last week, I implemented the plan, and it worked.

Background

Back in 2017 (nine years ago!), I bought an ASRock motherboard. At the time, AM4 CPUs were all you could get. I did not buy the latest and greatest CPU, because the latest and greatest is always overpriced. Although Ryzen 7 was available, an AMD Ryzen 7 1800X went for $500. Instead, I bought a Ryzen 5 1500X for $150.

Later, in 2020, I bought an MSI motherboard because I was having trouble with random lockups. But I was still on the Ryzen 5 1500X and 16 GB RAM.

This rig did just fine for me for the last six years. Really, for the last nine years. I’d kept a video card from 2013, though, and it has only 3 GB of RAM. Okay, back in 2013, I splurged the heck out of my purchase there because the video card was $700 before tax and shipping. But for the games I enjoyed playing (Factorio, Tropico, Endless Space, XCOM2, World of Warcraft), it was plenty.

Somewhat, I’d like to play Baldur’s Gate III next. The 3 GB RAM video card was nowhere near good enough.

May 2026 purchases

  • 32 GB RAM (Man, those prices hurt.)
  • MSI Radeon RX Vega 56 (8G RAM)
  • AMD Ryzen 7 5700X3D CPU

The RX Vega 56 was the video card with the largest amount of RAM that I could get without growing into a PCI 5 slot. The motherboard has only PCI 4 slots, so it didn’t make sense to me to pay more money for a card with a capability it cannot use. In the back of my mind too, I didn’t want to mix a card with PCI 5 capability being stymied by a PCI 4 motherboard. If everything matches and lines up, then there won’t be weird problems because some software expects PCI 5 access.

I did want the Ryzen 7 5700X3D processor because it is built with a 7 nm process (oh my goodness!) and consumes (only) 105 watts. The X3D part of it has a whopping 93 MB of L3 cache.

Well, when I received the parts, of course I added the RAM first. Easy peasy, but it didn’t change things much. I wasn’t doing much work that needed RAM.

I went to install the RX Vega 56, and… this video card is about 1/4″ too long for my case. Oof.

Well, way back when, I’d built a Windows 7 box and bought gear for it to be beefy. It sat in a large and quiet Antec P180 case, with an Antec 850 watt power supply.1 So I’ve got a case large enough, and it has the power supply with the extra connections for the RX Vega 56. I’m just going to have to find the time to do the migration.

Migration

I had moved the Antec case / Windows 7 box into my old office, where I used to work from home, and then pulled all the components out.

Last week, on Wednesday, everything lined up, and the whole day was mine. I started with vacuuming and dust blowing the Antec case. Then I powered down my main machine and started the migration process. Between the cleaning and component moves, it took about five hours before I hit the power switch on the new rig.2 It had the same MSI motherboard but now with the Radeon RX Vega 56 and the Ryzen 7 5700X3D.

The magic smoke did not escape! In short order, Debian was running, and I was happy. I did move the audio from electrical to S/PDIF (IEC 60958-3).

The project

I asked an AI to guide me through adding ROCm to this machine. It did so pretty easily. However, the devil is in the details, and pretty quickly, I was running into all sorts of obstacles. It helped that I had a specific goal: turn an MP3 file into a summary of a meeting.3

The AI guided me through installing llama.cpp4 with HIP to connect to the hardware via ROCm. There were numerous packages that HIP expected would be there that I had to find out about because they were missing. Thankfully, pasting the messages of the failed compiles would lead the AI to figuring out what was missing.

By the time I was done, I’d added all these:

rocminfo
clinfo
hipcc
cmake
build-essential
git
pkg-config
mesa-utils
libamdhip64-dev
rocm-cmake
clang
llvm
libclang-rt-dev
libclang-rt-19-dev
libomp-17-dev
libhipblas-dev
libcurl4-openssl-dev

One of the snags was that llama.cpp had a dependency on HIP 6.1, but my hardware can only do HIP 5.7.

git clone https://github.com/ggml-org/llama.cpp.git llama.cpp-rocm57

Amazingly, the AI guided me through downloading the llama.cpp-rocm57 source code, find the check-in for the code that depends on HIP version 6.1, and guided me through deleting it!

This worked!

I am an utter git newbie, but the AI has enough training to guide me through the whole process.

After getting the initial part of ROCm, HIP and llama.cpp-rocm57 installed, I asked about how to do transcription. “Please guide me through installing an LLM to do speech to text, inputting an mp3 file recording of a meeting, and outputting a transcript of what was said.”

The AI then guided me through installing Whisper (whisper.cpp). That went easy. I got about 11,000 words out of the transcription.

Then I asked how to summarize it. It generated a prompt file for me, and threw that at llama.cpp-rocm57. I did run into the problem that the 11,000 words were too much for the 8 GB video card. But I split the transcript file in two, and ran llama.cpp-rocm57 on each piece.

It worked.

I even got a nice little summary like this:

Action items

ActionOwnerDeadlineEvidence/notes
Find a cookUnclearUnclearThe group is looking for a cook.
Decide on menuUnclearUnclearThe group needs to decide on a menu.
Determine ticket priceUnclearUnclearThe price of the event is still uncertain.

This is just so cool: from an hour-long MP3 recording of the first planning meeting to a list of action items, all on my local machine. I’m thrilled.

  1. Foolishly, I hadn’t kept on with the task of installing a Bitcoin miner in it. I would later find in this machine a Bitcoin wallet from summer 2014. But back in 2014, the idea of running an 850 watt power supply full-time during summer really did not appeal to me. But if I had been able to mine 1 Bitcoin, that would be worth about $80,000 today. ↩︎
  2. I’m retired now, so I don’t have to be in a rush. ↩︎
  3. I had volunteered to be the project lead on a New Year’s Eve party. I had recorded the first meeting on my Sony ICD-UX570. In the past, I would pay Amazon to transcribe it for me, but that would cost around $3. I wasn’t terribly thrilled about uploading a recording of a meeting to the cloud, either. ↩︎
  4. I tagged this post with “Facebook” because although I have my beef with them, they do provide their large language model, Llama, as open source and for free. Credit where credit is due, man. ↩︎

Leave a Reply