LLM on Raspberry Pi: GPU offload and cross compiling

Finally got some time to play with my Raspberry Pi again!

In my earlier post "LlamaPi: Experiments with VideoCore GPU", I tried to run llama.cpp on Raspberry Pi's VideoCore GPU through Vulkan. That experiment was not very successful: Although he program could run, the data it generated was garbage and the generation speed is even slower than CPU.

Recently I looked into this experiment again and made some progress.

I got llama.cpp to run with GPU offload on Raspberry Pi and generate correct data. Athough the generation speed with Raspberry Pi GPU was still slower than CPU, it demonstrated the technical feasibilty of GPU offloading on Raspberry Pi.
Built the environment for cross compiling Vulkan program that runs on Raspberry Pi.

Cross Compiling of RPi Vulkan program

Motivation: Native compiling doesn't work

The first challenge was compiling llama.cpp with Vulkan support for Raspberry Pi.

I had the luck to compile the program natively on my Raspberry Pi 4 (4GB RAM) with old llama.cpp codebase (dated 2024/11), but it no longer works with the recent revisions. When I tried to compile recent llama.cpp code with Vulkan support on my RPi4, it struggled for 30+ minutes before crashing with OOM error. So cross compilation is the only option.

Step 1: Download the Toolchain

cd /path/to/workspace

wget https://developer.arm.com/-/media/Files/downloads/gnu/13.3.rel1/binrel/arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-linux-gnu.tar.xz
tar xJf arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-linux-gnu.tar.xz
mv arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-linux-gnu aarch64-toolchain

# Verify
./aarch64-toolchain/bin/aarch64-none-linux-gnu-gcc --version
# Expected output: Arm GNU Toolchain 13.3.Rel1, GCC 13.3.1 20240614

GLIBC version note: This toolchain bundles GLIBC 2.38, while RPi4 typically has GLIBC 2.31. This is addressed later by bundling the toolchain's GLIBC alongside the binary.

Step 2: Build Shader Compiler

glslc runs on the HOST (x86₆₄) and compiles GLSL shaders to SPIR-V at build time. It is required by the Vulkan CMake find_package.

cd /path/to/workspace

git clone https://github.com/google/shaderc.git
cd shaderc
python3 utils/git-sync-deps
cmake -G Ninja -S . -B build -DSHADERC_SKIP_TESTS=ON -DCMAKE_BUILD_TYPE=Release
ninja -C build glslc

# Verify
./build/glslc/glslc --version
# Expected: shaderc v2026.x, spirv-tools, glslang

Step 3: Download Updated Vulkan Headers

The Vulkan headers from Ubuntu 20.04 (libvulkan-dev) are too old — modern llama.cpp's ggml-vulkan uses Vulkan 1.3 APIs. I need to download the latest:

cd /path/to/workspace

wget https://github.com/KhronosGroup/Vulkan-Headers/archive/refs/tags/v1.3.296.tar.gz
tar xzf v1.3.296.tar.gz

# Verify
ls Vulkan-Headers-1.3.296/include/vulkan/vulkan.h
ls Vulkan-Headers-1.3.296/include/vk_video/

Its contents will be merged into the toolchain's sysroot later.

Step 4: Setup arm64 sysroot

The toolchain's sysroot has basic C/C++ libraries but is missing Vulkan. I copied the missing libraries from the arm64/Ubuntu 20.04 Docker container:

4a) Start the Docker container:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

# Start arm64 Ubuntu 20.04 container (keep it running in background)
docker run -d -it --platform linux/arm64 \
    --name rpi-sysroot \
    arm64v8/ubuntu:20.04

4b) Install the Vulkan packages inside the container:

docker exec rpi-sysroot bash -c "
    apt update && apt install -y libvulkan-dev libvulkan1 spirv-headers
"

NOTE: spirv-headers provides SPIR-V header files. On Ubuntu 20.04 it does NOT include a CMake config file, so one will be created manually in Step 4d.

4c) Copy Vulkan libraries from the container to sysroot:

cd /path/to/workspace

# --- Vulkan headers (from container) ---
docker cp rpi-sysroot:/usr/include/vulkan \
    aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/include/

# --- SPIRV headers (from container) ---
docker cp rpi-sysroot:/usr/include/spirv \
    aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/include/

# --- Vulkan library (copy actual file, not the symlink) ---
# docker cp copies symlinks as-is, so copy the target file then recreate symlinks
docker cp rpi-sysroot:/usr/lib/aarch64-linux-gnu/libvulkan.so.1.2.131 \
    aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/lib64/
cd aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/lib64
ln -sf libvulkan.so.1.2.131 libvulkan.so.1
ln -sf libvulkan.so.1 libvulkan.so

# --- Vulkan-Headers 1.3.296 (overwrite the container's older headers) ---
cd /path/to/workspace
cp -r Vulkan-Headers-1.3.296/include/vulkan \
    aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/include/
cp -r Vulkan-Headers-1.3.296/include/vk_video \
    aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/include/

4d) Create SPIRV-headers CMake config

The Ubuntu 20.04 spirv-headers package only installs headers, no CMake config. I need to create one:

cd /path/to/workspace/
mkdir -p aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/lib64/cmake/SPIRV-Headers

cat > /path/to/workspace/aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/lib64/cmake/SPIRV-Headers/SPIRV-HeadersConfig.cmake << 'EOF'
# Minimal SPIRV-Headers config for cross-compilation
add_library(SPIRV-Headers::SPIRV-Headers INTERFACE IMPORTED)
set_target_properties(SPIRV-Headers::SPIRV-Headers PROPERTIES
    INTERFACE_INCLUDE_DIRECTORIES "${CMAKE_CURRENT_LIST_DIR}/../../../include"
)
EOF

Alternatives: There might be other ways to setup all these. E.g. copying the files from RPi4 directly, or extracting from arm64 deb packages.

The resulting sysroot should have things like these:

ls /path/to/workspace/aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/include/vulkan/vulkan.h
ls /path/to/workspace/aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/include/spirv/unified1/spirv.h
ls /path/to/workspace/aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/lib64/libvulkan.so
ls /path/to/workspace/aarch64-toolchain/aarch64-none-linux-gnu/libc/usr/lib64/cmake/SPIRV-Headers/SPIRV-HeadersConfig.cmake
ls /path/to/workspace/aarch64-toolchain/aarch64-none-linux-gnu/libc/lib/ld-linux-aarch64.so.1

Step 5: CMake for the toolchain

Create the file /path/to/workspace/aarch64-rpi4-toolchain.cmake:

# aarch64-rpi4-toolchain.cmake
# Cross-compilation toolchain for Raspberry Pi 4 (aarch64)

set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR aarch64)

# Workspace root (directory containing this file)
get_filename_component(WORKSPACE_DIR "${CMAKE_CURRENT_LIST_DIR}" ABSOLUTE)

set(TOOLCHAIN_DIR    "${WORKSPACE_DIR}/aarch64-toolchain")
set(SYSROOT_DIR      "${TOOLCHAIN_DIR}/aarch64-none-linux-gnu/libc")
set(GLSLC_EXECUTABLE "${WORKSPACE_DIR}/shaderc/build/glslc/glslc")

# Cross-compilers
set(CMAKE_C_COMPILER   "${TOOLCHAIN_DIR}/bin/aarch64-none-linux-gnu-gcc")
set(CMAKE_CXX_COMPILER "${TOOLCHAIN_DIR}/bin/aarch64-none-linux-gnu-g++")

# Sysroot
set(CMAKE_SYSROOT      "${SYSROOT_DIR}")
set(CMAKE_FIND_ROOT_PATH "${SYSROOT_DIR}")

# Only search sysroot for libraries/headers/packages (not host paths)
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

# Vulkan-specific: glslc is a HOST (x86_64) tool, not in sysroot
set(Vulkan_GLSLC_EXECUTABLE "${GLSLC_EXECUTABLE}")
set(Vulkan_INCLUDE_DIR "${SYSROOT_DIR}/usr/include")
set(Vulkan_LIBRARY    "${SYSROOT_DIR}/usr/lib64/libvulkan.so")

# Architecture flags
set(CMAKE_C_FLAGS   "${CMAKE_C_FLAGS}   -march=armv8-a+crc+simd -Wno-psabi")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=armv8-a+crc+simd -Wno-psabi")

Step 6: Configure and Build

CMake config for cross compiling

cd /path/to/workspace/llama.cpp

cmake -B ../build-cross-vulkan \
    -DCMAKE_TOOLCHAIN_FILE=../aarch64-rpi4-toolchain.cmake \
    -DGGML_VULKAN=ON \
    -DGGML_VULKAN_RUN_TESTS=OFF \
    -DGGML_VULKAN_VALIDATE=OFF \
    -DGGML_OPENMP=OFF \
    -DGGML_BLAS=OFF \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=OFF \
    -DLLAMA_CURL=OFF \
    -DLLAMA_SERVER=OFF \
    -DCMAKE_EXE_LINKER_FLAGS="-static-libgcc -static-libstdc++"

Cross compile llama.cpp:

cmake --build ../build-cross-vulkan -j$(nproc)

Check the resulting binary:

file /path/to/workspace/build-cross-vulkan/bin/llama-cli
# Expected: ELF 64-bit LSB executable, ARM aarch64

Step 7: Packaging for deployment on RPi4

The ARM toolchain I downloaded links against GLIBC 2.38, but RPi4 likely has GLIBC 2.31. Bundling the toolchain's GLIBC alongside the binary resolves this issue.

cd /path/to/workspace

mkdir -p deploy-vulkan/lib

# Binary
cp build-cross-vulkan/bin/llama-cli deploy-vulkan/

# Bundled GLIBC (from toolchain sysroot)
SYSROOT=aarch64-toolchain/aarch64-none-linux-gnu/libc
cp $SYSROOT/lib64/libc.so.6           deploy-vulkan/lib/
cp $SYSROOT/lib64/libm.so.6           deploy-vulkan/lib/
cp $SYSROOT/usr/lib64/libgcc_s.so.1   deploy-vulkan/lib/
cp $SYSROOT/lib/ld-linux-aarch64.so.1 deploy-vulkan/lib/

# Create tarball
tar czf llama-rpi4-vulkan.tar.gz deploy-vulkan/

The tarball is ready for deployment on RPi4 now!

Resolve the "Hard Error" from ggml-vulkan

The V3D 4.2 GPU has 16KB of shared memory. Most quantization types fit, but IQ2_S requires 16,640 bytes (256 bytes over). The ggml-vulkan initialization loop iterates through ALL types and throws a hard error on the first type that doesn't fit. To resolve this, I changed it to graceful degradation (by just disabling the type instead of failing entirely).

In llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp (around line 4012):

// BEFORE (lines 4012-4014) — throws hard error on ANY type that doesn't fit:
if (!ggml_vk_matmul_shmem_support(device, s_warptile_mmq, false, t)) {
    std::cerr << "ggml_vulkan: Error: Shared memory size too small for matrix multiplication." << std::endl;
    throw std::runtime_error("Shared memory size too small for matrix multiplication.");
}

// AFTER — gracefully degrades (disables matmul for that type, same pattern as lines 4015-4020):
if (!ggml_vk_matmul_shmem_support(device, s_warptile_mmq, false, t)) {
    device->mul_mat_s[i] = false;
    device->mul_mat_m[i] = false;
    device->mul_mat_l[i] = false;

Build and package the tarball for deployment again.

Testing the result: GPU offloading on RPi4

Copy the tarball to RPi4, extract it, and run the llama.cpp CLI:

scp /path/to/workspace/llama-rpi4-vulkan.tar.gz user@<rpi4-ip>:~/

Run llama.cpp with GPU offloading:

# On RPi4
tar xzf llama-rpi4-vulkan.tar.gz
cd deploy-vulkan

# Run with bundled GLIBC (use ld-linux directly, not the system's dynamic linker)
./lib/ld-linux-aarch64.so.1 --library-path ./lib ./llama-cli \
    -m /path/to/model.gguf \
    -p "Hello" -n 16 -t 1 -ngl 1

Key to the LD_LIBRARY_PATH approach: By invoking ld-linux-aarch64.so.1 directly with --library-path ./lib, we use the toolchain's GLIBC 2.38 .so files while ignoring the system's GLIBC 2.31. This avoids the GLIBC_2.38 not found error.

I tested with TinyLlama 1.1B Q4_K_M and got something like this:

> Hi Certainly! Here's an example use case for the keyword filter function

[ Prompt: 2.5 t/s | Generation: 1.0 t/s ]

It was slower than CPU, but at least it didn't generate garbage. :-) This means the pipeline was working correctly.

NOTE: If you see errors like GGML_ASSERT(n_tokens_all <= cparams.n_batch) failed, increase batch size by adding -b 12 -c 12 to the command line argument.

Discussion/Open Questions

Generation speed with GPU offloading was slower than CPU. This is expected and aligns with my previous experiments. But at least we are not getting garbage data this time.
The program hangs with ngl larger than 1. While the root cause is still being investigated, I'm pretty sure the root cause was: Vulkan backend claims to support all MUL_MAT ops without checking whether the pipeline actually exists, so when shmem limits forces mul_mat_mi = false, the scheduler never falls back to CPU and submits a missing pipeline to the GPU.

Conclusion

It was an interesting journey from the question "can I run LLMs faster with Raspberry Pi's GPU" to a working prototype that demonstrates the GPU offloading idea. Although the answer was "it's even slower than CPU", I enjoyed the learnings from the experiments. I believe these learnings will be useful to running AI models on other edge devices too.

Cross Compiling of RPi Vulkan program#

Motivation: Native compiling doesn't work#

Step 1: Download the Toolchain#

Step 2: Build Shader Compiler#

Step 3: Download Updated Vulkan Headers#

Step 4: Setup arm64 sysroot#

Step 5: CMake for the toolchain#

Step 6: Configure and Build#

Step 7: Packaging for deployment on RPi4#

Resolve the "Hard Error" from ggml-vulkan#

Testing the result: GPU offloading on RPi4#

Discussion/Open Questions#

Conclusion#