ailia_llm  1.5.3.0
Running QNN on Yocto Linux

Validated environment

Component Environment
Board / SoC Advantech AOM-2721 / Qualcomm QCS6490
Yocto Based on 4.0.22 (Kirkstone)
OS QCOM Robotics Reference Distro with ROS 1.3-ver.1.1
OS ID qcom-robotics-ros2-humble
Linux kernel 6.6.52-yl01301-ga97804e47e26
CPU architecture aarch64
glibc 2.35
Python 3.10.15
QNN runtime QAIRT 2.47.0.260601 / Hexagon v68

The Linux ARM64 binaries require at most glibc 2.35 and GLIBCXX_3.4.29. They can use the system C++ runtime in the environment listed below.

Generate text with the public QCS6490 model. The BSP must provide CDSP firmware, FastRPC libcdsprpc.so, and access to the CDSP device. Install a valid ailia LLM license as described in the setup guide.

1. Get the required files (host PC)

Extract the ailia LLM SDK package and download:

The cpp/prebuilt/linux/arm64-v8a/ directory contains prebuilt samples, matching ailia LLM, QNN plug-in, and GGML / llama shared libraries. No sample build is needed. Copy Qualcomm libraries from the extracted QAIRT SDK:

export LLM_PACKAGE=/path/to/ailia_llm_package
export QNN_SDK_PATH=/path/to/qairt/2.47.0.260601
export DEPLOY="$PWD/llm-qnn-yocto"
mkdir -p "$DEPLOY"
cp -a "$LLM_PACKAGE/cpp/prebuilt/linux/arm64-v8a/." "$DEPLOY/"
curl -L --fail -o "$DEPLOY/gemma4-e2b-qcs6490-int16-mtp.qnn" \
https://storage.googleapis.com/ailia-models/gemma/qnn/v1.5.1/gemma4-e2b-qcs6490-int16-mtp.qnn
cp "$QNN_SDK_PATH/lib/aarch64-oe-linux-gcc11.2/libQnnHtp.so" "$DEPLOY/"
cp "$QNN_SDK_PATH/lib/aarch64-oe-linux-gcc11.2/libQnnSystem.so" "$DEPLOY/"
cp "$QNN_SDK_PATH/lib/aarch64-oe-linux-gcc11.2/libQnnHtpV68Stub.so" "$DEPLOY/"
cp "$QNN_SDK_PATH/lib/hexagon-v68/unsigned/libQnnHtpV68Skel.so" "$DEPLOY/"
scp -r "$DEPLOY" root@169.254.239.42:/home/root/

2. Run the C++ sample (board)

Place QNN libraries and the v68 skel beside libailia_llm_qnn.so. The sample and libraries load their dependencies from their own folder, so LD_LIBRARY_PATH is not needed. The system libstdc++ (GLIBCXX_3.4.29 or later) is used as the C++ runtime. The model extension selects QNN; text generation uses HTP / NPU. The sample runs two conversation turns and prints the responses.

cd /home/root/llm-qnn-yocto
chmod +x ./ailia_llm_sample
./ailia_llm_sample --model ./gemma4-e2b-qcs6490-int16-mtp.qnn

3. Run from Python (board)

Install the Python package

Use Python 3 and pip3. If pip3 is missing from the BSP, install it first:

curl -L --fail -o get-pip.py https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py

Extract the SDK on the board. Run bootstrap.py to copy native libraries before installing:

cd /path/to/ailia_llm_package/python
python3 bootstrap.py
pip3 install .

Alternatively, install the public package with pip3 install ailia_llm.

Copy QNN libraries into the installed package

Copy QNN libraries and the v68 skel into ailia_llm/linux/arm64-v8a/ beneath the Location reported by pip3 show. The system C++ runtime must support GLIBCXX_3.4.29 or later; no additional LD_LIBRARY_PATH setting is needed:

pip3 show ailia_llm
SITE_PACKAGES=$(pip3 show ailia_llm | sed -n 's/^Location: //p')
LLM_NATIVE="$SITE_PACKAGES/ailia_llm/linux/arm64-v8a"
cd /home/root/llm-qnn-yocto
test -f "$LLM_NATIVE/libailia_llm_qnn.so"
cp libQnnHtp.so libQnnSystem.so libQnnHtpV68Stub.so libQnnHtpV68Skel.so "$LLM_NATIVE/"

Open local models from an application

From Python, pass the path to the downloaded .qnn file to the existing open() method. The following example loads the model and generates a response. Get the device model name with get_qnn_model_name(). Pass the downloaded model to open() and print responses incrementally with generate(). This text generation example does not use a multimodal projector or MTP assistant.

import ailia_llm
llm = ailia_llm.AiliaLLM()
try:
print("QNN model:", llm.get_qnn_model_name())
llm.open("./gemma4-e2b-qcs6490-int16-mtp.qnn", n_ctx=8192)
messages = [{"role": "user", "content": "What is the capital of Japan? Answer briefly."}]
for text in llm.generate(messages):
print(text, end="", flush=True)
print()
finally:
llm.close()

Troubleshooting

Error Action
GLIBCXX_x.x.xx not found The system C++ runtime must support GLIBCXX_3.4.29 or later (GCC 11 or later).
Cannot load the QNN plug-in Keep libailia_llm_qnn.so with the matching ailia LLM, GGML and llama shared libraries.
QNN initialization fails Check QAIRT 2.47 Yocto libraries, FastRPC, CDSP firmware, permissions and the v68 skel.