|
ailia_llm
1.5.3.0
|
| Component | Environment |
|---|---|
| Board / SoC | Advantech AOM-2721 / Qualcomm QCS6490 |
| Yocto | Based on 4.0.22 (Kirkstone) |
| OS | QCOM Robotics Reference Distro with ROS 1.3-ver.1.1 |
| OS ID | qcom-robotics-ros2-humble |
| Linux kernel | 6.6.52-yl01301-ga97804e47e26 |
| CPU architecture | aarch64 |
| glibc | 2.35 |
| Python | 3.10.15 |
| QNN runtime | QAIRT 2.47.0.260601 / Hexagon v68 |
The Linux ARM64 binaries require at most glibc 2.35 and GLIBCXX_3.4.29. They can use the system C++ runtime in the environment listed below.
Generate text with the public QCS6490 model. The BSP must provide CDSP firmware, FastRPC libcdsprpc.so, and access to the CDSP device. Install a valid ailia LLM license as described in the setup guide.
Extract the ailia LLM SDK package and download:
The cpp/prebuilt/linux/arm64-v8a/ directory contains prebuilt samples, matching ailia LLM, QNN plug-in, and GGML / llama shared libraries. No sample build is needed. Copy Qualcomm libraries from the extracted QAIRT SDK:
Place QNN libraries and the v68 skel beside libailia_llm_qnn.so. The sample and libraries load their dependencies from their own folder, so LD_LIBRARY_PATH is not needed. The system libstdc++ (GLIBCXX_3.4.29 or later) is used as the C++ runtime. The model extension selects QNN; text generation uses HTP / NPU. The sample runs two conversation turns and prints the responses.
Use Python 3 and pip3. If pip3 is missing from the BSP, install it first:
Extract the SDK on the board. Run bootstrap.py to copy native libraries before installing:
Alternatively, install the public package with pip3 install ailia_llm.
Copy QNN libraries and the v68 skel into ailia_llm/linux/arm64-v8a/ beneath the Location reported by pip3 show. The system C++ runtime must support GLIBCXX_3.4.29 or later; no additional LD_LIBRARY_PATH setting is needed:
From Python, pass the path to the downloaded .qnn file to the existing open() method. The following example loads the model and generates a response. Get the device model name with get_qnn_model_name(). Pass the downloaded model to open() and print responses incrementally with generate(). This text generation example does not use a multimodal projector or MTP assistant.
| Error | Action |
|---|---|
| GLIBCXX_x.x.xx not found | The system C++ runtime must support GLIBCXX_3.4.29 or later (GCC 11 or later). |
| Cannot load the QNN plug-in | Keep libailia_llm_qnn.so with the matching ailia LLM, GGML and llama shared libraries. |
| QNN initialization fails | Check QAIRT 2.47 Yocto libraries, FastRPC, CDSP firmware, permissions and the v68 skel. |