Hello,
My AI HAT+ module was working initially and I was able to use Degirum SDK to run models like scdepthv3 and yolo8x without issues. However, an error began to occur when I tried to run both yolo8x and scdepthv3 simultaneously, and the Raspberry Pi stopped detecting the Hailo NPU.
After this, I tried multiple debugging steps, but the Hailo NPU failed to be detected. I also mistakenly removed the PCIe cable while the Pi was still running.
After the hot-unplug, hailortcli scan fails to connect unless I manually force a PCIe reset using:
bash
echo 1 | sudo tee /sys/bus/pci/devices/0001:01:00.0/reset
or
bash
echo 1 | sudo tee /sys/bus/pci/devices/0001:01:00.0/remove
echo 1 | sudo tee /sys/bus/pci/rescan
```
After running these commands, `/dev/hailo0` appears, but when I try to use the device I get:
```
$ hailortcli fw-control identify
[HailoRT] [warning] FW is not loaded to the device. Please load FW before using the device.
Executing on device: 0001:01:00.0
[HailoRT] [error] Ioctl HAILO_FW_CONTROL failed with 19. Read dmesg log for more info
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) - Failed in fw_control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) - Failed to send fw control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36)
Failed to execute on device: 0001:01:00.0. status= HAILO_DRIVER_OPERATION_FAILED(36)
```
**Kernel logs show:**
```
[ 3.941930] hailo 0001:01:00.0: Writing file hailo/hailo8_fw.bin
[ 4.010889] hailo 0001:01:00.0: File hailo/hailo8_fw.bin written successfully
[ 4.011002] hailo 0001:01:00.0: File hailo/hailo8_board_cfg.bin written successfully
[ 4.011535] hailo 0001:01:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[ 9.180756] hailo 0001:01:00.0: Timeout waiting for NNC firmware file, boot status 4294967295
[ 9.180766] hailo 0001:01:00.0: Firmware load failed
[ 9.180828] hailo 0001:01:00.0: Failed activating board -110
[ 9.180850] hailo 0001:01:00.0: probe with driver hailo failed with error -110
```
When attempting resets, I also see:
```
[ 215.394330] hailo 0001:01:00.0: not ready 1023ms after FLR; waiting
[ 283.618729] hailo 0001:01:00.0: not ready 65535ms after FLR; giving up
[ 611.492005] hailo 0001:01:00.0: Failed hailo pcie soft reset. err -110
[ 686.409781] hailo 0001:01:00.0: Failed writing fw control to pcie
[ 700.880031] hailo 0001:01:00.0: Device disconnected while opening device
I have tried:
Multiple complete power cycles (including 10+ minute waits with module removed)
Full reinstall of Hailo software packages
When I run sudo setpci -s 0001:01:00.0 0x0.L, it returns 28641e60 (the correct device ID), so the chip does respond to basic PCIe config reads. But the Neural Network Core won’t boot - it just returns boot status 0xFFFFFFFF.
Can anyone please help me?
My AI HAT+ module was working initially and I was able to use Degirum SDK to run models like scdepthv3 and yolo8x without issues. However, an error began to occur when I tried to run both yolo8x and scdepthv3 simultaneously, and the Raspberry Pi stopped detecting the Hailo NPU.
After this, I tried multiple debugging steps, but the Hailo NPU failed to be detected. I also mistakenly removed the PCIe cable while the Pi was still running.
After the hot-unplug, hailortcli scan fails to connect unless I manually force a PCIe reset using:
bash
echo 1 | sudo tee /sys/bus/pci/devices/0001:01:00.0/reset
or
bash
echo 1 | sudo tee /sys/bus/pci/devices/0001:01:00.0/remove
echo 1 | sudo tee /sys/bus/pci/rescan
```
After running these commands, `/dev/hailo0` appears, but when I try to use the device I get:
```
$ hailortcli fw-control identify
[HailoRT] [warning] FW is not loaded to the device. Please load FW before using the device.
Executing on device: 0001:01:00.0
[HailoRT] [error] Ioctl HAILO_FW_CONTROL failed with 19. Read dmesg log for more info
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) - Failed in fw_control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36) - Failed to send fw control
[HailoRT] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36)
[HailoRT CLI] [error] CHECK_SUCCESS failed with status=HAILO_DRIVER_OPERATION_FAILED(36)
Failed to execute on device: 0001:01:00.0. status= HAILO_DRIVER_OPERATION_FAILED(36)
```
**Kernel logs show:**
```
[ 3.941930] hailo 0001:01:00.0: Writing file hailo/hailo8_fw.bin
[ 4.010889] hailo 0001:01:00.0: File hailo/hailo8_fw.bin written successfully
[ 4.011002] hailo 0001:01:00.0: File hailo/hailo8_board_cfg.bin written successfully
[ 4.011535] hailo 0001:01:00.0: File hailo/hailo8_fw_cfg.bin written successfully
[ 9.180756] hailo 0001:01:00.0: Timeout waiting for NNC firmware file, boot status 4294967295
[ 9.180766] hailo 0001:01:00.0: Firmware load failed
[ 9.180828] hailo 0001:01:00.0: Failed activating board -110
[ 9.180850] hailo 0001:01:00.0: probe with driver hailo failed with error -110
```
When attempting resets, I also see:
```
[ 215.394330] hailo 0001:01:00.0: not ready 1023ms after FLR; waiting
[ 283.618729] hailo 0001:01:00.0: not ready 65535ms after FLR; giving up
[ 611.492005] hailo 0001:01:00.0: Failed hailo pcie soft reset. err -110
[ 686.409781] hailo 0001:01:00.0: Failed writing fw control to pcie
[ 700.880031] hailo 0001:01:00.0: Device disconnected while opening device
I have tried:
Multiple complete power cycles (including 10+ minute waits with module removed)
Full reinstall of Hailo software packages
When I run sudo setpci -s 0001:01:00.0 0x0.L, it returns 28641e60 (the correct device ID), so the chip does respond to basic PCIe config reads. But the Neural Network Core won’t boot - it just returns boot status 0xFFFFFFFF.
Can anyone please help me?
Statistics: Posted by Abcmouse — Sat Nov 29, 2025 1:03 am — Replies 0 — Views 30