# BirdNET-RKNN

**Repository Path**: cqnews/BirdNET-RKNN

## Basic Information

- **Project Name**: BirdNET-RKNN
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-09-09
- **Last Updated**: 2026-09-09

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# BirdNET-Rockchip

Run [BirdNET](https://github.com/birdnet-team/BirdNET-Analyzer) bird sound recognition on Rockchip NPU (RK3566/RK3568/RK3588).

This project extracts the BirdNET CNN from the original Keras model, converts it to RKNN format, and provides efficient inference on Rockchip's NPU. Achieves **~240ms inference time** for 3-second audio clips on RK3566 - **12x faster than real-time**.

## Features

- 🚀 NPU-accelerated inference on Rockchip SoCs
- 🎤 Real-time audio detection from microphone or audio files
- 🌍 Location-based species filtering using BirdNET meta model
- 🔋 Low power consumption (~0.1W average with 8% duty cycle)
- 📦 Pre-computed location masks for offline operation

## Hardware Requirements

- Rockchip RK3566, RK3568, or RK3588 board (tested on Radxa Zero 3W)
- USB microphone or audio input
- Ubuntu 22.04/24.04 with NPU driver enabled

## Quick Start

### 1. Install Dependencies

```bash
# On RK3566 board
pip install rknn-toolkit-lite2 numpy soundfile scipy sounddevice

# For development machine (model conversion)
pip install rknn-toolkit2 tensorflow onnx onnxruntime
```

### 2. Download Models

Download from [BirdNET-Analyzer releases](https://github.com/birdnet-team/BirdNET-Analyzer/releases):
- `BirdNET_GLOBAL_6K_V2.4_Model_FP32.tflite` (acoustic model)
- `BirdNET_GLOBAL_6K_V2.4_MData_Model_FP32.tflite` (meta model)
- `BirdNET_GLOBAL_6K_V2.4_Labels.txt` (species labels)

Or use the original Keras models and convert them yourself (see [Model Conversion](docs/MODEL_CONVERSION.md)).

### 3. Convert Model to RKNN

```bash
# On development machine with rknn-toolkit2
python scripts/convert_to_rknn.py \
    --model models/birdnet_cnn_static.onnx \
    --output models/birdnet_cnn.rknn \
    --target rk3566
```

### 4. Generate Location Mask (Optional)

```bash
# Pre-compute species probabilities for your location
python tools/compute_location_probs.py \
    --lat 39.0 --lon -77.0 \
    --model models/meta-model.tflite \
    --output models/my_location.npz
```

### 5. Run Detection

```bash
# Test with audio file
python src/detector.py \
    --model models/birdnet_cnn.rknn \
    --labels models/birdnet_labels.txt \
    --location-mask models/my_location.npz \
    --audio test_audio.wav

# Real-time detection from microphone
python src/detector.py \
    --model models/birdnet_cnn.rknn \
    --labels models/birdnet_labels.txt \
    --location-mask models/my_location.npz \
    --realtime
```

## Performance

| Metric | RK3566 (FP16) |
|--------|---------------|
| Inference time | ~240ms |
| Real-time factor | 0.08x |
| Throughput | 4.2 inferences/sec |
| Power (during inference) | ~1-1.5W |
| Power (average @ 1s interval) | ~0.1-0.15W |

## Architecture

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│   Audio Input   │────▶│  Preprocessing  │────▶│   NPU Inference │
│   (3s @ 48kHz)  │     │  (CPU, ~5ms)    │     │   (~240ms)      │
└─────────────────┘     └─────────────────┘     └────────┬────────┘
                                                         │
┌─────────────────┐     ┌─────────────────┐              │
│ Location Mask   │────▶│    Filtering    │◀─────────────┘
│ (pre-computed)  │     │                 │
└─────────────────┘     └────────┬────────┘
                                 │
                                 ▼
                        ┌─────────────────┐
                        │   Detections    │
                        │  (species, %)   │
                        └─────────────────┘
```

## Preprocessing Details

BirdNET uses a dual-channel mel spectrogram:

| Channel | Frequency Range | FFT Size | Purpose |
|---------|----------------|----------|---------|
| 1 | 0-3 kHz | 2048 | Low frequency detail |
| 2 | 500 Hz-15 kHz | 1024 | Broad frequency coverage |

Key parameters:
- Sample rate: 48 kHz
- Duration: 3 seconds (144,000 samples)
- Output shape: (1, 96, 512, 2) → transposed to (1, 512, 2, 96) for RKNN

## Location Filtering

The BirdNET meta model predicts species occurrence probability based on latitude, longitude, and week of year (1-48). Species below the threshold (default 0.03) are filtered out.

For stationary deployments, pre-compute masks for all 48 weeks:
```bash
python tools/compute_location_probs.py --lat YOUR_LAT --lon YOUR_LON -o location.npz
```

## Documentation

- [Hardware Setup](docs/SETUP.md) - NPU driver installation, board configuration
- [Model Conversion](docs/MODEL_CONVERSION.md) - Converting BirdNET to RKNN
- [Deployment Guide](docs/DEPLOYMENT.md) - Production deployment tips

## File Descriptions

| File | Description |
|------|-------------|
| `src/preprocessing.py` | Mel spectrogram generation matching BirdNET |
| `src/detector.py` | Main inference script with NPU support |
| `src/location_filter.py` | Location-based species filtering |
| `scripts/extract_cnn_from_keras.py` | Extract CNN from Keras H5 model |
| `scripts/convert_to_rknn.py` | ONNX to RKNN conversion |
| `tools/compute_location_probs.py` | Generate location probability masks |

## Troubleshooting

### NPU not detected
```bash
# Check if NPU driver is loaded
dmesg | grep -i npu

# Enable NPU overlay (Ubuntu Rockchip)
sudo nano /boot/extlinux/extlinux.conf
# Add: fdtoverlays /path/to/rk3568-npu-enable.dtbo
```

### Wrong predictions
- Verify input shape is (1, 512, 2, 96) for RKNN
- Check that preprocessing matches BirdNET exactly
- Ensure labels file matches model version

### Slow inference
- Confirm NPU is being used (check dmesg for RKNPU messages)
- Use FP16 model for best speed/accuracy tradeoff

## Credits

- [BirdNET](https://github.com/birdnet-team/BirdNET-Analyzer) - Original model by Cornell Lab of Ornithology
- [RKNN-Toolkit2](https://github.com/airockchip/rknn-toolkit2) - Rockchip NPU SDK

## License

This project is licensed under the MIT License. Note that BirdNET models have their own [license terms](https://github.com/birdnet-team/BirdNET-Analyzer/blob/main/LICENSE).

## Contributing

Contributions welcome! Please open an issue or PR.