# smart_enclosure_server

**Repository Path**: seaoftranquility/smart_enclosure_server

## Basic Information

- **Project Name**: smart_enclosure_server
- **Description**: Smart regulator server, relays command and live video, future data analysis and computer vision features
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-07-12
- **Last Updated**: 2026-08-31

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Server — Smart Enclosure Regulator

FastAPI backend connecting ESP32-S3 devices, Mosquitto, PostgreSQL, and the WeChat Mini Program.

## Architecture

- FastAPI/SQLModel application on internal port 8000
- PostgreSQL 16 for users, devices, profiles, care records, and pairing state
- Mosquitto on TLS-only port 8883 with Dynamic Security
- Nginx on 80/443 for HTTP(S) and WebSocket proxying
- one active telemetry socket and one active video socket per device
- one Uvicorn worker because socket/frame state is in memory

## Device identity and pairing

Firmware generates:

- `device_secret`: permanent private 32-hex credential
- `pairing_code`: visible `XXXX-XXXX` one-time password

Registration:

```http
POST /api/devices/register
Content-Type: application/json

{
  "device_id": "AA:BB:CC:11:22:33",
  "device_secret": "0123456789abcdef0123456789abcdef",
  "pairing_code": "ABCD-2345"
}
```

The server authenticates repeat registration with the private credential, hashes the OTP, and opens a five-minute binding window. It also provisions a Mosquitto client whose username is the MAC address and whose ACLs are restricted to that device's topics.

The Mini Program binds with:

```http
POST /api/devices/bind
Authorization: Bearer <JWT>
Content-Type: application/json

{"secret":"ABCD-2345"}
```

Successful binding clears the OTP hash. Unbind publishes the device command before clearing ownership.

## MQTT

External broker access is TLS-only on port 8883. Anonymous access is disabled.

| Topic | Direction | QoS |
|---|---|---:|
| `device/{id}/telemetry` | device → server | 0 |
| `device/{id}/status` | device → server | 1/LWT |
| `device/{id}/command` | server → device | 1 |
| `device/{id}/config` | server → device | 1 |

Status uses `{"online":true}` and `{"online":false}` as defined by `../docs/protocol-contract.json`.

## Local backend development

Prerequisites: Python 3.12+ and `uv`.

```powershell
cd server
$env:APP_DEV_MODE='true'
$env:APP_MQTT_TLS='false'
uv run uvicorn app.main:app --reload
```

Without a local broker the HTTP API remains usable, while MQTT-dependent operations return 503 or retry in the background.

Run tests:

```powershell
uv run --frozen pytest -q
uv run alembic -c config/alembic.ini current
```

For a read-only camera/MQTT deployment report that excludes passwords and private keys:

```bash
cd /opt/smart-regulator/server
sudo bash scripts/collect-video-diagnostics.sh | tee /tmp/video-diagnostics.txt
```

Current migration head: `0003_pairing_otp`.

## Docker deployment

Target: Ubuntu 22.04/24.04 with Docker Compose, OpenSSL, Bash, and at least 2 GB free disk.

Human operators should keep [the deployment repair runbook](docs/deployment-repair.md) available before changing persistent volumes, credentials, or certificates. It includes the verified failure history, directory rules, manual diagnostics, secret rotation, and recovery commands.

From the repository's `server/` directory:

```bash
sudo ./scripts/deploy.sh --dry-run
sudo ./scripts/deploy.sh
```

The deployment script:

1. scans ports and existing services
2. copies the server source into `/opt/smart-regulator/server`
3. generates JWT and MQTT administrator secrets
4. generates a private MQTT CA and broker certificate
5. configures UFW for 22, 80, 443, and 8883
6. builds and starts PostgreSQL, Mosquitto, API, Nginx, and certbot

After deployment, copy `config/mosquitto/certs/ca.crt` into firmware as `MQTT_CA_CERT_PEM` and set the firmware broker hostname to the certificate's public DNS SAN.

### Recovering a Mosquitto permission failure

If logs contain `Unable to load server key file`, `Permission denied`, or `Dynamic security plugin config ... is not readable`, update the repository and rerun the deployment in update mode:

```bash
cd /path/to/your/cloned/repository/server
git pull
sudo ./scripts/deploy.sh --update
```

For an IP-only development deployment, reissue only the broker server certificate from the preserved CA with the public IP in its SAN:

```bash
sudo ./scripts/deploy.sh --update --renew-certs --mqtt-host 106.55.151.104
```

Afterward, copy the deployed `config/mosquitto/certs/ca.crt` into firmware. Do not copy or expose `ca.key` or `server.key`.

`deploy.sh` persists the selected broker identity as `MQTT_PUBLIC_HOST` in `.env`. If no domain or explicit host is supplied, it explicitly requests the public IPv4 address and fails closed if detection is unavailable. Every deployment verifies both the public SAN and internal `DNS:mosquitto`; SAN drift automatically reissues only the server certificate from the preserved CA and recreates Mosquitto. After startup it also compares SHA-256 hashes across the host files, container source mounts, container runtime copies, and the leaf certificate actually served on port 8883. A mismatch makes deployment fail and the script prints the CA SHA-256 fingerprint that production firmware must trust.

Deployment recreates service containers after replacing the deployment directory. A process restart is insufficient for bind-mounted files replaced with `mv` or restored into a newly created directory because the old container can retain the deleted inode. `docker compose up -d --force-recreate` refreshes those mounts while preserving the PostgreSQL and Mosquitto named volumes and their data.

### Certificate strategy for manufactured devices

Do not create a new CA for every device or deployment. Treat `ca.crt` as the product trust anchor: embed the same public CA in a product generation or fleet, keep encrypted offline backups, and never distribute `ca.key` to devices. Broker `server.crt` and `server.key` are replaceable leaf credentials and can be renewed repeatedly from that preserved CA without reflashing devices. A device needs new firmware only when the product CA itself is intentionally rotated, compromised, or expires.

For production, keep the CA private key in an offline signing system or secrets manager and deploy only signed broker leaf certificates. Plan CA rollover before expiry by shipping firmware that trusts both the old and new CA during a transition window, then move the broker to the new chain and later remove the old CA. Record the CA SHA-256 fingerprint in the manufacturing release manifest and compare it with the fingerprint printed by `deploy.sh`.

The updated broker entrypoint repairs ownership of the existing `mosq_data` named volume and stages the root-owned TLS key into a broker-readable runtime directory. Do not delete `mosq_data`; it contains Dynamic Security clients, roles, and ACLs. If recovery still fails, collect:

```bash
cd /opt/smart-regulator/server
sudo docker compose ps
sudo docker compose logs --tail=100 mosquitto api nginx
sudo stat -c '%a %U:%G %n' config/mosquitto/certs/ca.crt config/mosquitto/certs/server.crt config/mosquitto/certs/server.key
```

Mosquitto must become healthy before the API is created, and the API must become healthy before Nginx can proxy `/api/health`. Nginx 502 errors are therefore downstream symptoms, not a separate Nginx fault in this case. The update script restarts Nginx after API recovery so it does not retain the removed API container's old Docker IP.

### Recovering an MQTT TLS `bad certificate` failure

If Mosquitto starts and accepts a connection on 8883 but logs `tls alert bad certificate`, inspect the connecting client before replacing certificates. A TLS client sends this alert when it rejects the certificate presented by the broker. In the deployment healthcheck, connecting to `localhost` was incorrect because the broker certificate contains the public broker domain and the internal Docker name `mosquitto`, not `localhost`. The healthcheck now connects to `mosquitto` and keeps hostname verification enabled.

Update and redeploy:

```bash
cd /path/to/your/cloned/repository/server
git pull
sudo ./scripts/deploy.sh --update
```

The deploy script now verifies the CA chain, validity period, `DNS:mosquitto` SAN, and certificate/private-key match before starting Docker. It reissues only `server.crt` and `server.key` from the existing CA when an older certificate lacks `DNS:mosquitto`; preserving `ca.crt` means already-flashed firmware keeps the same trust anchor.

For manual diagnosis on the deployed copy:

```bash
cd /opt/smart-regulator/server
openssl verify -CAfile config/mosquitto/certs/ca.crt config/mosquitto/certs/server.crt
openssl x509 -in config/mosquitto/certs/server.crt -noout -checkhost mosquitto
openssl x509 -in config/mosquitto/certs/server.crt -noout -subject -issuer -dates -ext subjectAltName
sudo docker compose config
sudo docker compose ps
sudo docker compose logs --tail=100 mosquitto api nginx
```

Interpret similar failures in this order:

1. `does NOT match certificate`: the client hostname is absent from SAN; use a SAN hostname or reissue the broker certificate with that DNS name.
2. `unable to get local issuer certificate`: the client has the wrong CA, or the broker certificate was signed by a different CA.
3. An expired or not-yet-valid date: renew the server certificate and verify the server clock with `timedatectl`.
4. A certificate/key mismatch: compare the public-key fingerprints and restore the matching pair.
5. A client-certificate error when `require_certificate true` is configured: either provision a client certificate deliberately or disable that requirement; this project uses username/password plus TLS and does not require client certificates.

Do not use `--insecure`, disable hostname verification, regenerate/delete `ca.crt` casually, use `chmod 777`, or delete `mosq_data`. Those actions either hide the fault, break firmware trust, expose secrets, or erase device ACL state.

### Recovering `not authorised` for `regulator-server`

If TLS succeeds but every healthcheck connection ends with `disconnected: not authorised`, the password in `.env` does not match the administrator stored in the persistent Dynamic Security volume. Restarting containers cannot change that stored password. This can happen when an older Mosquitto startup auto-generated `dynamic-security.json` before the managed bootstrap initialized it, or when `.env` was replaced while `mosq_data` was retained.

Run the explicit one-time repair from the updated source checkout:

```bash
cd /path/to/your/cloned/repository/server
git pull
sudo ./scripts/deploy.sh --update --repair-mqtt-auth
```

The repair stops only API/Mosquitto, writes a timestamped backup such as `dynamic-security.json.bak.20260715104500` inside `mosq_data`, atomically initializes `regulator-server` from the preserved `.env`, and marks existing device MQTT credentials for rebuilding. The healthcheck authenticates through the administrator's Dynamic Security control topic. When the API connects, it grants the server only the required telemetry/status receive and command/config send patterns, then reconstructs pending per-device clients and MAC-scoped ACLs from PostgreSQL without changing ownership, pairing data, or device secrets. Use plain `--update` for later deployments; a full `docker compose down` is unnecessary.

Every redeployment path, including an accidental run without `--update`, preserves `.env`, `ca.crt`, `ca.key`, `server.crt`, and `server.key` before replacing `/opt/smart-regulator/server`. It never silently rotates JWT/MQTT secrets or the CA. Losing or rotating the CA would require updating every flashed device, while changing only `.env` and retaining `mosq_data` would make broker authentication fail.

Required production environment values include:

| Variable | Purpose |
|---|---|
| `APP_JWT_SECRET` | JWT signing key |
| `APP_CORS_ORIGINS` | comma-separated browser-origin allowlist; no wildcard in production |
| `APP_WECHAT_APPID` / `APP_WECHAT_SECRET` | WeChat login |
| `APP_MQTT_PASSWORD` | Dynamic Security administrator password |
| `APP_MQTT_CA_FILE` | broker CA path inside API container |
| `APP_DATABASE_URL` | async database URL |
| `DOMAIN` / `CERTBOT_EMAIL` | HTTPS certificate provisioning |

## API overview

| Method | Path | Auth | Purpose |
|---|---|---|---|
| GET | `/api/health` | none | health check |
| POST | `/api/auth/wechat` | none | WeChat code → JWT |
| POST | `/api/devices/register` | device credential | registration and OTP window |
| POST | `/api/devices/bind` | JWT | consume OTP and bind |
| GET | `/api/devices` | JWT | list owned devices |
| GET | `/api/devices/{id}/snapshot` | JWT | latest state |
| GET/PUT | `/api/devices/{id}/config` | JWT | setpoints |
| POST | `/api/devices/{id}/command` | JWT | device command |
| POST | `/api/devices/{id}/unbind` | JWT | delivered unbind |
| GET/POST | `/api/devices/{id}/care/records` | JWT | care records |
| GET | `/api/devices/{id}/care/summary` | JWT | week/month counts |
| GET/PUT | `/api/user/profile` | JWT | user profile |
| GET/POST | `/api/profiles` | optional/JWT | species profiles |

WebSockets:

- `/ws/telemetry/{id}?token=<JWT>`
- `/ws/video/{id}?token=<JWT>`
- `/upload/telemetry/{id}` with Basic device authentication
- `/upload/video/{id}` with Basic device authentication

## Admin Panel

An operations dashboard for platform administrators. Provides user/device management, analytics, system monitoring, and database access through a web interface.

- **Access**: `https://<domain>/admin/login`
- **Default account**: `admin` / password from `APP_ADMIN_INITIAL_PASSWORD` (default: `admin123`)
- **Full operator guide**: [docs/admin-panel.md](docs/admin-panel.md)
- **LLM construction guide**: [admin/.ai/README.md](admin/.ai/README.md)

### Admin API Endpoints

| Method | Path | Required Permission |
|---|---|---|
| GET/POST | `/admin/login` | none |
| POST | `/admin/logout` | any authenticated |
| GET | `/admin/auth/check` | none (used by Nginx) |
| GET | `/admin/dashboard` | `dashboard.view` |
| GET | `/admin/dashboard/data` | `dashboard.view` |
| GET | `/admin/users` | `users.list` |
| GET | `/admin/users/list` | `users.list` |
| GET | `/admin/users/{id}` | `users.view_detail` |
| GET | `/admin/devices` | `devices.list` |
| GET | `/admin/devices/list` | `devices.list` |
| GET | `/admin/devices/{id}` | `devices.list` |
| POST | `/admin/devices/{id}/edit` | `devices.edit` |
| POST | `/admin/devices/{id}/unbind` | `devices.unbind` |
| GET | `/admin/care-records` | `care_records.list` |
| GET | `/admin/care-records/list` | `care_records.list` |
| GET | `/admin/profiles` | `profiles.list` |
| POST | `/admin/profiles/{id}/delete` | `profiles.list` |
| GET | `/admin/system/metrics` | `system.metrics` |
| GET | `/admin/system/metrics/data` | `system.metrics` |
| GET | `/admin/system/logs` | `system.logs` |
| GET | `/admin/system/logs/data` | `system.logs` |
| GET/POST | `/admin/system/admins` | `admin.users.manage` |
| POST | `/admin/system/admins/create` | `admin.users.manage` |
| POST | `/admin/system/admins/{id}/toggle` | `admin.users.manage` |
| POST | `/admin/system/admins/{id}/delete` | `admin.users.manage` |
| GET | `/admin/system/roles` | `admin.roles.manage` |
| POST | `/admin/system/roles/create` | `admin.roles.manage` |
| POST | `/admin/system/roles/{id}/permissions` | `admin.roles.manage` |
| POST | `/admin/system/roles/{id}/delete` | `admin.roles.manage` |

### Adminer

Database management tool at `/adminer`. Requires `system.adminer` permission (only `super_admin` by default). Access is gated by Nginx `auth_request` to the admin panel.

## Known limitations

- Config updates are stored, published, and applied by firmware, but firmware persistence/version ACK is not implemented.
- Dynamic Security provisioning is acknowledged by the broker, but production recovery still depends on preserving the PostgreSQL device secrets and Mosquitto authorization volume.
- Device command delivery has no end-to-end firmware ACK.
- Rate limits are process-local and rely on the single-worker architecture; use shared storage before scaling.
- JWT WebSocket query tokens are omitted from Nginx access logs but should eventually be replaced with short-lived socket tickets.
- HTTPS Nginx/certbot deployment and Dynamic Security must be tested on the actual Ubuntu host; Windows unit tests cannot validate containers.
- Multiple API workers are unsupported until WebSocket/frame state moves to shared storage.

## LLM guide

See [server/.ai/README.md](.ai/README.md) and the root [.ai knowledge base](../.ai/README.md).