Mastering Local AI-Powered Surveillance: Integrating Frigate with Home Assistant

Mastering Local AI-Powered Surveillance: Integrating Frigate with Home Assistant

In the evolving landscape of smart home security, traditional motion detection often falls short, plagued by false positives triggered by swaying trees, changing light, or passing animals. While cloud-based solutions offer more advanced analytics, they come with privacy concerns and recurring subscription fees. Enter Frigate: a powerful, open-source network video recorder (NVR) that leverages local AI for real-time object detection, bringing enterprise-grade intelligence to your Home Assistant ecosystem.

Frigate uses Google Coral TPUs (Tensor Processing Units) or OpenVINO for highly efficient, local machine learning inference, identifying specific objects like persons, cars, and even pets in your camera feeds. This precision enables far more reliable automations and significantly reduces irrelevant notifications, transforming your surveillance from reactive to truly intelligent.

Why Frigate? The Power of Local AI

• Accuracy: Drastically reduces false positives by detecting specific objects, not just motion.
• Privacy: All processing is done locally, keeping your video streams and data off the cloud.
• Speed: Real-time detection allows for immediate triggering of automations.
• Integration: Seamlessly integrates with Home Assistant via MQTT for rich event data and media.
• Control: Define detection zones, object types, and retention policies with granular control.

Prerequisites

Before diving into the setup, ensure you have the following:

• A running Home Assistant instance (preferably in Docker or a VM for easier Frigate co-location).
• Docker and Docker Compose installed on a Linux machine (e.g., Ubuntu Server, Debian, or even a powerful Raspberry Pi 4).
• An MQTT broker (Mosquitto is ideal and can run in Docker).
• IP cameras with RTSP streams (most modern IP cameras support this).
• (Optional, but highly recommended) A Google Coral USB Accelerator or m.2 TPU for optimal performance.

Frigate Setup: A Docker Compose Approach

The most straightforward way to run Frigate is using Docker Compose. Create a directory (e.g., /opt/frigate) and inside it, create a docker-compose.yml file and a config.yml file.

1. Docker Compose Configuration (docker-compose.yml)

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: true # Required for Coral TPU access
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config.yml:/config/config.yml:ro
      - /mnt/storage/frigate_media:/media/frigate # Persistent storage for clips/snapshots
      - /dev/bus/usb:/dev/bus/usb # For Coral USB Accelerator
      - type: tmpfs # Optional: 1GB of memory for motion detection, reduce disk I/O
        target: /tmp/cache
        tmpfs: {size: 1024m}
    ports:
      - "5000:5000" # Frigate Web UI
      - "1935:1935" # RTMP (if needed)
    environment:
      - FRIGATE_MQTT_PASSWORD=YOUR_MQTT_PASSWORD
      - FRIGATE_MQTT_USER=YOUR_MQTT_USERNAME
    shm_size: "256mb" # Important for performance, adjust based on number of cameras
  mqtt:
    container_name: mosquitto
    image: eclipse-mosquitto:latest
    restart: unless-stopped
    volumes:
      - ./mosquitto/config:/mosquitto/config
      - ./mosquitto/data:/mosquitto/data
    ports:
      - "1883:1883"

Note: Adjust /mnt/storage/frigate_media to your desired media storage path. If you don't use a Coral TPU, remove the /dev/bus/usb volume mount. For MQTT, set up a basic mosquitto.conf with a user/password or allow anonymous for testing.

2. Frigate Configuration (config.yml)

mqtt:
  host: mqtt
  user: YOUR_MQTT_USERNAME
  password: YOUR_MQTT_PASSWORD

# Optional: If using a Coral TPU
detector:
  # Using Google Coral
  tfl_coral:
    type: edgetpu
    device: usb # or pci for m.2

# Or, if not using Coral (slower CPU inference)
# detector:
#   tfl_cpu:
#     type: cpu

# Global settings for your cameras
ffmpeg:
  hwaccel_args: preset-vaapi # Optional: Hardware acceleration (e.g., Intel Quick Sync)

# Define your cameras
cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://YOUR_CAMERA_IP:554/stream1 # Your camera's RTSP stream
          roles:
            - detect # Stream used for object detection
            - record # Stream used for recordings
        - path: rtsp://YOUR_CAMERA_IP:554/stream2 # Lower resolution substream for detection
          roles: [detect]
    detect:
      enabled: True
      width: 1280 # Resolution for detection stream
      height: 720
      fps: 5 # Frames per second for detection
    record:
      enabled: True
      events:
        pre_capture: 5 # Seconds to record before event
        post_capture: 5 # Seconds to record after event
    # Optional: Define zones for detection (e.g., ignore sidewalk)
    zones:
      driveway:
        coordinates: 
          - 0,400
          - 800,400
          - 800,0
          - 0,0
        objects:
          - person
          - car
    # Objects to detect for this camera
    objects:
      track:
        - person
        - car
        - dog
        - cat
      filters:
        person:
          min_area: 5000
          max_area: 100000
          threshold: 0.7

  backyard:
    ffmpeg:
      inputs:
        - path: rtsp://YOUR_BACKYARD_CAMERA_IP:554/main
          roles: [detect, record]
    detect:
      enabled: True
      width: 1920
      height: 1080
      fps: 5
    record:
      enabled: True
    objects:
      track:
        - person
        - animal
    zones:
      patio:
        coordinates:
          - 100,100
          - 500,100
          - 500,500
          - 100,500
        objects:
          - person

# Frigate NVR configuration
clips:
  enabled: True
  retain:
    default: 10 # Retain 10 clips by default
  # Optional: send to Home Assistant media browser
  # These will show up in HA's media browser under 'Frigate'
  # Required for Clips and Snapshots to show up in the HA Media Browser
  # This is the base URL for your Frigate instance, usually http://[ip_address]:5000
  # Recommended to use IP address or a hostname reachable by HA
  cloud_url: "http://192.168.1.100:5000"

snapshots:
  enabled: True
  retain:
    default: 5 # Retain 5 snapshots by default
  # As above, for HA Media Browser access
  cloud_url: "http://192.168.1.100:5000"

After creating both files, navigate to the directory in your terminal and run docker compose up -d. Give it a few minutes to download images and start up. You can check the Frigate UI at http://your-frigate-ip:5000.

Integrating Frigate with Home Assistant

Once Frigate is running and configured to send events to MQTT, Home Assistant will automatically discover it (assuming your MQTT integration is set up). If not, add the MQTT integration and ensure it's connected to your Mosquitto broker.

Frigate will create several entities for each camera:

• Binary sensors (binary_sensor.front_door_person, binary_sensor.front_door_car, etc.) that turn on when the object is detected.
• Camera entities (camera.front_door, camera.front_door_clips, camera.front_door_snapshots) for viewing live streams, recorded clips, and event snapshots.
• Input numbers/sensors for Frigate statistics (FPS, CPU/TPU usage).

Lovelace Dashboard Integration

You can add Frigate cameras to your Lovelace dashboard using the standard Picture Glance card or Camera card. For a more dynamic view of events, consider using custom cards like Frigate Card from HACS, which provides a comprehensive interface for viewing events, timelines, and clips.

Device Integration Tips

• Camera Streams: Always try to use a low-resolution substream (if available) for detection (e.g., 640x480 or 1280x720) and a higher-resolution main stream for recording. This optimizes performance without sacrificing recording quality. Keep the detection FPS low (typically 5-10 FPS is sufficient).
• Google Coral TPU: This is a game-changer. It offloads the computationally intensive AI inference from your CPU, allowing for higher frame rates and more cameras. Ensure it's correctly passed through to the Docker container.
• Storage: Plan your media storage carefully. Frigate can generate a lot of data. Use a dedicated drive, an NFS/SMB mount, or a large SSD for the /media/frigate volume. Configure retention policies in config.yml to manage disk space.
• Shared Memory (shm_size): Increase the shm_size in your Docker Compose if you experience stability issues or crashes, especially with multiple high-resolution camera streams. 256MB is a good starting point.

Best Practices for a Reliable Smart Home Ecosystem with Frigate

Automation Examples

The real power of Frigate lies in its ability to trigger intelligent automations:

• Smart Lighting: Turn on exterior lights for 5 minutes when binary_sensor.front_door_person is detected after sunset.
• Rich Notifications: Send a notification to your phone with a snapshot when a person is detected in a specific zone, but only if you're not home (using Home Assistant's presence detection). Example payload with a snapshot:

service: notify.mobile_app_your_phone
data:
  message: "Person detected at the front door!"
  data:
    image: "http://YOUR_FRIGATE_IP:5000/api/front_door/latest.jpg?h=400"
    notification_icon: mdi:bell-alert
    tag: frigate_person_detection
    ttl: 0
    priority: high

• Smart Doorbells: Integrate with an external chime or TTS announcement when a person is detected at the front door, offering a more robust alternative to traditional motion-based doorbells.
• Specific Alerts: Only send alerts for cars if they are detected in a specific 'unusual parking' zone, or for pets if they enter a forbidden area (e.g., a garden bed).

Optimizing Performance and Stability

• Stream Optimization: Use the lowest possible resolution and FPS for the detect stream that still provides accurate results. More streams at high resolutions/FPS will overload your CPU/TPU.
• Zone Configuration: Define detection zones to exclude public areas or irrelevant motion sources (e.g., roads, distant trees). This reduces unnecessary processing.
• Object Filtering: Limit the objects Frigate tracks to only those you care about (e.g., only person and car).
• Hardware Monitoring: Keep an eye on Frigate's performance metrics (available in its UI and exposed as Home Assistant sensors). If your CPU or Coral TPU utilization is consistently high, consider optimizing camera settings or upgrading hardware.

Maintenance and Updates

• Regular Updates: Keep Frigate and its Docker image updated to benefit from bug fixes, performance improvements, and new features. Use docker compose pull && docker compose up -d.
• Storage Management: Regularly review your media storage usage. Frigate has built-in retention settings, but manual cleanup may occasionally be needed if disk space becomes critically low.
• Configuration Backup: Always back up your config.yml and docker-compose.yml files.

Conclusion

Integrating Frigate with Home Assistant is a significant leap forward for smart home security. By embracing local AI, you gain unparalleled accuracy in object detection, reduce frustrating false alarms, and maintain complete privacy over your surveillance data. This powerful combination empowers you to create truly intelligent and responsive automations, elevating your smart home's security and convenience to a new level. Embrace the power of local AI and transform your surveillance from a chore into a reliable, proactive guardian of your home.