Mastering Real-time Object Detection with Frigate and Home Assistant

The Evolution of Smart Home Security: Beyond Basic Motion

Traditional smart home security cameras often rely on basic pixel change detection for motion alerts. While functional, this approach frequently leads to an overwhelming number of false positives – a tree swaying in the wind, a shadow shifting, or a pet roaming the yard can trigger unnecessary notifications. This 'alert fatigue' can desensitize users to actual threats.

Enter Frigate, an open-source Network Video Recorder (NVR) built with real-time AI object detection. Frigate leverages Google's Coral Edge TPU (or CPU/GPU for less demanding setups) to perform lightning-fast object recognition on video streams, distinguishing between people, cars, animals, and more. When integrated with Home Assistant, Frigate transforms your surveillance system into a truly intelligent and proactive guardian.

What is Frigate?

Frigate is a lightweight and powerful NVR that processes video streams from your IP cameras. Its core strength lies in its ability to perform local, real-time object detection using pre-trained machine learning models. Key features include:

• Real-time AI Detection: Identifies specific objects (e.g., person, car, dog) rather than just motion.
• Local Processing: All detection happens on your hardware, ensuring privacy and speed.
• Event Recording: Records clips only when specified objects are detected, saving storage.
• 24/7 Recording: Optional full-time recording with event overlays.
• MQTT Integration: Publishes detection events and states to MQTT, making integration with Home Assistant seamless.
• Home Assistant Companion: A dedicated Home Assistant integration provides entities for cameras, sensors, and snapshots.

Prerequisites for a Frigate Setup

Before diving into the setup, ensure you have the following:

• Home Assistant Instance: Running and accessible.
• MQTT Broker: Mosquitto is recommended and can be installed as an add-on in Home Assistant OS/Supervised.
• Docker: Frigate runs efficiently as a Docker container.
• IP Cameras: Compatible with RTSP streams.
• Hardware for AI: While Frigate can run on a CPU, for optimal performance and multiple cameras, a Google Coral Edge TPU is highly recommended. Alternatively, a powerful CPU or a compatible GPU (e.g., NVIDIA with CUDA) can be used.

Frigate Setup Steps

1. Install Docker and Docker Compose (if not already present)

Ensure Docker and Docker Compose are installed on your host machine (e.g., Ubuntu server, Raspberry Pi 4).

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

2. Create Docker Compose File

Create a `docker-compose.yml` file for Frigate. Adjust paths, ports, and camera details to match your environment.

version: '3.8'
services:
  frigate:
    container_name: frigate
    privileged: true # Required for USB devices like Coral
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    volumes:
      - /path/to/your/frigate/config:/config
      - /etc/localtime:/etc/localtime:ro
      - /dev/bus/usb:/dev/bus/usb # For Coral TPU
      - /path/to/your/frigate/media:/media/frigate # For recordings
    ports:
      - "5000:5000" # Frigate UI
      - "1935:1935" # RTMP (if needed)
    environment:
      - TZ=America/New_York # Your timezone
    devices:
      - /dev/dri:/dev/dri # For Intel Quick Sync or similar (optional)

3. Create Frigate Configuration (config.yml)

Inside `/path/to/your/frigate/config`, create `config.yml`. This is the heart of your Frigate setup.

mqtt:
  host: 192.168.1.10 # Your Home Assistant / MQTT IP
  user: mqtt_user
  password: mqtt_password

detectors:
  coral:
    type: edgetpu
    device: usb # Or 'pci' if using PCIe Coral

ffmpeg:
  hwaccel_args: -c:v h264_cuvid # Example for Nvidia GPU. Use empty for CPU or appropriate for your hardware.

cameras:
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://user:[email protected]/stream_path # Your camera RTSP URL
          roles:
            - detect
            - record
            - rtmp
    detect:
      enabled: True
      threshold: 0.7 # Confidence threshold for detection
      max_disappeared: 30 # How long object can be gone before marked disappeared
      stationary:
        threshold: 20 # How many frames object must be stationary for
      objects:
        - person
        - car
        - cat
    zones:
      driveway:
        coordinates:
          - 100,0
          - 800,0
          - 800,600
          - 100,600
        objects:
          - person
    record:
      enabled: True
      retain:
        days: 7
    rtmp:
      enabled: True

# Add more cameras as needed

Explanation of Key Configuration Options:

• mqtt: Connects Frigate to your MQTT broker.
• detectors: Configures your AI accelerator (e.g., Coral TPU).
• ffmpeg: Defines hardware acceleration for video decoding if available.
• cameras: Each camera has its own section:
  • inputs: The RTSP URL of your camera stream. You might need to find the correct high-resolution and low-resolution streams for your camera.
  • roles: What Frigate should do with this stream (e.g., `detect`, `record`, `rtmp`).
  • detect: Enables object detection. `threshold` adjusts detection sensitivity. `objects` specifies which objects to detect.
  • zones: Define specific areas of interest. Detections inside these zones can trigger specific actions or refine alerts.
  • record: Configures event-based recording retention.
  • rtmp: Enables an RTMP stream for Home Assistant's `generic` camera integration if you want to use that instead of the Frigate integration.

4. Start Frigate

Navigate to the directory containing your `docker-compose.yml` and run:

docker compose up -d

Frigate should now be running. You can access its web UI at `http://your-frigate-ip:5000`.

Integrating Frigate with Home Assistant

1. Install Frigate Integration via HACS

If you don't have HACS (Home Assistant Community Store), install it first. Then, add the Frigate integration:

1. Open HACS in Home Assistant.
2. Go to 'Integrations' and click '+ Explore & Download Repositories'.
3. Search for 'Frigate'.
4. Click 'Download' and restart Home Assistant.

2. Configure the Frigate Integration

1. After restarting, go to 'Settings' > 'Devices & Services' > 'Add Integration'.
2. Search for 'Frigate'.
3. Enter your Frigate instance's host (IP address or hostname) and the MQTT topic prefix (default is `frigate`).
4. Submit, and Home Assistant will discover all your Frigate cameras and sensors.

3. Device Integration Tips and Best Practices

• Camera Entities: Each camera in Frigate will create several entities in Home Assistant: a camera entity showing the live stream, a snapshot entity for the last detection, and binary sensors for each detected object (e.g., `binary_sensor.front_door_person_detection`).
• Optimizing Camera Streams: Frigate benefits from a high-resolution, low-framerate stream for detection, and a separate low-resolution, high-framerate stream for live viewing (though not strictly necessary). Ensure your camera's RTSP URLs are stable.
• Fine-tuning Detection Zones: Use zones in your `config.yml` to ignore irrelevant areas (e.g., public sidewalk) and focus detection on critical areas (e.g., your porch). This significantly reduces false positives.
• Object Filtering: Specify only the `objects` you care about (e.g., `person`, `car`) to avoid processing unnecessary detections.
• Hardware Acceleration: If using Coral, ensure it's correctly exposed to the Docker container. For GPUs, configure `ffmpeg.hwaccel_args` appropriately for your card (e.g., `h264_cuvid` for Nvidia, `vaapi` for Intel).

Building Smart Automations with Frigate and Home Assistant

The real power of Frigate shines when integrated with Home Assistant automations.

Example 1: Person Detected Notification with Snapshot

Receive a rich notification with a snapshot when a person is detected by your front door camera.

alias: Front Door Person Detection
description: ''
trigger:
  - platform: state
    entity_id: binary_sensor.front_door_person_detection
    to: 'on'
condition: []
action:
  - service: notify.mobile_app_your_phone
    data:
      message: A person was detected at the front door!
      title: Security Alert
      data:
        image: /api/frigate/notifications/{{ states('sensor.front_door_latest_person_id') }}/thumbnail.jpg
        clickAction: /lovelace/security
mode: single

Note: Replace `notify.mobile_app_your_phone` with your notification service and `/lovelace/security` with your desired dashboard view.

Example 2: Turn on Lights When Person Detected at Night

Automate outdoor lights to turn on only when a person is detected at night.

alias: Outdoor Lights On Person Detection at Night
description: ''
trigger:
  - platform: state
    entity_id: binary_sensor.front_door_person_detection
    to: 'on'
condition:
  - condition: sun
    after: sunset
    before: sunrise
action:
  - service: light.turn_on
    target:
      entity_id: light.outdoor_porch_light
  - delay:
      hours: 0
      minutes: 2
      seconds: 0
      milliseconds: 0
  - service: light.turn_off
    target:
      entity_id: light.outdoor_porch_light
mode: single

Managing a Reliable Smart Home Ecosystem with Frigate

• Hardware Reliability: Ensure your Frigate host has sufficient cooling and a stable power supply, especially if running multiple cameras or using a Coral TPU.
• Storage Management: Configure retention settings (`record.retain.days`) in `config.yml` to manage disk space for recordings. Consider using a dedicated drive for media.
• False Positive Reduction: Continuously refine your `zones` and `objects` settings. Experiment with `detect.threshold` to balance sensitivity and accuracy.
• Monitoring: Use Home Assistant's built-in sensors from Frigate (e.g., Frigate's health sensors) to monitor its performance and ensure it's running smoothly.
• Updates: Regularly update Frigate and its Home Assistant integration to benefit from new features, bug fixes, and improved detection models. Always check release notes for breaking changes.

Conclusion

Frigate revolutionizes smart home surveillance by bringing advanced, local AI object detection to your fingertips. Its seamless integration with Home Assistant empowers you to create highly intelligent and responsive automations, moving beyond the limitations of traditional motion detection. By investing a little time in setup and configuration, you can build a robust, privacy-focused security system that truly understands what's happening around your home, delivering unparalleled peace of mind.