Twitch Queue System

This README explains how to use the Huey-based queue system with your Twitch bot.

Overview

The queue system allows your Twitch bot to process commands and messages asynchronously. This is useful for:

Handling high-volume chat without performance issues
Processing commands that take time to complete
Distributing processing across multiple processes or machines
Maintaining history of commands for analytics

Components

The queue system consists of three main components:

Queue Server: Defined in src/queue/server.py, this establishes the Huey queue with SQLite backend
Consumer: Processes tasks from the queue (can run in a separate process)
Producer: Your Twitch bot that sends tasks to the queue

Setup

Make sure you have Huey installed:
```
pip install huey
```
Enable the queue in your bot by using the --use-queue flag:
```
python main.py --use-queue
```

Start a consumer to process tasks (in a separate terminal):

# Option 1: Use the Huey consumer CLI
python -m huey.bin.huey_consumer src.queue.server.huey

# Option 2: Use our simple wrapper
python scripts/run_consumer.py

# Option 3: Use the bot with built-in consumer (not recommended for production)
python main.py --use-queue --start-consumer

How It Works

When --use-queue is enabled, the bot sends all commands to the queue instead of processing them directly.
Each command becomes a task in the SQLite database.
The consumer process picks up tasks and executes them in the background.
Results can be stored and retrieved later.

Commands

The queue system adds the following command to the bot:

!qstats - Display statistics about the queue (processed tasks, pending tasks, etc.)

Scaling

For high-volume applications:

Run multiple consumers to process tasks in parallel:

python -m huey.bin.huey_consumer src.queue.server.huey -w 4  # 4 worker processes

Consider using Redis instead of SQLite for better performance: (Requires modifying src/queue/server.py to use RedisHuey instead of SqliteHuey)

Architecture Diagram

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│                 │     │                 │     │                 │
│   Twitch Chat   │────▶│    Twitch Bot   │────▶│  Queue (SQLite) │
│                 │     │    (Producer)   │     │                 │
└─────────────────┘     └─────────────────┘     └────────┬────────┘
                                                          │
                                                          ▼
                                               ┌─────────────────────┐
                                               │                     │
                                               │  Huey Consumer(s)   │
                                               │                     │
                                               └─────────────────────┘

Customization

You can customize the queue system by:

Adding more task types in src/queue/server.py
Creating custom handlers for processed tasks
Implementing result handling in the bot

3.5 KiB Raw Permalink Blame History