# Twitch Queue System

This README explains how to use the Huey-based queue system with your Twitch bot.

## Overview

The queue system allows your Twitch bot to process commands and messages asynchronously. This is useful for:

- Handling high-volume chat without performance issues
- Processing commands that take time to complete
- Distributing processing across multiple processes or machines
- Maintaining history of commands for analytics

## Components

The queue system consists of three main components:

1. **Queue Server**: Defined in `src/queue/server.py`, this establishes the Huey queue with SQLite backend
2. **Consumer**: Processes tasks from the queue (can run in a separate process)
3. **Producer**: Your Twitch bot that sends tasks to the queue

## Setup

1. Make sure you have Huey installed:
   ```bash
   pip install huey
   ```

2. Enable the queue in your bot by using the `--use-queue` flag:
   ```bash
   python main.py --use-queue
   ```

3. Start a consumer to process tasks (in a separate terminal):
   ```bash
   # Option 1: Use the Huey consumer CLI
   python -m huey.bin.huey_consumer src.queue.server.huey

   # Option 2: Use our simple wrapper
   python scripts/run_consumer.py
   
   # Option 3: Use the bot with built-in consumer (not recommended for production)
   python main.py --use-queue --start-consumer
   ```

## How It Works

1. When `--use-queue` is enabled, the bot sends all commands to the queue instead of processing them directly.
2. Each command becomes a task in the SQLite database.
3. The consumer process picks up tasks and executes them in the background.
4. Results can be stored and retrieved later.

## Commands

The queue system adds the following command to the bot:

- `!qstats` - Display statistics about the queue (processed tasks, pending tasks, etc.)

## Scaling

For high-volume applications:

1. Run multiple consumers to process tasks in parallel:
   ```bash
   python -m huey.bin.huey_consumer src.queue.server.huey -w 4  # 4 worker processes
   ```

2. Consider using Redis instead of SQLite for better performance:
   (Requires modifying `src/queue/server.py` to use RedisHuey instead of SqliteHuey)

## Architecture Diagram

```
┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│                 │     │                 │     │                 │
│   Twitch Chat   │────▶│    Twitch Bot   │────▶│  Queue (SQLite) │
│                 │     │    (Producer)   │     │                 │
└─────────────────┘     └─────────────────┘     └────────┬────────┘
                                                          │
                                                          ▼
                                               ┌─────────────────────┐
                                               │                     │
                                               │  Huey Consumer(s)   │
                                               │                     │
                                               └─────────────────────┘
```

## Customization

You can customize the queue system by:

1. Adding more task types in `src/queue/server.py`
2. Creating custom handlers for processed tasks
3. Implementing result handling in the bot