94 lines
No EOL
3.5 KiB
Markdown
94 lines
No EOL
3.5 KiB
Markdown
# Twitch Queue System
|
|
|
|
This README explains how to use the Huey-based queue system with your Twitch bot.
|
|
|
|
## Overview
|
|
|
|
The queue system allows your Twitch bot to process commands and messages asynchronously. This is useful for:
|
|
|
|
- Handling high-volume chat without performance issues
|
|
- Processing commands that take time to complete
|
|
- Distributing processing across multiple processes or machines
|
|
- Maintaining history of commands for analytics
|
|
|
|
## Components
|
|
|
|
The queue system consists of three main components:
|
|
|
|
1. **Queue Server**: Defined in `src/queue/server.py`, this establishes the Huey queue with SQLite backend
|
|
2. **Consumer**: Processes tasks from the queue (can run in a separate process)
|
|
3. **Producer**: Your Twitch bot that sends tasks to the queue
|
|
|
|
## Setup
|
|
|
|
1. Make sure you have Huey installed:
|
|
```bash
|
|
pip install huey
|
|
```
|
|
|
|
2. Enable the queue in your bot by using the `--use-queue` flag:
|
|
```bash
|
|
python main.py --use-queue
|
|
```
|
|
|
|
3. Start a consumer to process tasks (in a separate terminal):
|
|
```bash
|
|
# Option 1: Use the Huey consumer CLI
|
|
python -m huey.bin.huey_consumer src.queue.server.huey
|
|
|
|
# Option 2: Use our simple wrapper
|
|
python scripts/run_consumer.py
|
|
|
|
# Option 3: Use the bot with built-in consumer (not recommended for production)
|
|
python main.py --use-queue --start-consumer
|
|
```
|
|
|
|
## How It Works
|
|
|
|
1. When `--use-queue` is enabled, the bot sends all commands to the queue instead of processing them directly.
|
|
2. Each command becomes a task in the SQLite database.
|
|
3. The consumer process picks up tasks and executes them in the background.
|
|
4. Results can be stored and retrieved later.
|
|
|
|
## Commands
|
|
|
|
The queue system adds the following command to the bot:
|
|
|
|
- `!qstats` - Display statistics about the queue (processed tasks, pending tasks, etc.)
|
|
|
|
## Scaling
|
|
|
|
For high-volume applications:
|
|
|
|
1. Run multiple consumers to process tasks in parallel:
|
|
```bash
|
|
python -m huey.bin.huey_consumer src.queue.server.huey -w 4 # 4 worker processes
|
|
```
|
|
|
|
2. Consider using Redis instead of SQLite for better performance:
|
|
(Requires modifying `src/queue/server.py` to use RedisHuey instead of SqliteHuey)
|
|
|
|
## Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
│ │ │ │ │ │
|
|
│ Twitch Chat │────▶│ Twitch Bot │────▶│ Queue (SQLite) │
|
|
│ │ │ (Producer) │ │ │
|
|
└─────────────────┘ └─────────────────┘ └────────┬────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────┐
|
|
│ │
|
|
│ Huey Consumer(s) │
|
|
│ │
|
|
└─────────────────────┘
|
|
```
|
|
|
|
## Customization
|
|
|
|
You can customize the queue system by:
|
|
|
|
1. Adding more task types in `src/queue/server.py`
|
|
2. Creating custom handlers for processed tasks
|
|
3. Implementing result handling in the bot |