jyapayne/warp-pipe
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
warp-pipe/GUI_PLAN.md

22 KiB
Raw Blame History

Warppipe GUI Plan (Qt6 Node-Based Audio Router)

Overview

A Qt6-based node editor GUI for warppipe using the QtNodes (nodeeditor) library. Visualizes PipeWire audio nodes, ports, and links as draggable nodes with connection lines. Supports creating virtual sinks/sources via context menu and displays ephemeral sources with visual fade when inactive.

Milestones

  • Milestone 0 - Qt6 Project Setup

    • Create gui/ subdirectory in warppipe project
    • Add Qt6 + QtNodes to CMakeLists.txt (FetchContent for nodeeditor from github.com/paceholder/nodeeditor)
    • Create warppipe-gui target with Qt6::Widgets and QtNodes dependencies
    • Enable CMAKE_AUTOMOC, CMAKE_AUTORCC, CMAKE_AUTOUIC
    • Create minimal main.cpp with QApplication + QMainWindow
    • Verify GUI launches and shows empty window

  • Milestone 1 - Core Model Integration

    • Create WarpGraphModel : public QtNodes::AbstractGraphModel
      • Implement AbstractGraphModel interface (newNodeId, allNodeIds, nodeData, portData, etc.)
      • Add warppipe::Client* member, connect to PipeWire on construction
      • Map warppipe::NodeInfo to QtNodes NodeId via internal maps (m_nodes, m_pwToNode)
      • Implement node refresh: call Client::ListNodes() and sync graph
    • Create GraphEditorWidget : public QWidget
      • Instantiate WarpGraphModel, QtNodes::BasicGraphicsScene, QtNodes::GraphicsView
      • Lay out view in widget
      • Connect model signals to refresh handlers
    • Synthesize display title from NodeInfo:
      • If application_name is non-empty and differs from name, use application_name as title
      • Otherwise use name field
      • Store synthesized title in nodeData(NodeRole::Caption)
    • Map warppipe ports to QtNodes ports:
      • Input ports (is_input=true) appear on LEFT side of node (QtNodes PortType::In)
      • Output ports (is_input=false) appear on RIGHT side of node (QtNodes PortType::Out)
      • Use port name from PortInfo as port label
    • Verify nodes appear in graph view with correct titles and ports

  • Milestone 2 - Visual Styling and Node Types

    • Define node type classification based on media_class:
      • Sink → "Hardware Sink" (blue-gray base color)
      • Source → "Hardware Source" (blue-gray base color)
      • Virtual sinks created by warppipe → "Virtual Sink" (green base color)
      • Virtual sources created by warppipe → "Virtual Source" (green base color)
      • Application audio streams (ephemeral) → "Application" (brown/orange base color)
    • Implement custom NodeStyle via nodeData(NodeRole::Style):
      • Return QtNodes::NodeStyle::toJson().toVariantMap()
      • Set GradientColor0-3, NormalBoundaryColor, FontColor based on node type
      • Reference potato's nodeStyleVariant() function for color scheme
    • Detect ephemeral (application) nodes:
      • Track node appearance/disappearance via Client poll or registry events
      • Mark node as "inactive" if it disappears (no audio playing)
      • Persist inactive nodes in graph model (do NOT remove from visual graph)
    • Apply "ghost" styling to inactive nodes:
      • Set Opacity = 0.6f (vs 1.0f for active)
      • Darken gradient colors (use .darker(150-180))
      • Fade font color (lighter gray)
      • Keep connections visible with faded style
    • Verify: Application nodes appear vibrant when active, fade when inactive, never disappear

  • Milestone 3 - Link Visualization and Drag-Connect

    • Implement connection mapping:
      • Call Client::ListLinks() to get existing PipeWire links
      • For each Link, find corresponding NodeId and PortIndex for output/input
      • Create QtNodes::ConnectionId from (outNodeId, outPortType, outPortIndex, inNodeId, inPortType, inPortIndex)
      • Store in model's m_connections set
    • Implement addConnection(ConnectionId):
      • Extract output port and input port from ConnectionId
      • Call Client::CreateLink(outputPortId, inputPortId, LinkOptions{})
      • If successful, add connection to m_connections
      • If failed, emit error and do NOT add to graph
    • Implement deleteConnection(ConnectionId):
      • Find corresponding warppipe LinkId from connection
      • Call Client::RemoveLink(linkId)
      • Remove from m_connections
    • Verify: Drag connection from output port to input port creates PipeWire link; delete removes it

  • Milestone 4 - Context Menu and Virtual Node Creation

    • Add context menu to GraphEditorWidget:
      • Right-click on canvas (not on node) shows menu
      • Menu items: "Create Virtual Sink", "Create Virtual Source"
    • Implement "Create Virtual Sink":
      • Prompt user for name (QInputDialog or inline text field)
      • Call Client::CreateVirtualSink(name, VirtualNodeOptions{}) with default options
      • On success, node appears in graph at context menu position
    • Implement "Create Virtual Source":
      • Same as sink but call Client::CreateVirtualSource()
    • Add context menu on nodes:
      • Right-click on virtual node shows "Delete Node" option
      • Call Client::RemoveNode(nodeId) and remove from graph
    • Verify: Can create/delete virtual sinks and sources via right-click

  • Milestone 5 - Layout Persistence and Polish

    • Implement layout save/load:
      • Save node positions to JSON file in ~/.config/warppipe-gui/layout.json
      • Store by stable ID (use NodeInfo.name as stable key)
      • Save on position change (debounced)
      • Load on startup and restore positions
    • Implement auto-arrange:
      • Menu or button to auto-layout nodes (left-to-right: sources → sinks)
      • Use simple grid or layered layout algorithm
    • Add visual polish:
      • Connection lines styled (color, width, curvature)
      • Highlight connections on hover
      • Port connection points visible and responsive
    • Add status bar:
      • Show connection status to PipeWire daemon
      • Show count of nodes, links
    • Verify: Layout persists across sessions, UI feels responsive and polished

  • Milestone 6 - Screenshot Infrastructure (AI-Assisted Debugging)

    • Add CLI flags to main.cpp via QCommandLineParser:
      • --screenshot <path> / -s <path>: Capture window to PNG and exit
      • --screenshot-delay <ms>: Configurable render delay before capture (default 800ms)
      • --debug-screenshot-dir <dir>: Continuous mode — save timestamped screenshot on every graph state change (node add/remove, connection change, ghost toggle)
    • Implement two-tier QPixmap capture (from potato pattern):
      • Primary: window.grab() (renders widget tree to pixmap)
      • Fallback: screen->grabWindow(window.winId()) if .grab() returns null
      • Exit code 3 on capture failure
    • Add F12 hotkey for interactive screenshot:
      • Save to $XDG_PICTURES_DIR/warppipe/warppipe_YYYYMMDD_HHmmss.png
      • Auto-create directory via QDir::mkpath()
    • Implement "render complete" signal:
      • GraphEditorWidget emits graphReady() after initial node sync completes
      • Use signal instead of hardcoded delay for --screenshot when possible
      • Fall back to --screenshot-delay if signal doesn't fire within timeout
    • Support headless rendering for CI/AI:
      • Verify screenshots render correctly without a display server
      • Add --offscreen convenience flag that sets QT_QPA_PLATFORM=offscreen internally via qputenv()
    • Implement debug screenshot naming convention:
      • Format: warppipe_<timestamp>_<event>.png (e.g., warppipe_20260129_143052_node_added.png)
      • In --debug-screenshot-dir mode, capture on: initial load, node add, node remove, node ghost/unghost, connection add, connection remove
    • Verify: warppipe-gui --screenshot /tmp/test.png produces a valid PNG with visible nodes; headless mode works with QT_QPA_PLATFORM=offscreen

  • Milestone 7 - GUI Tests

    • Create tests/gui/ directory and warppipe_gui_tests.cpp test file
    • Add warppipe-gui-tests CMake target linking warppipe, Qt6::Widgets, Qt6::Test, QtNodes, Catch2
    • Model unit tests (no display server needed, pure logic):
      • WarpGraphModel: inject nodes via WARPPIPE_TESTING helpers → verify allNodeIds(), nodeData(Caption), nodeData(Style)
      • WarpGraphModel: inject ports → verify portData(PortCount), portData(Caption) for correct port labels
      • WarpGraphModel: inject links → verify allConnectionIds(), connectionExists()
      • WarpGraphModel: ghost state tracking — mark node ghost → verify Opacity=0.6 in style, mark unghost → verify Opacity=1.0
      • WarpGraphModel: title synthesis — node with application_name="Firefox" → caption="Firefox"; node with empty application_name → caption=name
      • WarpGraphModel: port orientation — is_input=true ports map to PortType::In (left); is_input=false → PortType::Out (right)
      • WarpGraphModel: node removal doesn't crash when connections exist
      • WarpGraphModel: duplicate node ID handling (update vs reject)
    • Connection logic tests:
      • addConnection() with valid ports → succeeds, stored in model
      • addConnection() with mismatched types (output→output) → connectionPossible() returns false
      • deleteConnection() removes from model
      • Ghost connections: connection to ghost node remains in model, isGhostConnection() returns true
    • Screenshot smoke tests (require QT_QPA_PLATFORM=offscreen):
      • Launch warppipe-gui with --screenshot → exit code 0, PNG file exists, file size > 0
      • Launch with WARPPIPE_TESTING injected nodes → screenshot contains non-trivial content (file size > 10KB as heuristic)
      • Launch with --debug-screenshot-dir → directory populated after state changes
    • Integration tests with warppipe test harness:
      • Create Client with WARPPIPE_TESTING → inject nodes/ports/links → construct WarpGraphModel → verify graph state matches injected data
      • Inject node, then remove → verify ghost state in model
      • Inject node, add rule, trigger policy check → verify model reflects auto-linked connections
    • Add CTest integration:
      • Model tests run without display server (always)
      • Screenshot tests gated behind WARPPIPE_GUI_VISUAL_TESTS CMake option (default OFF)
      • ctest --test-dir build runs model tests; ctest --test-dir build -L visual runs screenshot tests

  • Milestone 8 (Optional) - Advanced Features

    • Add routing rule UI (separate panel or dialog)
      • List existing rules from Client::ListRouteRules()
      • Add/remove rules with RuleMatch fields
      • Show which nodes are affected by rules
    • Add volume/mute controls (if warppipe adds port parameters API)
    • Add audio level meters (requires PipeWire param monitoring)
    • Add config save/load UI (call Client::SaveConfig()/LoadConfig())
    • Add presets system (save/restore full node+link+rule configuration)

Architecture

warppipe/
├── CMakeLists.txt               # Add Qt6 + QtNodes, warppipe-gui + test targets
├── gui/
│   ├── main.cpp                 # QApplication entry, CLI flags (--screenshot, --debug-screenshot-dir, --offscreen)
│   ├── WarpGraphModel.h         # QtNodes::AbstractGraphModel implementation
│   ├── WarpGraphModel.cpp       # Model logic, warppipe::Client integration
│   ├── GraphEditorWidget.h      # Main UI widget with scene + view
│   ├── GraphEditorWidget.cpp    # Event handlers, context menus, refresh logic, graphReady() signal
│   ├── NodeStyleHelper.h/cpp    # Node styling functions (colors, ghost mode)
│   └── ScreenshotHelper.h/cpp   # QPixmap capture, debug dir, naming conventions
├── tests/
│   ├── warppipe_tests.cpp       # Existing library tests
│   └── gui/
│       └── warppipe_gui_tests.cpp  # Model unit tests + screenshot smoke tests
├── include/warppipe/
│   └── warppipe.hpp             # No changes needed
└── src/
    └── warppipe.cpp             # No changes needed

Design Notes

Node Title Synthesis

Display title priority: description > application_name > name

  • Hardware/Virtual nodes: Use description (PW_KEY_NODE_DESCRIPTION), e.g., "Speakers", "Headphones"
  • Application nodes: Use application_name if non-empty (e.g., "Firefox", "Spotify")
  • Fallback to name if both are empty

Port Orientation

  • Input ports (is_input=true): LEFT side of node (QtNodes::PortType::In)
  • Output ports (is_input=false): RIGHT side of node (QtNodes::PortType::Out)

Ephemeral Node Handling

Application audio streams are ephemeral — they appear when an app plays audio and can disappear when stopped.

  • Do NOT remove from visual graph when inactive
  • Mark as "ghost" and apply faded styling (opacity 0.6, darker colors)
  • Persist connections visually even when node is inactive
  • Re-activate styling when node reappears (audio resumes)

Tracking strategy:

  1. Poll Client::ListNodes() periodically (e.g., every 500ms)
  2. Compare current list to previous list
  3. Nodes that disappeared → mark as ghost, keep in graph
  4. Nodes that reappeared → restore active styling

Node Type Colors (inspired by potato)

Type Base Color Description
Hardware Sink/Source Blue-gray (72, 94, 118) Physical audio devices
Virtual Sink/Source Green (62, 122, 104) Virtual nodes created by warppipe
Application Brown/Orange (138, 104, 72) Ephemeral app audio streams

Active vs Ghost:

  • Active: Opacity = 1.0f, lighter gradient, bright font
  • Ghost: Opacity = 0.6f, darker gradient (.darker(150-180)), faded font

Connection Creation Flow

  1. User drags from output port to input port in UI
  2. QtNodes calls WarpGraphModel::addConnection(connectionId)
  3. Model extracts port IDs from connectionId
  4. Model calls warppipe::Client::CreateLink(outputPortId, inputPortId, LinkOptions{})
  5. If success: add to m_connections, connection appears
  6. If failure: show error, do NOT add connection to graph

Context Menu Actions

  • On canvas: "Create Virtual Sink", "Create Virtual Source"
  • On virtual node: "Delete Node"
  • Future: "Add Routing Rule", "Set as Default", "Edit Properties"

Layout Persistence

  • Save to ~/.config/warppipe-gui/layout.json (XDG_CONFIG_HOME)
  • Format: 
    {
  "version": 1,
  "nodes": [
    {
      "stable_id": "alsa_output.pci-0000_00_1f.3.analog-stereo",
      "position": {"x": 100.0, "y": 200.0}
    }
  ],
  "view": {
    "scale": 1.0,
    "center": {"x": 0.0, "y": 0.0}
  }
}
  • Use NodeInfo.name as stable_id (unique across sessions)

Qt/CMake Integration

CMakeLists.txt additions

# After existing warppipe library target:

option(BUILD_GUI "Build Qt6 GUI application" ON)

if(BUILD_GUI)
  find_package(Qt6 6.2 REQUIRED COMPONENTS Core Widgets)
  
  set(CMAKE_AUTOMOC ON)
  set(CMAKE_AUTORCC ON)
  set(CMAKE_AUTOUIC ON)
  
  include(FetchContent)
  FetchContent_Declare(
    QtNodes
    GIT_REPOSITORY https://github.com/paceholder/nodeeditor
    GIT_TAG master
  )
  FetchContent_MakeAvailable(QtNodes)
  
  add_executable(warppipe-gui
    gui/main.cpp
    gui/WarpGraphModel.cpp
    gui/GraphEditorWidget.cpp
    gui/NodeStyleHelper.cpp
  )
  
  target_link_libraries(warppipe-gui PRIVATE
    warppipe
    Qt6::Widgets
    QtNodes
  )
  
  install(TARGETS warppipe-gui RUNTIME DESTINATION bin)
endif()

GUI test target

if(BUILD_GUI)
  # Model unit tests (no display needed)
  add_executable(warppipe-gui-tests
    tests/gui/warppipe_gui_tests.cpp
    gui/WarpGraphModel.cpp
    gui/NodeStyleHelper.cpp
  )
  
  target_compile_definitions(warppipe-gui-tests PRIVATE WARPPIPE_TESTING)
  
  target_link_libraries(warppipe-gui-tests PRIVATE
    warppipe
    Qt6::Widgets
    QtNodes
    Catch2::Catch2WithMain
  )
  
  # Screenshot smoke tests (opt-in, need Qt runtime)
  option(WARPPIPE_GUI_VISUAL_TESTS "Enable screenshot-based visual tests" OFF)
  
  if(WARPPIPE_GUI_VISUAL_TESTS)
    add_test(NAME gui_screenshot_smoke
      COMMAND ${CMAKE_COMMAND} -E env QT_QPA_PLATFORM=offscreen
              $<TARGET_FILE:warppipe-gui> --screenshot ${CMAKE_BINARY_DIR}/test_screenshot.png
    )
    set_tests_properties(gui_screenshot_smoke PROPERTIES
      LABELS "visual"
      TIMEOUT 10
    )
  endif()
endif()

Dependencies

  • Qt6 >= 6.2 (Core, Widgets)
  • Qt6::Test (for GUI test target)
  • QtNodes (nodeeditor) — fetched via CMake FetchContent
  • Catch2 v3 — fetched via CMake FetchContent (shared with existing tests)
  • warppipe library (existing)

Key QtNodes API Usage

AbstractGraphModel Interface

class WarpGraphModel : public QtNodes::AbstractGraphModel {
  Q_OBJECT
public:
  // Node management
  QtNodes::NodeId newNodeId() override;
  std::unordered_set<QtNodes::NodeId> allNodeIds() const override;
  bool nodeExists(QtNodes::NodeId) const override;
  QtNodes::NodeId addNode(QString const nodeType) override;
  bool deleteNode(QtNodes::NodeId) override;
  
  // Node data (caption, style, position, widget)
  QVariant nodeData(QtNodes::NodeId, QtNodes::NodeRole) const override;
  bool setNodeData(QtNodes::NodeId, QtNodes::NodeRole, QVariant) override;
  
  // Port data (count, label, type, connection policy)
  QVariant portData(QtNodes::NodeId, QtNodes::PortType, QtNodes::PortIndex, QtNodes::PortRole) const override;
  
  // Connection management
  std::unordered_set<QtNodes::ConnectionId> allConnectionIds(QtNodes::NodeId) const override;
  bool connectionPossible(QtNodes::ConnectionId) const override;
  void addConnection(QtNodes::ConnectionId) override;
  bool deleteConnection(QtNodes::ConnectionId) override;
  
  // Custom methods
  void refreshFromClient();  // Poll warppipe::Client and sync graph
  void markNodeAsGhost(QtNodes::NodeId, bool isGhost);
};

Scene and View Setup

// In GraphEditorWidget constructor:
m_model = new WarpGraphModel(client, this);
m_scene = new QtNodes::BasicGraphicsScene(*m_model, this);
m_view = new QtNodes::GraphicsView(m_scene);

QVBoxLayout *layout = new QVBoxLayout(this);
layout->addWidget(m_view);

// Connect signals
connect(m_scene, &QtNodes::BasicGraphicsScene::connectionCreated,
        this, &GraphEditorWidget::onConnectionCreated);
connect(m_scene, &QtNodes::BasicGraphicsScene::connectionDeleted,
        this, &GraphEditorWidget::onConnectionDeleted);

Testing Strategy

Three test tiers

Tier 1 — Model Unit Tests (no display server) Pure logic tests for WarpGraphModel. Use warppipe's WARPPIPE_TESTING helpers to inject fake nodes/ports/links into a Client, then construct the model and assert state. These tests run everywhere (CI, headless, local).

Coverage targets:

  • Node mapping (PipeWire ID → QtNodes NodeId, caption synthesis, style by type)
  • Port mapping (is_input orientation, port count, labels)
  • Connection mapping (link → ConnectionId, add/delete roundtrip)
  • Ghost state (inactive toggle, opacity values, ghost connections)
  • Edge cases (duplicate IDs, remove with active connections, empty graph)

Tier 2 — Screenshot Smoke Tests (headless, QT_QPA_PLATFORM=offscreen) Launch warppipe-gui with --screenshot in offscreen mode. Verify:

  • Exit code 0 and PNG file produced
  • File size heuristic (>10KB = non-trivial content rendered)
  • Debug screenshot dir populates on state changes

Gated behind WARPPIPE_GUI_VISUAL_TESTS CMake option (default OFF) since they need Qt runtime.

Tier 3 — Manual QA with AI-Assisted Debugging Run warppipe-gui with a live PipeWire daemon. Use --debug-screenshot-dir to capture every state transition. When something looks wrong:

  1. Run warppipe-gui --debug-screenshot-dir /tmp/warppipe-debug/
  2. Reproduce the issue
  3. Hand the screenshot directory to the AI for visual analysis

Checklist:

  • All audio nodes appear with correct titles
  • Ports on correct sides (input=left, output=right)
  • Dragging connections creates PipeWire links
  • Virtual sink/source creation via context menu
  • Ephemeral nodes fade when inactive, connections persist
  • Layout persists across restarts

AI Debugging Workflow

The screenshot infrastructure in Milestone 6 is specifically designed so the AI can debug visual issues:

# Quick single screenshot for AI analysis
QT_QPA_PLATFORM=offscreen warppipe-gui --screenshot /tmp/gui-state.png

# Or use convenience flag
warppipe-gui --offscreen --screenshot /tmp/gui-state.png

# Continuous debug capture (every state change gets a timestamped screenshot)
warppipe-gui --debug-screenshot-dir /tmp/warppipe-debug/

# Then hand to AI:
# "Here are the screenshots from /tmp/warppipe-debug/, the ghost nodes
#  aren't fading correctly — can you see what's wrong?"

The AI can then use its multimodal capabilities to examine the PNGs, compare expected vs actual visual state, and identify styling or layout bugs without needing a live display.

Future Enhancements

  • Routing rule editor: Visual panel to add/edit/remove per-app routing rules
  • Audio level meters: Real-time level monitoring (requires PipeWire param API)
  • Volume/mute controls: Inline controls on nodes (requires warppipe volume API)
  • Presets: Save/load full configurations (nodes + links + rules)
  • Search/filter: Filter nodes by type, name, connection status
  • Minimap: Overview of entire graph in large setups
  • Themes: Light/dark mode, custom color schemes
  • Keyboard shortcuts: Create nodes, delete selection, undo/redo

References