# Creating Acequia Apps: Common Patterns and Architecture Guide
> **Platform Reference** — describes implemented, production behavior.
This document outlines the common patterns, structures, and best practices that have emerged from developing acequia-integrated applications like Camera, SensorTracker, and DomeMaster.
## Table of Contents
1. [Application Structure](#application-structure)
2. [Acequia Integration Patterns](#acequia-integration-patterns)
3. [Component Architecture](#component-architecture)
4. [Main Application Class Pattern](#main-application-class-pattern)
5. [CSS and Styling Patterns](#css-and-styling-patterns)
6. [Storage Management](#storage-management)
7. [Event System Patterns](#event-system-patterns)
8. [Error Handling and Logging](#error-handling-and-logging)
9. [Utility Functions](#utility-functions)
10. [Template for New Apps](#template-for-new-apps)
11. [Worker Pool — Distributed Job Processing](#worker-pool--distributed-job-processing)
- [Dashboard Component](#dashboard-component)
- [Gotchas and Lessons Learned](#gotchas-and-lessons-learned)
12. [Group Shared State](#group-shared-state)
- [API](#shared-state-api)
- [How It Works](#how-shared-state-works)
- [Patterns](#shared-state-patterns)
13. [Event Bus — Decentralized Pub/Sub](#event-bus--decentralized-pubsub)
- [API](#event-bus-api)
- [Topic Matching](#topic-matching)
- [Retained Events](#retained-events)
- [Patterns](#event-bus-patterns)
## Application Structure
### Recommended Directory Structure
```
/your-app-name/
├── index.html # Single HTML entry point
├── src/
│ ├── components/ # Web components (custom elements)
│ │ ├── app-header.js
│ │ ├── status-display.js
│ │ ├── info-modal.js
│ │ └── app-controls.js
│ ├── modules/ # Core business logic modules
│ │ ├── acequia-integration.js
│ │ ├── storage-manager.js
│ │ └── core-manager.js
│ ├── styles/ # CSS organization
│ │ ├── base.css # CSS variables and base styles
│ │ ├── components.css # Component-specific styles
│ │ └── responsive.css # Mobile/responsive styles
│ ├── utils/ # Utility functions
│ │ └── helpers.js
│ └── main.js # Main application entry point
└── README.md
```
### Entry Point Pattern
**index.html structure:**
```html
Your App Name
```
## Acequia Integration Patterns
### Acequia Framework Files
Before diving into server architecture, it's important to understand where the acequia framework files are located:
- **Acequia Core Library**: `/public/acequia.js` - This is the main acequia framework file that provides WebRTC, group management, peer-to-peer functionality, and **WebDAV client capabilities**. The WebDAV server serves this file from the `/public/` directory and makes it available at the root path (`/acequia.js`).
**Two ways to access acequia:**
1. **HTML Script Tag (Legacy)**: Include in your HTML files as:
```html
```
This makes acequia available on `window.acequia` for all modules.
2. **ES6 Module Import (Recommended)**: Import directly in JavaScript modules:
```javascript
import acequia from '/acequia.js'
```
This provides direct access to the acequia framework and is the preferred method for modern apps.
**WebDAV Client Integration**:
- Acequia.js includes a built-in WebDAV client via `acequia.webdav`
- Provides `createAcequiaClient()` function for authenticated WebDAV operations
- Built on the standard webdav.js library with acequia authentication integration
- Enables file operations directly from acequia applications
**Important Notes**:
- Always use the root path `/acequia.js` - this file is served from `/public/acequia.js` but is accessible at the root of any acequia app
- The acequia.js module is ES6 compatible and makes itself available on `window.acequia` when imported via script tag
- When using ES6 imports, you get direct access to the acequia object without needing to access `window.acequia`
- This file is built from the acequia source project located at `../acequia2/` and copied into the `/public/` directory during the build process
- The acequia module is continuously updated on the dev server (e.g., http://oceanx.acequia.live/acequia.js)
- **Configuration Files**:
- `/public/config.js` - Main configuration file that sets up acequia framework settings
- `/public/config.local.js` - Local overrides for development/deployment-specific settings
The acequia.js framework uses these configuration files to determine where the discovery server and auth server are located relative to the webDAV server.
- **Acequia Service Worker**: `/public/sw.js` - The acequia service worker that runs in the browser to handle group routing and caching. Like acequia.js, this is served from `/public/sw.js` but accessible at the root path `/sw.js` for all acequia apps. The service worker handles:
- **Group Route Handling**: Intercepts and routes HTTP requests to registered acequia group routes
- **Caching Management**: Manages caching for scripts, external resources, and data
- **Client Communication**: Handles message passing between acequia components
- **Request Interception**: Routes requests like `/groups/{group}/{name}/{route}` to appropriate handlers
**Important**: Always reference as `/sw.js` in your apps - the WebDAV server automatically serves this from the `/public/` directory at the root path.
**Note**: This file is also built from the acequia source project located at `../acequia2/` and copied into the `/public/` directory during the build process.
### Server Architecture Understanding
Acequia applications work with two distinct servers that serve different purposes:
#### Discovery Server
- **Access**: Use `acequia.getDiscoveryOrigin()` to get the base URL
- **Purpose**: Handles acequia protocol services and peer discovery
- **Services**:
- `/peers.html?group={group}` - Peer management interface
- `/groups/{group}/{name}/shot` - HTTP shot endpoints
- Group discovery and peer coordination services
- **Subdomain isolation**: Groups are namespaced by subdomain server-side — apps just use bare group names
- **Usage**: For acequia-specific functionality and peer communication
#### WebDAV Server
- **Access**: Available via `window.location` (current page location)
- **Purpose**: Serves application files and static content
- **Services**:
- `/VideoDisplay.html` - Video display application
- `/VideoClient.html` - Video client application
- Your app files, CSS, JavaScript, and static assets
- **Usage**: For serving actual application files and resources
#### Correct URL Generation Examples
```javascript
// CORRECT - Use getDiscoveryOrigin() for discovery/group services
const discoveryURL = acequia.getDiscoveryOrigin();
const peersURL = `${discoveryURL}/peers.html?group=${group}`;
const shotURL = `${discoveryURL}/groups/${group}/${name}/shot`;
// CORRECT - Use window.location.origin for app files and auth
const webdavURL = window.location.origin;
const videoDisplayURL = `${webdavURL}/VideoDisplay.html?group=${group}&peer=${peer}`;
const authURL = `${webdavURL}/auth/...`;
// INCORRECT - Don't use discovery URL for app files or auth!
const wrongURL = `${discoveryURL}/VideoDisplay.html`; // ❌
const wrongAuth = `${discoveryURL}/auth/users`; // ❌
```
### Configuration System
The acequia framework uses a configuration system to determine server locations and settings:
#### Main Configuration (`/public/config.js`)
```javascript
export const config = {
build_number: null,
build_branch: null,
build_time: null,
build_root: '/',
debug: true,
dataVersion: 'v0019.9999',
cacheSource: false,
acequia: {
authURL: '/auth',
discoveryServer: '', // Port number (e.g. 31313) or full URL; empty = same origin
},
alwaysFetch: ['/'],
...localConfig, // Local overrides merged in
}
```
#### Local Configuration (`/public/config.local.js`)
```javascript
export const config = {
location: 'public',
acequia: {
// For multi-port deployments (WebDAV on :3333, discovery on :31313):
discoveryServer: 31313, // Numeric = port-only, preserves subdomain
// OR for full URL override:
// discoveryServer: 'http://localhost:31313',
}
}
```
#### Discovery URL Resolution
Use `acequia.getDiscoveryOrigin()` to get the correct discovery server base URL. It reads `config.acequia.discoveryServer`:
- Number or numeric string → port-only mode (`${protocol}//${hostname}:${port}`, preserves subdomain)
- URL string → full URL override
- Empty / null → `window.location.origin` (same-origin default)
**Important:** Auth calls (`/auth/*`) must always use `window.location.origin`, not the discovery URL. Auth is handled by the WebDAV server. Only `/groups/*` and `/ws` go to the discovery server.
#### Accessing Configuration in Your Apps
```javascript
// Discovery URL — use the helper, not the raw config
const discoveryURL = acequia.getDiscoveryOrigin();
// Auth — always same origin
const authURL = `${window.location.origin}/auth/...`;
// Example usage
const peersURL = `${discoveryURL}/peers.html?group=${group}`;
const shotURL = `${discoveryURL}/groups/${group}/${name}/shot`;
```
#### Configuration Hierarchy
1. **Base config** (`config.js`) provides default settings
2. **Local config** (`config.local.js`) overrides base settings for local development
3. **Runtime access** through `acequia.config` object in your applications
### Acequia Service Worker
The acequia service worker (`/public/sw.js`) is a critical component that runs in the browser and enables acequia's distributed functionality:
#### Key Responsibilities
1. **Group Route Handling**
- Intercepts HTTP requests matching `/groups/{groupName}/{displayName}/{route}` pattern
- Routes them to appropriate registered handlers in your applications
- Enables HTTP-based communication between peers and external systems
2. **Caching Management**
- Manages multiple cache layers: scripts, external resources, source code
- Configurable caching strategies through `config.js` settings
- Version-based cache invalidation using `dataVersion` from config
3. **Request Interception**
- Handles WebDAV requests for file operations
- Provides fallback handling for network requests
- Manages offline/online state transitions
4. **Client Communication**
- Facilitates message passing between acequia components
- Supports broadcast messaging to all connected clients
- Manages service worker lifecycle and updates
#### Service Worker Integration
The service worker is automatically registered by the acequia framework and works seamlessly with your applications. When you register routes using:
```javascript
// In your acequia integration
this.acequiaGroup.registerRoute(`/${this.displayName}/myRoute`, async (data) => {
// Handle request
return {
status: 200,
headers: { contentType: 'application/json' },
body: JSON.stringify({ result: 'success' })
};
});
```
The service worker automatically handles:
- Routing incoming HTTP requests to your handler
- Managing request/response formatting
- Applying proper CORS headers
- Error handling and fallback responses
#### Service Worker Configuration
The service worker respects configuration from `config.js`:
```javascript
export const config = {
debug: true, // Enable service worker debug logging
cacheSource: true, // Cache source code files
cacheExternal: true, // Cache external resources
cacheData: false, // Cache data responses
dataVersion: 'v0009.9999' // Cache version for invalidation
}
```
### Core Integration Structure
Every acequia app should have an `acequia-integration.js` module that handles:
1. **Dynamic Acequia Loading** with fallback handling
2. **WebRTC Group Management** with capabilities
3. **HTTP Route Registration** for external access
4. **Peer Communication** via data channels
5. **Proper Server Usage** for URLs and routing
### Content-Type Aware Request/Response Handling
> Binary response bodies are now automatically encoded by `normalizeResponse()`. See [Binary Data](binary-data.md) for the consolidated guide. The manual patterns below still work but are no longer required.
**CRITICAL**: Acequia apps must handle request/response bodies according to their content type. The service worker normalizes request bodies to strings for local/remote consistency, but response bodies must match their content type:
#### Request Body Handling
All incoming request bodies (from service worker) are normalized to **strings**:
```javascript
// Service worker normalizes all requests to text format
const requestBody = await ev.request.text(); // Always string format
// Client should parse based on content-type
parseRequestBody(data) {
if (typeof data.body === 'string') {
return JSON.parse(data.body); // For JSON content
}
// Legacy fallback for older clients
}
```
#### Response Body Handling - Content-Type Specific
Response bodies must match the `contentType` header:
**✅ JSON Responses** (contentType: 'application/json'):
```javascript
// CORRECT - Return string body for JSON
return {
headers: { contentType: 'application/json' },
status: 200,
body: JSON.stringify(data) // ← String for JSON
};
```
**✅ Image Responses** (contentType: 'image/jpeg'):
```javascript
// CORRECT - Return byte array for binary data
return {
headers: { contentType: 'image/jpeg' },
status: 200,
body: Array.from(new Uint8Array(await blob.arrayBuffer())) // ← Bytes for images
};
```
**❌ Common Mistake**:
```javascript
// WRONG - Don't use byte arrays for JSON
return {
headers: { contentType: 'application/json' },
status: 200,
body: Array.from(new TextEncoder().encode(JSON.stringify(data))) // ❌ Binary for JSON!
};
```
#### Service Worker Processing
The service worker handles responses based on body type:
```javascript
// String bodies → TextEncoder (for JSON/text)
responseBody = typeof response.data.body === 'string'
? new TextEncoder().encode(response.data.body)
: new Blob([new Uint8Array(response.data.body)], { type: contentType }); // Array bodies → Blob (for binary)
```
#### Content-Type Examples
| Content Type | Body Format | Use Case | Example |
|-------------|-------------|----------|---------|
| `application/json` | String | API responses, settings | `JSON.stringify({value: 42})` |
| `text/plain` | String | Simple text | `"Hello World"` |
| `image/jpeg` | Byte Array | Camera shots | `Array.from(new Uint8Array(...))` |
| `image/png` | Byte Array | Screenshots | `Array.from(new Uint8Array(...))` |
| `application/octet-stream` | Byte Array | Binary files | `Array.from(new Uint8Array(...))` |
#### Best Practices
1. **Always set correct contentType** in response headers
2. **Match body format to content type** (string for text/JSON, bytes for binary)
3. **Test cross-browser** to ensure consistency (phone ↔ desktop)
4. **Use helper functions** to ensure correct formatting:
```javascript
// Helper for JSON responses
createJSONResponse(status, data) {
return {
headers: { contentType: 'application/json' },
status,
body: JSON.stringify(data) // String body
};
}
// Helper for binary responses
createBinaryResponse(status, contentType, arrayBuffer) {
return {
headers: { contentType },
status,
body: Array.from(new Uint8Array(arrayBuffer)) // Byte array body
};
}
```
#### ⭐ **NEW: Efficient Binary Data Transfer with Base64** ⭐
For better performance when transferring binary data (images, files), use Base64 encoding instead of byte arrays:
**✅ RECOMMENDED - Base64 Binary Responses** (much more efficient):
```javascript
// Convert binary data to Base64 for efficient transport
async createImageResponse() {
const blob = await captureImage() // Your image capture method
const arrayBuffer = await blob.arrayBuffer()
const uint8Array = new Uint8Array(arrayBuffer)
// Build binary string in chunks for large images
const chunkSize = 8192
let binaryString = ''
for (let i = 0; i < uint8Array.length; i += chunkSize) {
const chunk = uint8Array.slice(i, i + chunkSize)
binaryString += String.fromCharCode(...chunk)
}
const base64 = btoa(binaryString)
return {
headers: {
'Content-Type': blob.type || 'image/jpeg',
'Content-Length': blob.size.toString()
},
status: 200,
body: base64,
bodyEncoding: 'base64' // ← KEY: Signals automatic conversion
}
}
```
**❌ INEFFICIENT - Byte Array Method** (800-1600% memory overhead):
```javascript
// AVOID - Memory intensive and slow
return {
headers: { 'Content-Type': 'image/jpeg' },
status: 200,
body: Array.from(new Uint8Array(await blob.arrayBuffer())) // ❌ Huge overhead!
}
```
**Why Base64 is Better:**
- **Memory Efficiency**: ~33% overhead vs 800-1600% for byte arrays
- **Transfer Speed**: Faster JSON transport through WebSocket/WebRTC
- **Automatic Conversion**: Service worker and discovery server automatically convert Base64 → binary for HTTP responses
- **Universal Support**: Works in both browser (service worker) and external HTTP clients (discovery server)
**Key Requirements:**
1. **Use `bodyEncoding: 'base64'`** - This signals automatic conversion
2. **Set proper `Content-Type`** - Must match the original binary format
3. **Include `Content-Length`** - Should reflect original binary size
4. **Handle large images** - Use chunked string building to avoid stack overflow
This ensures consistent behavior across local (same browser) and remote (cross-browser) requests.
### Settings Route Payload Format
When implementing remote control functionality via HTTP routes (e.g., for peerControls integration), follow these payload format standards to ensure consistent parsing and JSON compatibility:
#### Standardized Payload Format
All settings values should be sent as either:
- **Objects `{}`** for single values: `{value: actualValue}`
- **Arrays `[]`** for multi-value data: `[lng, lat]`, `[r, g, b]`
#### Implementation Pattern
**In the sending client (peerControls):**
```javascript
// Handle payload formatting at the applySetting level
async applySetting(settingKey, value) {
let requestBody
if (Array.isArray(value)) {
// Arrays (like coordinates) are sent directly
requestBody = value
} else if (typeof value === 'object' && value !== null && value.hasOwnProperty('value')) {
// Value is already wrapped in {value: X} format, use as-is
requestBody = value
} else {
// Single values need to be wrapped in {value: X} format
requestBody = { value: value }
}
const response = await fetch(settingUrl, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(requestBody)
})
}
```
**In the receiving app (Camera, SandTable):**
```javascript
// Individual setting route handler
this.group.registerRoute(`/${instanceId}/settings/${settingName}`, async (data) => {
try {
const bodyData = this.parseRequestBody(data, `${settingName}`)
// Handle both wrapped {value: X} format and direct array values
let newValue
if (typeof bodyData === 'object' && !Array.isArray(bodyData) && bodyData.hasOwnProperty('value')) {
// Single value settings come as {value: actualValue}
newValue = bodyData.value
} else if (Array.isArray(bodyData)) {
// Array values like coordinates come directly
newValue = bodyData
} else {
// Direct value (for backward compatibility)
newValue = bodyData
}
// Apply the setting
await this.applySetting(settingName, newValue)
return this.createJSONResponse(200, {
success: true,
[settingName]: newValue,
timestamp: new Date().toISOString()
})
} catch (error) {
return this.createJSONResponse(400, {
error: 'Invalid request body',
message: error.message
})
}
})
```
#### Example Payload Formats
**✅ Boolean Setting:**
```javascript
// peerControls sends
{ "value": true }
// App receives and extracts
const visible = bodyData.value // true
```
**✅ Number Setting:**
```javascript
// peerControls sends
{ "value": 15.5 }
// App receives and extracts
const zoom = bodyData.value // 15.5
```
**✅ String Setting:**
```javascript
// peerControls sends
{ "value": "continuous" }
// App receives and extracts
const focusMode = bodyData.value // "continuous"
```
**✅ Coordinate Array:**
```javascript
// peerControls sends directly
[-122.4194, 37.7749]
// App receives directly
const center = bodyData // [-122.4194, 37.7749]
```
#### Benefits of This Pattern
1. **JSON Parse Compatibility**: Avoids `[Object object]` string conversion issues
2. **Type Safety**: Preserves boolean, number, and string types correctly
3. **Array Efficiency**: Multi-value data (coordinates, colors) sent directly
4. **Consistent Handling**: Single approach works across all setting types
5. **Cross-Browser Reliability**: Ensures proper data transmission between different clients
#### Migration from Legacy Format
If you have existing apps using direct value sending:
```javascript
// OLD - can cause JSON parsing issues
body: JSON.stringify(rawValue) // "true", "15", "[object Object]"
// NEW - standardized format
body: JSON.stringify(Array.isArray(value) ? value : { value })
```
The receiving app's `parseRequestBody` method should handle both formats for backward compatibility during migration.
### WebDAV Client Usage in Acequia Apps
The acequia framework includes a built-in WebDAV client that can be used for file operations within acequia applications:
#### Accessing the WebDAV Client
```javascript
import acequia from '/acequia.js'
// Access WebDAV functionality
const webdavClient = await acequia.webdav.createAcequiaClient();
// The client is authenticated and ready for WebDAV operations
const directoryContents = await webdavClient.getDirectoryContents('/');
```
#### WebDAV Client Features
- **Authenticated Access**: Uses acequia device tokens for secure WebDAV operations
- **Standard Operations**: Supports all standard WebDAV operations (GET, PUT, DELETE, PROPFIND, MKCOL)
- **Built-in Library**: Based on the proven webdav.js library
- **Seamless Integration**: Works with acequia's authentication system
#### Example Usage
```javascript
// Create an authenticated WebDAV client
const client = await acequia.webdav.createAcequiaClient();
// List directory contents
const files = await client.getDirectoryContents('/data');
// Upload a file
await client.putFileContents('/data/myfile.txt', 'Hello World');
// Download file content
const content = await client.getFileContents('/data/myfile.txt');
// Create directory
await client.createDirectory('/data/newfolder');
// Delete file
await client.deleteFile('/data/oldfile.txt');
```
#### Integration with WebDAV Manager
The WebDAV Manager app will initially use the existing WebDAVSync client implementation but can eventually migrate to use the acequia.webdav client for better integration with the acequia ecosystem.
### Basic Integration Template
```javascript
import acequia from '/acequia.js'
export default class AcequiaIntegration {
constructor(options = {}) {
this.groupName = options.groupName || 'default';
this.displayName = options.displayName || 'app';
this.acequia = acequia; // Direct access via ES6 import
this.acequiaGroup = null;
this.isInitialized = false;
this.capabilities = ['stream', 'data'];
// Server URLs
this.discoveryURL = null;
this.webdavURL = `${window.location.protocol}//${window.location.host}`;
}
async initialize() {
try {
// Check if acequia is available (ES6 import should always work)
if (!this.acequia) {
console.log('🌐 Acequia not available, app will work in standalone mode');
return false;
}
// Wait for acequia to be ready
await this.acequia.acequiaReady();
// Setup discovery URL — getDiscoveryOrigin() handles discoveryPort, preserving subdomain
this.discoveryURL = this.acequia.getDiscoveryOrigin();
console.log('🔍 Discovery URL:', this.discoveryURL);
console.log('💾 WebDAV URL:', this.webdavURL);
await this.setupWebRTCGroup();
this.registerRoutes();
this.setupPeerHandlers();
this.isInitialized = true;
console.log('✅ Acequia integration initialized');
return true;
} catch (error) {
console.error('❌ Failed to initialize acequia:', error);
return false;
}
}
async setupWebRTCGroup() {
this.acequiaGroup = await this.acequia.webrtc.joinGroup(
this.groupName,
this.displayName,
this.capabilities
);
}
registerRoutes() {
// Register HTTP endpoints for external access
this.acequiaGroup.registerRoute(`/${this.displayName}/status`, async () => {
return {
status: 200,
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ status: 'active', timestamp: Date.now() })
};
});
this.acequiaGroup.registerRoute(`/${this.displayName}/data`, async () => {
return {
status: 200,
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(this.getCurrentData())
};
});
}
setupPeerHandlers() {
this.acequia.webrtc.onNewPeer((peer) => {
peer.on('dataBig', (data) => {
this.handlePeerRequest(peer, data);
});
});
}
handlePeerRequest(peer, data) {
switch (data.request) {
case 'getData':
peer.emit('dataBig', {
response: 'getData',
data: this.getCurrentData()
});
break;
case 'getStatus':
peer.emit('dataBig', {
response: 'getStatus',
status: 'active'
});
break;
default:
console.warn('Unknown peer request:', data.request);
}
}
getCurrentData() {
// Override in your app to return current app data
return { timestamp: Date.now() };
}
// Server URL utility methods
getDiscoveryURL() {
return this.discoveryURL;
}
getWebDAVURL() {
return this.webdavURL;
}
generatePeersURL() {
return `${this.discoveryURL}/peers.html?group=${encodeURIComponent(this.groupName)}`;
}
generateShotURL() {
return `${this.discoveryURL}/groups/${encodeURIComponent(this.groupName)}/${encodeURIComponent(this.displayName)}/shot`;
}
generateVideoDisplayURL(peerId) {
return `${this.webdavURL}/VideoDisplay.html?group=${encodeURIComponent(this.groupName)}&peer=${encodeURIComponent(peerId)}`;
}
generateVideoClientURL(peerId) {
return `${this.webdavURL}/VideoClient.html?group=${encodeURIComponent(this.groupName)}&peer=${encodeURIComponent(peerId)}`;
}
}
```
## Component Architecture
### Web Components Pattern
Use custom web components for reusable UI elements:
```javascript
export class AppHeader extends HTMLElement {
constructor() {
super();
this.isActive = false;
}
connectedCallback() {
this.render();
this.setupEventListeners();
}
render() {
this.innerHTML = `
📱 Your App
`;
}
setupEventListeners() {
const powerBtn = this.querySelector('#powerBtn');
const infoBtn = this.querySelector('#infoBtn');
powerBtn?.addEventListener('click', () => {
this.dispatchEvent(new CustomEvent('power-toggle', { bubbles: true }));
});
infoBtn?.addEventListener('click', () => {
this.dispatchEvent(new CustomEvent('show-info', { bubbles: true }));
});
}
updatePowerState(isActive) {
this.isActive = isActive;
const powerBtn = this.querySelector('#powerBtn');
powerBtn?.classList.toggle('active', isActive);
}
}
// Register the component
customElements.define('app-header', AppHeader);
```
### Status Display Component
Every app should have a status display system:
```javascript
export class StatusDisplay extends HTMLElement {
connectedCallback() {
this.render();
this.setupEventListeners();
}
render() {
this.innerHTML = '';
}
setupEventListeners() {
document.addEventListener('app-status', (event) => {
this.showStatus(event.detail.message, event.detail.type);
});
}
showStatus(message, type = 'info', duration = 3000) {
const container = this.querySelector('.status-container');
const statusEl = document.createElement('div');
statusEl.className = `status-message ${type}`;
statusEl.textContent = message;
container.appendChild(statusEl);
// Trigger show animation
requestAnimationFrame(() => {
statusEl.classList.add('show');
});
// Auto-hide
setTimeout(() => {
statusEl.classList.remove('show');
setTimeout(() => statusEl.remove(), 300);
}, duration);
}
}
customElements.define('status-display', StatusDisplay);
```
## Main Application Class Pattern
### Standard App Class Structure
```javascript
import AcequiaIntegration from './modules/acequia-integration.js';
import StorageManager from './modules/storage-manager.js';
import { getQueryParam, updateQueryParam } from './utils/helpers.js';
class YourApp {
constructor() {
// Core managers
this.coreManager = null;
this.acequiaIntegration = null;
this.storageManager = null;
// Configuration
this.groupName = null;
this.displayName = null;
// DOM references
this.mainContainer = null;
// App state
this.isInitialized = false;
this.isActive = false;
}
async init() {
console.log('🚀 Initializing app...');
try {
// Wait for DOM ready
if (document.readyState === 'loading') {
await new Promise(resolve =>
document.addEventListener('DOMContentLoaded', resolve)
);
}
// Initialize storage manager
this.storageManager = new StorageManager();
// Get DOM references
this.getDOMReferences();
// Setup configuration
await this.setupGroupConfiguration();
// Initialize acequia integration
await this.initializeAcequia();
// Setup event listeners
this.setupEventListeners();
// Initialize your core systems
await this.initializeCore();
this.isInitialized = true;
console.log('✅ App initialized successfully');
} catch (error) {
console.error('❌ Failed to initialize app:', error);
this.showStatus('Failed to initialize app', 'error');
throw error;
}
}
getDOMReferences() {
this.mainContainer = document.getElementById('mainContainer');
// Validate required elements exist
if (!this.mainContainer) {
throw new Error('Required DOM element not found: mainContainer');
}
}
async setupGroupConfiguration() {
// Group name from URL params, storage, or default
this.groupName =
getQueryParam('group') ||
this.storageManager.getSessionItem('groupName') ||
this.storageManager.getLocalItem('groupName') ||
'default';
// Generate unique display name
this.displayName =
getQueryParam('displayName') ||
this.storageManager.getSessionItem('displayName') ||
`${this.groupName}-${Date.now().toString(36).slice(-6)}`;
// Store configuration
this.storageManager.setSessionItem('groupName', this.groupName);
this.storageManager.setLocalItem('groupName', this.groupName);
this.storageManager.setSessionItem('displayName', this.displayName);
// Update URL
updateQueryParam('group', this.groupName);
updateQueryParam('displayName', this.displayName);
}
async initializeAcequia() {
this.acequiaIntegration = new AcequiaIntegration({
groupName: this.groupName,
displayName: this.displayName
});
await this.acequiaIntegration.initialize();
}
setupEventListeners() {
// Power button
document.addEventListener('power-toggle', () => {
this.isActive ? this.stop() : this.start();
});
// Info modal
document.addEventListener('show-info', () => {
document.querySelector('info-modal')?.show();
});
}
async start() {
console.log('▶️ Starting app...');
this.isActive = true;
// Your app start logic here
document.querySelector('app-header')?.updatePowerState(true);
this.showStatus('App started', 'success');
}
async stop() {
console.log('⏹️ Stopping app...');
this.isActive = false;
// Your app stop logic here
document.querySelector('app-header')?.updatePowerState(false);
this.showStatus('App stopped', 'info');
}
showStatus(message, type = 'info') {
document.dispatchEvent(new CustomEvent('app-status', {
detail: { message, type }
}));
}
// Methods for acequia integration
getAcequiaIntegration() {
return this.acequiaIntegration;
}
getStorageManager() {
return this.storageManager;
}
}
// Bootstrap the application
const app = new YourApp();
window.yourApp = app; // Make available for debugging
app.init().catch(error => {
console.error('Failed to start app:', error);
});
export default app;
```
## CSS and Styling Patterns
### CSS Custom Properties System
**base.css** - Define your design system:
```css
:root {
/* Color System */
--color-bg-primary: #000000;
--color-bg-secondary: #1a1a1a;
--color-bg-panel: rgba(255, 255, 255, 0.05);
--color-text-primary: #ffffff;
--color-text-secondary: rgba(255, 255, 255, 0.7);
--color-accent: #22c55e;
--color-success: #10b981;
--color-warning: #f59e0b;
--color-danger: #ef4444;
--color-info: #3b82f6;
--color-border: rgba(255, 255, 255, 0.1);
/* Spacing System */
--space-xs: 0.25rem;
--space-sm: 0.5rem;
--space-md: 1rem;
--space-lg: 1.5rem;
--space-xl: 2rem;
/* Typography */
--font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
--font-size-xs: 0.75rem;
--font-size-sm: 0.875rem;
--font-size-base: 1rem;
--font-size-lg: 1.125rem;
--font-size-xl: 1.25rem;
/* Transitions */
--transition-fast: 0.15s ease;
--transition-normal: 0.3s ease;
--transition-slow: 0.5s ease;
/* Z-index Scale */
--z-dropdown: 40;
--z-header: 50;
--z-modal: 100;
/* Touch Targets */
--touch-target-min: 44px;
/* Border Radius */
--border-radius: 0.375rem;
--border-radius-lg: 0.5rem;
/* Shadows */
--shadow-sm: 0 1px 2px 0 rgba(0, 0, 0, 0.05);
--shadow-md: 0 4px 6px -1px rgba(0, 0, 0, 0.1);
}
/* Base Styles */
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 0;
font-family: var(--font-family);
background: var(--color-bg-primary);
color: var(--color-text-primary);
line-height: 1.5;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
/* Button System */
.btn {
display: inline-flex;
align-items: center;
justify-content: center;
padding: var(--space-sm) var(--space-md);
border: 1px solid var(--color-border);
border-radius: var(--border-radius);
background: var(--color-bg-panel);
color: var(--color-text-primary);
font-family: inherit;
font-size: var(--font-size-sm);
font-weight: 500;
text-decoration: none;
cursor: pointer;
transition: all var(--transition-fast);
min-height: var(--touch-target-min);
user-select: none;
}
.btn:hover {
background: var(--color-accent);
border-color: var(--color-accent);
transform: translateY(-1px);
}
.btn--primary {
background: var(--color-accent);
border-color: var(--color-accent);
color: white;
}
.btn--secondary {
background: var(--color-bg-secondary);
}
```
### Responsive Design Pattern
**responsive.css** - Mobile-first approach:
```css
/* Base mobile styles */
@media screen and (max-width: 767px) {
.main-container {
padding: var(--space-md);
}
}
/* Tablet styles */
@media screen and (min-width: 768px) and (max-width: 1023px) {
.main-container {
display: grid;
grid-template-columns: 1fr 1fr;
gap: var(--space-lg);
padding: var(--space-lg);
}
}
/* Desktop styles */
@media screen and (min-width: 1024px) {
.main-container {
max-width: 1200px;
margin: 0 auto;
padding: var(--space-xl);
}
}
/* Landscape orientation */
@media screen and (orientation: landscape) and (max-width: 1023px) {
.main-container {
display: grid !important;
grid-template-columns: 1fr 1fr !important;
}
}
```
## Storage Management
### StorageManager Pattern
```javascript
export default class StorageManager {
constructor() {
this.isLocalStorageAvailable = this.checkStorageAvailability('localStorage');
this.isSessionStorageAvailable = this.checkStorageAvailability('sessionStorage');
}
checkStorageAvailability(storageType) {
try {
const storage = window[storageType];
const testKey = '__storage_test__';
storage.setItem(testKey, 'test');
storage.removeItem(testKey);
return true;
} catch (e) {
return false;
}
}
// Local Storage Methods
setLocalItem(key, value) {
if (!this.isLocalStorageAvailable) return false;
try {
localStorage.setItem(key, JSON.stringify(value));
return true;
} catch (e) {
console.warn('Failed to set localStorage item:', e);
return false;
}
}
getLocalItem(key, defaultValue = null) {
if (!this.isLocalStorageAvailable) return defaultValue;
try {
const item = localStorage.getItem(key);
return item !== null ? JSON.parse(item) : defaultValue;
} catch (e) {
console.warn('Failed to get localStorage item:', e);
return defaultValue;
}
}
// Session Storage Methods
setSessionItem(key, value) {
if (!this.isSessionStorageAvailable) return false;
try {
sessionStorage.setItem(key, JSON.stringify(value));
return true;
} catch (e) {
console.warn('Failed to set sessionStorage item:', e);
return false;
}
}
getSessionItem(key, defaultValue = null) {
if (!this.isSessionStorageAvailable) return defaultValue;
try {
const item = sessionStorage.getItem(key);
return item !== null ? JSON.parse(item) : defaultValue;
} catch (e) {
console.warn('Failed to get sessionStorage item:', e);
return defaultValue;
}
}
}
```
## Event System Patterns
### Custom Event Communication
Use custom events for component communication:
```javascript
// Dispatching events
document.dispatchEvent(new CustomEvent('app-status', {
detail: { message: 'Status update', type: 'success' }
}));
// Listening for events
document.addEventListener('app-status', (event) => {
console.log(event.detail.message);
});
// Common event types to use across apps:
// - 'app-status' - Status messages
// - 'power-toggle' - Start/stop functionality
// - 'show-info' - Show info modal
// - 'data-update' - Data changes
// - 'connection-change' - Network/acequia status
```
## Error Handling and Logging
### Consistent Logging Pattern
Use emoji-based console logging for easy recognition:
```javascript
// Success states
console.log('✅ Operation completed successfully');
// Errors
console.error('❌ Operation failed:', error);
// Warnings
console.warn('⚠️ Warning message');
// App-specific operations
console.log('📱 App-specific action');
// Network/acequia operations
console.log('🌐 Network operation');
console.log('📡 WebRTC operation');
// Data operations
console.log('💾 Data saved');
console.log('📊 Data updated');
```
### Error Handling Pattern
```javascript
async function riskyOperation() {
try {
const result = await someAsyncOperation();
console.log('✅ Operation successful');
this.showStatus('Operation completed', 'success');
return result;
} catch (error) {
console.error('❌ Operation failed:', error);
this.showStatus('Operation failed: ' + error.message, 'error');
// Re-throw if caller needs to handle
throw error;
}
}
```
## Utility Functions
### Common Helper Functions
**utils/helpers.js**:
```javascript
// URL Parameter Management
export function getQueryParam(param) {
const urlParams = new URLSearchParams(window.location.search);
return urlParams.get(param);
}
export function updateQueryParam(key, value) {
const url = new URL(window.location);
if (value && value !== 'default') {
url.searchParams.set(key, value);
} else {
url.searchParams.delete(key);
}
window.history.replaceState({}, '', url);
}
// Number Formatting
export function formatNumber(value, decimals = 2) {
if (value === null || value === undefined) return '--';
return typeof value === 'number' ? value.toFixed(decimals) : String(value);
}
// Event Throttling
export function debounce(func, wait) {
let timeout;
return function executedFunction(...args) {
const later = () => {
clearTimeout(timeout);
func(...args);
};
clearTimeout(timeout);
timeout = setTimeout(later, wait);
};
}
// DOM Ready Promise
export function domReady() {
return new Promise(resolve => {
if (document.readyState === 'loading') {
document.addEventListener('DOMContentLoaded', resolve);
} else {
resolve();
}
});
}
// Generate Unique ID
export function generateId(prefix = 'id') {
return `${prefix}-${Date.now().toString(36)}-${Math.random().toString(36).slice(2)}`;
}
```
## Template for New Apps
### Quick Start Template
1. **Create directory structure** as outlined above
2. **Copy these template files** and customize:
**src/main.js** - Use the Main Application Class Pattern
**src/modules/acequia-integration.js** - Use the Acequia Integration template
**src/modules/storage-manager.js** - Copy the StorageManager class
**src/components/app-header.js** - Adapt the header component
**src/components/status-display.js** - Copy the status display component
**src/styles/base.css** - Use the CSS custom properties system
3. **Customize for your app**:
- Update app name and branding
- Add your specific functionality
- Register your custom routes
- Implement your data handling
- Add app-specific components
### Essential Features Checklist
- [ ] **URL Parameter Management** - Group and display name configuration
- [ ] **Storage Persistence** - Configuration and state storage
- [ ] **Status Display System** - User feedback
- [ ] **Acequia Integration** - WebRTC and HTTP routes
- [ ] **Error Handling** - Graceful degradation
- [ ] **Mobile Responsive** - Touch-friendly interface
- [ ] **Power Button** - Start/stop functionality
- [ ] **Info Modal** - Help and configuration
- [ ] **Event Communication** - Custom event system
- [ ] **Consistent Logging** - Emoji-based console output
### Naming Conventions
- **Group Names**: Use URL params with storage fallback pattern
- **Display Names**: Generate unique identifiers per instance
- **Route Patterns**: `/${displayName}/endpoint`
- **CSS Classes**: Use BEM methodology with consistent naming
- **Component Names**: Use kebab-case for custom elements
- **Event Names**: Use kebab-case with descriptive names
## Route Registration System
### How registerRoute Works in Acequia
The acequia framework uses a two-layer route registration system that handles both local (same browser) and remote (cross-browser/HTTP) requests:
#### 1. Group Registration
When you call `group.registerRoute(pattern, handler)`, the route is registered with:
- **Service Worker**: For handling local requests within the same browser
- **Discovery Server**: For handling remote HTTP requests from other browsers/devices
#### 2. Request Flow
```
External HTTP Request → Discovery Server → WebSocket → Group Handler → Response
Local Browser Request → Service Worker → Group Handler → Response
```
#### 3. Route Pattern Matching
- **Wildcard patterns**: `/*path` matches all routes under the group
- **Specific patterns**: `/status`, `/data`, `/settings` match exact paths
- **Parameter patterns**: `/{id}/details` for dynamic routing
### Group Lifecycle and Ready State
The Group class connects to the discovery server via WebSocket. The connection lifecycle is:
1. WebSocket opens (internal)
2. Server sends `welcome` message
3. Group calls `register()` to announce deviceInfo
4. `group.ready` Promise resolves
**Always use `await group.ready`** before interacting with the group:
```javascript
import acequia from '/acequia.js'
await acequia.acequiaReady()
const group = new acequia.groups.Group('myGroup', {
capabilities: ['stream', 'data']
})
// Wait for the group to be fully connected
await group.ready
// Now register routes, send messages, etc.
group.registerRoute('/*path', async (data) => {
return {
status: 200,
headers: { contentType: 'application/json' },
body: JSON.stringify({ path: data.path, timestamp: Date.now() })
}
})
```
**There is no `'open'` event on Group.** The `onOpen` config option was removed — it never fired reliably. If you see legacy code using `group.on('open', ...)`, replace it with `await group.ready`.
### Testing Route Registration
To verify your routes are properly registered:
1. **Check browser console**: Look for route registration messages
2. **Test HTTP access**: Try accessing `${discoveryURL}/groups/${groupName}/${displayName}/yourRoute`
3. **Use browser dev tools**: Monitor network requests to see if they're being intercepted
4. **Add logging**: Include console.log in your route handlers to confirm they're being called
## Best Practices
1. **Start with the template** - Don't reinvent the wheel
2. **Follow the patterns** - Consistency across apps is valuable
3. **Test acequia integration** - Ensure graceful fallback when acequia unavailable
4. **Design mobile-first** - Most usage will be on mobile devices
5. **Use the status system** - Keep users informed of app state
6. **Handle errors gracefully** - Don't let the app crash on errors
7. **Log meaningfully** - Use the emoji logging system for easy debugging
8. **Document your routes** - Make HTTP endpoints discoverable
9. **Version your data** - Include version info in stored data
10. **Test across devices** - Ensure compatibility across different platforms
11. **Use correct servers** - Discovery server for acequia services, WebDAV server for app files
12. **Generate URLs properly** - Use the utility methods to avoid server confusion
13. **⭐ Use `await group.ready`** for one-time setup — but use `group.on('welcome', ...)` for connection status UI, since `ready` is a one-shot promise that won't re-fire after reconnects
## Common Issues and Troubleshooting
### Routes Not Working (Most Common Issue)
**Symptoms**:
- Routes like `/*path` don't respond to HTTP requests
- No console output from route handlers
- 404 errors when accessing group endpoints
**Cause**: Registering routes before the group is connected, or using a non-existent `'open'` event
**Solution**:
```javascript
await group.ready
group.registerRoute('/*path', handler)
```
### Route Pattern Issues
**Problem**: Specific routes not matching expected patterns
**Solution**:
- Use exact matches: `/status` for specific endpoints
- Use wildcards: `/*path` to catch all routes
- Test patterns with simple handlers first
- Check console for registration confirmations
### Cross-Browser Route Access
**Problem**: Routes work locally but not from other devices
**Solution**:
- Verify discovery server is running and accessible
- Check firewall settings for port 31313
- Ensure group names match exactly across devices
- Test direct HTTP access to discovery server endpoints
By following these patterns and using the provided templates, you can quickly create new acequia-integrated applications that are consistent, reliable, and maintainable.
## Development Server Information
### Running the Development Environment
To run acequia applications in development, you need to start both server components:
#### WebDAV Server
- **Location**: `server.mjs` (in the localWebDAV directory)
- **Purpose**: Serves application files, static content, and provides the main web server functionality
- **Start Command**: `node server.mjs`
- **Default Port**: Usually 3333 (check the server configuration)
#### Discovery Server
- **Location**: `../localDiscovery/server.mjs` (relative to localWebDAV directory)
- **Purpose**: Handles acequia protocol services, peer discovery, and WebRTC coordination
- **Start Command**: `node ../localDiscovery/server.mjs`
- **Default Port**: Usually 31313 (check the server configuration)
#### Starting Both Servers
To run your acequia applications in development:
1. **Start the Discovery Server** (from the localWebDAV directory):
```bash
node ../localDiscovery/server.mjs
```
2. **Start the WebDAV Server** (from the localWebDAV directory):
```bash
node server.mjs
```
3. **Access your application**: Navigate to `http://localhost:3333/acequia/localhost/your-app-name/`
## Production Environment (oceanx.acequia.live)
For testing and production use, the system is deployed at `https://oceanx.acequia.live` with a different URL structure:
### URL Structure Differences
**Development (localhost):**
- WebDAV Server: `http://localhost:3333`
- Discovery Server: `http://localhost:31313`
- App URLs: `http://localhost:3333/acequia/localhost/SandTable/`
**Production (oceanx.acequia.live):**
- WebDAV Server: `https://oceanx.acequia.live` (port 443, standard HTTPS)
- Discovery Server: `https://oceanx.acequia.live:31313`
- App URLs: `https://oceanx.acequia.live/SandTable/` ⭐ **Direct root mapping**
### Key Differences
1. **Root Directory Mapping**: The `/acequia/localhost/` directory is mapped directly to the root domain
- `acequia/localhost/SandTable/` → `https://oceanx.acequia.live/SandTable/`
- `acequia/localhost/Camera/` → `https://oceanx.acequia.live/Camera/`
- `acequia/localhost/peerControls/` → `https://oceanx.acequia.live/peerControls/`
2. **SSL/TLS Encryption**: Production uses Let's Encrypt certificates for HTTPS
3. **Multi-Device Access**: The oceanx.acequia.live domain allows access from multiple devices/computers
4. **Port Configuration**: WebDAV serves on standard HTTPS port 443, Discovery on 31313
### Example Production URLs
```bash
# SandTable app with remote control group
https://oceanx.acequia.live/SandTable/?group=remote-test
# Camera app in development group
https://oceanx.acequia.live/Camera/?group=dev-group
# PeerControls for remote controlling other apps
https://oceanx.acequia.live/peerControls/?group=remote-test
# Test page for tile constraints
https://oceanx.acequia.live/test-tile-constraints.html
```
### Configuration Considerations
When developing for both environments, the acequia framework automatically adapts:
```javascript
// The framework automatically determines the correct URLs
const discoveryURL = acequia.getDiscoveryOrigin(); // Auto-configured, subdomain-preserving
const webdavURL = window.location.origin; // Always correct for auth + app files
// Your app works in both environments without code changes
```
### Complete Project Structure
```
observer/
├── acequia2/ # Acequia framework source project
│ ├── acequia.js # Source acequia framework
│ ├── sw.js # Source service worker
│ ├── package.json # Build dependencies and scripts
│ ├── rollup.config.js # Build configuration
│ └── ... # Other source files
├── localWebDAV/
│ ├── server.mjs # Main WebDAV server
│ ├── public/
│ │ ├── acequia.js # Built acequia framework (copied from ../acequia2/)
│ │ ├── sw.js # Built service worker (copied from ../acequia2/)
│ │ ├── config.js # Main acequia configuration
│ │ └── config.local.js # Local configuration overrides
│ ├── acequia/localhost/ # Your acequia applications
│ │ ├── Camera/
│ │ ├── peerControls/
│ │ └── your-app/
│ └── ...
└── localDiscovery/
└── server.mjs # Acequia discovery server
```
### Acequia Framework Build Process
The acequia framework files (`acequia.js` and `sw.js`) are built from a separate source project:
#### Source Project Location
- **Source Directory**: `../acequia2/` (relative to localWebDAV)
- **Build Output**: Files are copied to `/public/` directory
#### Building Acequia Framework
To rebuild the acequia framework files:
1. **Navigate to acequia2 directory**:
```bash
cd ../acequia2/
```
2. **Install dependencies** (if not already installed):
```bash
yarn install
```
3. **Build the framework**:
```bash
yarn build
```
This command runs the complete build process:
- `yarn build:msgpack` - Builds msgpack dependencies
- `yarn build:simplepeer` - Builds WebRTC peer dependencies
- Copies webdav.js from node_modules
- Runs rollup build process to create final `acequia.js`
4. **Copy built files** to the public directory:
```bash
# The build process should automatically place files in the correct location
# Or manually copy if needed:
cp acequia.js ../localWebDAV/public/
cp sw.js ../localWebDAV/public/
```
### Development Tips
- **Both servers must be running** for full acequia functionality
- **Check console output** from both servers for debugging information
- **Use different terminal windows/tabs** to run each server
- **Discovery server handles peer coordination** while WebDAV server serves your app files
- **Application URLs** typically follow the pattern: `http://localhost:3333/acequia/localhost/app-name/`
- **Configuration system**: The acequia.js framework automatically loads `/public/config.js` which determines discovery and auth server locations
- **Local overrides**: Use `/public/config.local.js` to override settings for local development without modifying the main config
- **Debug mode**: Set `debug: true` in config.js to enable additional console logging from acequia framework
- **Service Worker**: The acequia service worker (`/public/sw.js`) automatically handles route registration and request routing - no manual registration needed
- **Cache Management**: If you experience caching issues during development, the service worker can be cleared via browser dev tools or by updating the `dataVersion` in config
- **Framework Updates**: The acequia framework files (`acequia.js`, `sw.js`) are built from the separate `../acequia2/` project - only rebuild if you need framework updates or fixes
- **Build Dependencies**: If working on acequia framework itself, use `yarn install` and `yarn build` in the acequia2 directory, then copy the built files to `/public/`
## Worker Pool — Distributed Job Processing
The `WorkerPool` class adds distributed job processing on top of Acequia groups. Peers declare capabilities, and submitters dispatch jobs directly to capable workers over WebRTC — no coordinator, no leader election.
**Design doc:** [worker-pool-implementation.md](../projects/completed/worker-pool-implementation.md)
**Implementation:** `acequia2/acequia/groups.js` (exported alongside `Group`)
### When to Use WorkerPool vs Routes
| Use WorkerPool when... | Use Routes when... |
|------------------------|-------------------|
| Multiple peers should handle the same work type | One canonical handler per URL pattern |
| You need load distribution across workers | You need service endpoint semantics |
| Jobs need retry, timeout, progress tracking | Simple request/response is enough |
| Workers may come and go dynamically | A stable handler is expected |
### Minimal Worker (5 lines)
```javascript
const pool = new acequia.groups.WorkerPool('my-pool', {
capabilities: ['thumbnail'],
concurrency: 2,
handler: async (job) => await makeThumbnail(job.payload)
})
```
### Minimal Submitter (5 lines)
```javascript
const pool = new acequia.groups.WorkerPool('my-pool')
const ticket = pool.submit({ type: 'thumbnail', payload: { path: '/photo.jpg' } })
const result = await ticket.result
```
### Combined Worker + Submitter
A single peer can be both. The `handler` makes it a worker; `submit()` makes it a submitter:
```javascript
const pool = new acequia.groups.WorkerPool('media-pool', {
capabilities: ['thumbnail', 'ocr'],
concurrency: 3,
handler: async (job, ctx) => {
ctx.progress({ status: 'starting' })
if (job.type === 'thumbnail') return await makeThumbnail(job.payload)
if (job.type === 'ocr') return await runOCR(job.payload)
}
})
// Submit a job (may be handled by this peer or another)
const ticket = pool.submit({
type: 'ocr',
payload: { path: '/document.pdf' },
timeout: 60000,
retries: 2
})
ticket.onProgress(data => console.log(data.status))
ticket.onComplete(result => console.log('Done:', result))
ticket.onFailed(err => console.error('Failed:', err))
```
### Route-Backed Workers
A registered route (single entry point, SW-interceptable) can delegate to a worker pool (distributed execution). This gives callers `fetch()` ergonomics while work is distributed across peers:
```javascript
const pool = new acequia.groups.WorkerPool('media-pool', {
capabilities: ['thumbnail'],
concurrency: 2,
handler: async (job) => makeThumbnail(job.payload)
})
// Register a route that dispatches to the pool
pool.group.registerRoute('/_process', async (request) => {
const body = await parseBody(request.body)
const ticket = pool.submit({ type: body.jobType, payload: body.payload })
try {
const result = await ticket.result
return { status: 200, body: JSON.stringify({ result }) }
} catch (err) {
return { status: 500, body: JSON.stringify({ error: err.message }) }
}
})
// Now any page can call: fetch('/groups/media-pool/_process', { ... })
```
### Dispatch Strategies
```javascript
new WorkerPool('pool', { dispatch: 'least-loaded' })
```
| Strategy | Behavior |
|----------|----------|
| `round-robin` (default) | Rotate through capable peers in sorted order |
| `least-loaded` | Pick peer with lowest `activeJobs/concurrency` ratio |
| `random` | Random capable peer |
| `affinity` | Prefer the peer that last handled this job type (cache locality) |
| `(workers, job) => id` | Custom function returning a worker instanceId |
### Observability
Workers broadcast pool stats in `deviceInfo` automatically. For real-time visibility, use the observer pattern:
```javascript
// Observer — joins the group passively, receives job lifecycle events
const observer = new acequia.groups.WorkerPool('media-pool', {
capabilities: ['pool-observer']
})
observer.onJobEvent(event => {
// event.event: 'job:started', 'job:completed', 'job:failed', 'job:rejected', etc.
// event.jobType, event.worker, event.durationMs, event.timestamp
console.log(`${event.event}: ${event.jobType} on ${event.worker}`)
})
```
The `` web component renders a live dashboard:
```javascript
import '/components/worker-pool-dashboard/worker-pool-dashboard.js'
const dashboard = document.createElement('worker-pool-dashboard')
dashboard.pool = myWorkerPool
document.body.appendChild(dashboard)
```
### API Reference
```javascript
// --- Creating a pool ---
const pool = new acequia.groups.WorkerPool(groupId, {
capabilities: string[], // job types this worker handles
concurrency: number, // max simultaneous jobs (default 1)
handler: async (job, ctx) => any, // worker handler
dispatch: string | Function, // dispatch strategy (default 'round-robin')
displayName: string, // peer display name
})
// --- Submitting jobs ---
const ticket = pool.submit({
type: string, // must match a worker capability
payload: any, // serializable data
timeout: number, // ms (default 30000)
retries: number, // retry count (default 0)
retryOnDisconnect: boolean, // resubmit if worker drops (default true)
})
// --- JobTicket ---
ticket.id // string (crypto.randomUUID)
ticket.status // 'pending' | 'offered' | 'completed' | 'failed' | 'timed_out' | 'lost'
ticket.result // Promise — resolves on complete, rejects on fail
ticket.worker // assigned worker instanceId
ticket.cancel() // cancel if not yet completed
ticket.onComplete(fn)
ticket.onFailed(fn)
ticket.onProgress(fn)
// --- Pool state ---
pool.workers // Map
pool.activeJobs // this worker's active count
pool.completedJobs // this worker's completed count
pool.group // underlying Group instance
pool.getTicket(id) // lookup ticket by ID
// --- Events ---
pool.onWorkerJoin(fn)
pool.onWorkerLeave(fn)
pool.onJobEvent(fn) // observer: receive lifecycle events
// --- Lifecycle ---
await pool.close() // stop accepting, finish active, cleanup
```
### Dashboard Component
The `` web component shows live worker stats, load, and event history. Two ways to use it:
**Attach to an existing pool** (shares the same Group connection):
```html
```
**Standalone observer** (auto-joins the group with `pool-observer` capability):
```html
```
The dashboard aggregates two data channels:
- **Worker stats** (passive) — `completedJobs`, `failedJobs`, `concurrency`, `avgDurationMs` from each worker's `deviceInfo`, broadcast via the discovery server's `peersList`
- **Job events** (active) — `job:started`, `job:completed`, `job:failed`, etc., sent over WebRTC data channels to peers with the `pool-observer` capability
### Gotchas and Lessons Learned
These are hard-won insights from building and debugging the worker pool over real WebRTC connections.
#### Group Lifecycle
**There is no `'open'` event.** The Group class resolves `group.ready` on the `welcome` message from the discovery server. The `onOpen` config option was removed. Use `await group.ready` for one-time initialization (route registration, etc.).
**`group.ready` is a one-shot promise.** It resolves on the first `welcome` and never resets. After a disconnect/reconnect cycle, `group.ready` stays resolved — the `.then()` callback won't fire again. For UI connection status in long-running apps, listen for the `welcome` event instead:
```javascript
// BAD — status never recovers after reconnect
group.ready.then(() => showConnected())
group.on('close', () => showDisconnected())
// GOOD — fires on every reconnect
group.on('welcome', () => showConnected())
group.on('close', () => showDisconnected())
```
Use `await group.ready` for one-time setup (route registration, initial config). Use `group.on('welcome', ...)` for anything that needs to update on reconnection.
**`acequiaReady()` is not optional.** The acequia framework initializes asynchronously (IndexedDB, service worker registration, key generation). Code that runs before `await acequia.acequiaReady()` will see undefined device IDs and missing APIs.
#### Two Data Channels
Acequia groups have two distinct communication paths. Mixing them up causes subtle bugs:
| Channel | Carries | Latency | Reliability |
|---------|---------|---------|-------------|
| **Discovery server** (`peersList`) | Metadata: deviceInfo, capabilities, stats | Seconds (server round-trip) | High — server broadcasts to all peers |
| **WebRTC data channel** (`sendRequestViaPeer`, `peer.sendBig`) | Data: job payloads, results, progress, events | Milliseconds (peer-to-peer) | Connection-dependent — peers may disconnect |
**Stats go via discovery server.** When a worker completes a job, it calls `group.updateDeviceInfo()` which re-registers with the server. The server broadcasts updated `peersList` to all peers. This is how the dashboard sees stats from remote workers.
**Job data goes via WebRTC.** `submit()` connects to the worker peer directly and sends/receives over the data channel. Progress messages also go peer-to-peer.
#### WebRTC Peer Connections
**Reuse connections.** Each `connectToPeer()` call creates a new WebRTC connection, which involves signaling through the discovery server. Rapid sequential calls to the same peer will thrash connections. The WorkerPool caches peers internally and checks for existing inbound connections before creating new ones. If building your own peer-to-peer features, do the same.
**Inbound vs outbound.** When peer A calls `connectToPeer(B)`, peer B receives the connection via `peerSignalCB`. Both sides get a peer object, but only A explicitly requested it. If B later needs to send to A, it should reuse the existing peer (available via `getPeers()`) rather than calling `connectToPeer(A)` and overwriting the existing connection.
**"Peer never became ready"** means the WebRTC connection didn't establish within the timeout (default 10 seconds, 100 retries x 100ms). Common causes:
- Connection overwrite: calling `connectToPeer` when a peer already exists destroys the existing connection and starts over
- Network conditions: NAT traversal failures, especially across different networks
- Stale peer info: the discovery server lists a peer that has actually disconnected
#### `sendRequestViaPeer` Protocol
The request/response protocol wraps messages as `peerProxiedRequest`/`peerProxiedResponse` with a `requestId` for correlation. The receiver side (in `webrtc.js`) checks for pool jobs first (via `group._workerPool`), then falls back to route matching. Key details:
- **Pool responses are unwrapped.** The pool handler returns `{ type: 'pool:job-result', result }` directly as `response`. Route-based responses are wrapped in `{ data: responseData }`. Don't mix the formats.
- **Progress messages are separate.** `ctx.progress()` sends `{ type: 'pool:progress', jobId, data }` directly on the data channel — not as a `peerProxiedResponse`. The submitter listens for these with a `dataBig` handler alongside the request/response flow.
#### `updateDeviceInfo` and Dynamic Metadata
`group.updateDeviceInfo(info)` merges extra properties into `getDeviceInfo()` and re-registers with the discovery server. This triggers a `peersList` broadcast so all peers see the update. Use it for publishing dynamic state:
```javascript
group.updateDeviceInfo({
pool: { completedJobs: 42, activeJobs: 2 },
customState: { mode: 'processing' }
})
```
The merged data persists for the life of the Group instance and is included in every subsequent `register()` call and heartbeat.
#### Service Worker Caching
The acequia service worker caches scripts aggressively. After rebuilding and deploying `acequia.js`, **hard-refresh the page** (Cmd+Shift+R / Ctrl+Shift+R) or the browser may serve the stale cached version. Symptoms: new APIs are undefined, new classes don't exist, behavior doesn't match the source code.
#### Self-Dispatch
A WorkerPool with a `handler` includes itself in the workers list and can dispatch jobs to itself. When the selected worker is the local peer, the handler runs directly (no WebRTC round-trip). This means:
- A single tab can test job dispatch — the pool handles its own jobs locally
- The dashboard shows the local peer as a worker alongside remote workers
- Progress events from local dispatch are delivered directly to the ticket (no data channel needed)
Submitter-only pools (no `handler`) still exclude themselves from the workers list since they can't handle jobs.
## Group Shared State
Groups support shared mutable state that any member can update and all members receive change notifications. The state is managed by a peer-elected leader that merges patches and broadcasts updates.
**Design doc:** [group-shared-state-webrtc.md](../projects/completed/group-shared-state-webrtc.md)
**Implementation:** `acequia2/acequia/groups.js` (part of the `Group` class)
### When to Use Shared State vs WorkerPool vs Routes
| Use Shared State when... | Use WorkerPool when... | Use Routes when... |
|--------------------------|------------------------|--------------------|
| All peers need the same data | Work should go to ONE peer | One canonical handler per URL |
| Updates should reach ALL members | You need load distribution | You need fetch() semantics |
| Data is small (metadata, config, status) | Payloads are large or compute-heavy | Simple request/response is enough |
### Shared State API
```javascript
// Read current state
const state = group.state // plain object, shallow
// Update state (shallow merge, null deletes keys)
group.setState({ counter: 1, status: 'active' })
group.setState({ status: null }) // deletes 'status' key
// Listen for changes (from any peer, including self)
group.on('stateChanged', ({ state, patch, peerId, version }) => {
// state — full state after merge
// patch — just the keys that changed
// peerId — who made the change (null for leader broadcasts)
// version — leader's version counter (null for optimistic local updates)
})
```
### How Shared State Works
**Leader election:** On every `peersList` change, peers elect the leader with the lowest `instanceId` among those with the `state-leader` capability. All peers use the same sort, so they always agree.
**WorkerPool integration:** Any WorkerPool with a `handler` automatically gets the `state-leader` capability. No extra configuration needed — shared state works out of the box with pools.
**Update flow:**
1. Peer calls `setState(patch)` — optimistic local merge + fires `stateChanged`
2. Patch is sent to the leader (WebRTC if connected, WS relay otherwise)
3. Leader merges the patch, increments version, broadcasts `groupState:changed` to all peers
4. Each peer updates its state and fires `stateChanged` with the authoritative version
**Collect-then-lead:** When a new leader is elected, it requests state from all peers before broadcasting. This prevents a fresh peer from overwriting established state with `{}`.
**Transport:** WebRTC data channel is preferred (fast, peer-to-peer). If WebRTC connection fails, the leader falls back to WebSocket relay through the discovery server. State is always delivered — the transport is an optimization, not a requirement.
### Shared State Patterns
#### Broadcast Notifications
Use shared state to notify ALL group members of an event. Unlike WorkerPool dispatch (which goes to ONE worker), state changes reach everyone:
```javascript
// Sender: notify all peers of a completed upload
group.setState({
lastUpload: { filename: 'photo.jpg', url: '/uploads/abc/photo.jpg', ts: Date.now() }
})
// All receivers:
group.on('stateChanged', ({ patch }) => {
if (patch?.lastUpload) showUploadNotification(patch.lastUpload)
})
```
#### Presence / Status
```javascript
// Each peer publishes its status to shared state
group.setState({
[`peer:${acequia.getInstanceId()}`]: { status: 'recording', since: Date.now() }
})
// All peers see everyone's status
group.on('stateChanged', ({ state }) => {
const activePeers = Object.entries(state)
.filter(([k, v]) => k.startsWith('peer:') && v?.status === 'recording')
updateUI(activePeers)
})
```
#### Clearing State
Set keys to `null` to delete them:
```javascript
// Clear specific keys
group.setState({ tempData: null, oldStatus: null })
// Clear all state
const nullPatch = {}
for (const key of Object.keys(group.state)) nullPatch[key] = null
group.setState(nullPatch)
```
### Shared State Gotchas
**Shallow merge only.** `setState({ nested: { a: 1 } })` followed by `setState({ nested: { b: 2 } })` results in `{ nested: { b: 2 } }` — the second call replaces the entire `nested` object. To update nested data, spread it yourself:
```javascript
group.setState({ nested: { ...group.state.nested, b: 2 } })
```
**Optimistic updates fire before leader confirmation.** The local `stateChanged` event fires immediately with `version: null`. The leader's broadcast arrives later with an authoritative `version` number. If two peers set the same key simultaneously, the leader's merge order wins.
**State requires `state-leader` capability.** At least one peer in the group must have `state-leader` in its capabilities for shared state to work. WorkerPool workers get this automatically. If using Group directly, add it explicitly:
```javascript
const group = new acequia.groups.Group('my-group', {
capabilities: ['state-leader'],
// ...
})
```
**State is ephemeral.** It exists only while group members are online. When all peers leave, state is lost. For persistence, write state to the WebDAV server separately.
**Keep state small.** Every update broadcasts the full state to all peers. State is best for metadata, counters, and status — not large payloads. For large data, use WorkerPool jobs or direct file operations.
## Event Bus — Decentralized Pub/Sub
The Event Bus adds fire-and-forget publish/subscribe messaging to Groups. Publishers emit events on colon-delimited topics, and only peers with matching subscriptions receive them. Messages flow directly over WebRTC (full-mesh flood) with WebSocket relay fallback — no coordinator, no leader.
**Design doc:** [group-event-bus.md](../projects/completed/group-event-bus.md)
**Implementation:** `acequia2/acequia/groups.js` (part of the `Group` class)
### When to Use Event Bus vs Shared State vs WorkerPool
| Use Event Bus when... | Use Shared State when... | Use WorkerPool when... |
|-----------------------|--------------------------|------------------------|
| Events are ephemeral (happen and are consumed) | All peers need the same persistent data | Work should go to ONE peer |
| Multiple subscribers should receive the same event | Updates should merge into shared state | You need request/response semantics |
| High-frequency, fire-and-forget messages | Data is small metadata/config/status | Jobs need retry, timeout, progress |
| You need topic-based filtering | You need last-write-wins convergence | You need load distribution |
### Event Bus API
```javascript
// Subscribe — returns an unsubscribe function
const unsub = group.subscribe('file:changed', (event) => {
// event.topic — the actual topic published to
// event.data — the payload
// event.peerId — publisher's instanceId
// event.ts — timestamp
// event.local — true if published by this peer
// event.retained — true if replayed from retained cache
console.log(`${event.topic}: ${JSON.stringify(event.data)}`)
})
// Unsubscribe
unsub()
// Publish — fire-and-forget to all matching subscribers
group.publish('file:changed', { path: '/photos/cat.jpg', action: 'upload' })
// Publish with retain — last value cached and replayed to new subscribers
group.publish('status:battery', { level: 0.42 }, { retain: true, ttl: 60000 })
// Query subscriber count for a specific topic
const count = group.subscriberCount('file:changed')
// Get all local subscription patterns
const patterns = group.subscriptions // ['file:*', 'cursor:move']
```
### Topic Matching
Topics are colon-delimited segments. Subscription patterns support two wildcards:
| Pattern | Matches | Does not match |
|---------|---------|----------------|
| `chat:general` | `chat:general` | `chat:random`, `chat:general:thread` |
| `chat:*` | `chat:general`, `chat:random` | `chat:general:thread` |
| `sensor:**` | `sensor:temp`, `sensor:temp:living-room` | `chat:general` |
| `**` | everything | — |
| `file:*:upload` | `file:photos:upload` | `file:photos:resize` |
- `*` matches exactly one segment
- `**` matches one or more segments (must be the last segment in the pattern)
### Retained Events
Retained events store the last-known-value for a topic. When a new subscriber matches a retained topic, the event is replayed immediately with `event.retained = true`. Useful for "current state" values like battery level or connection status.
```javascript
// Publisher: retain the latest sensor reading
group.publish('sensor:temperature', { value: 22.5, unit: 'C' }, {
retain: true,
ttl: 60000 // auto-expire after 60 seconds
})
// Subscriber: receives the retained value immediately on subscribe,
// then receives live updates as they arrive
group.subscribe('sensor:**', (event) => {
if (event.retained) {
console.log('Cached value:', event.data)
} else {
console.log('Live update:', event.data)
}
})
```
Without a TTL, retained events persist for the lifetime of the publishing peer. With a TTL, they auto-expire and are removed from the retained cache.
### How Event Bus Works
**Subscription advertisement:** When a peer subscribes, it broadcasts `eventBus:sub` to all connected peers. Each peer maintains a map of `peerId → Set` for remote subscriptions.
**Publish flow:**
1. Peer calls `publish(topic, data)` — local matching subscribers fire synchronously
2. Publisher iterates remote subscriptions — for each peer with a matching pattern, sends the event via WebRTC (or WS relay if WebRTC failed)
3. Receiving peer checks dedup (bounded seen-set of 1000 msgIds), then fires local matching callbacks
**Transport:** WebRTC data channels are preferred (shared with state sync and WorkerPool). On `subscribe()`, the peer proactively attempts `connectToPeer()` to all group members. If WebRTC handshake fails, the peer is added to `_ebWsRelayPeers` and messages route through the discovery server's WebSocket relay.
**Deduplication:** Each event gets a `crypto.randomUUID()` msgId. A bounded LRU set (1000 entries) rejects duplicates. This prevents double-delivery when messages arrive via both WebRTC and WS relay.
### Event Bus Patterns
#### Activity Feed
```javascript
// Publisher: announce file operations
group.publish('file:activity:upload', {
path: '/documents/report.pdf',
size: '2.4MB',
user: acequia.getInstanceId().slice(0, 8)
})
// Subscriber: display activity log
group.subscribe('file:activity:**', (event) => {
const action = event.topic.split(':').slice(2).join(':')
addToFeed(`${event.data.user} ${action}: ${event.data.path}`)
})
```
#### Collaborative Cursors
```javascript
// Each peer publishes cursor position at throttled rate
document.addEventListener('mousemove', throttle((e) => {
group.publish('cursor:move', { x: e.clientX, y: e.clientY })
}, 50))
// All peers render remote cursors
group.subscribe('cursor:move', (event) => {
if (!event.local) renderCursor(event.peerId, event.data)
})
```
#### App-to-App Pipeline
Multiple apps in the same group can form a processing pipeline:
```javascript
// App 1: Camera capture
group.publish('pipeline:frame', { imageUrl: '/tmp/frame-001.jpg' })
// App 2: Object detection (subscribes to frames, publishes detections)
group.subscribe('pipeline:frame', async (event) => {
const detections = await detect(event.data.imageUrl)
group.publish('pipeline:detections', { frame: event.data.imageUrl, objects: detections })
})
// App 3: Dashboard (subscribes to detections)
group.subscribe('pipeline:detections', (event) => {
updateOverlay(event.data.objects)
})
```
### Event Bus Gotchas
**Best-effort delivery.** Events are fire-and-forget. There is no acknowledgment, no guaranteed delivery, and no ordering guarantee. If a peer is disconnected when an event is published, it will not receive it (unless the event is retained).
**No backpressure.** High-frequency publishers will flood all matching subscribers. Throttle on the publisher side if needed.
**Callbacks are synchronous.** Subscriber callbacks are invoked directly (not via `Promise.allSettled` like `Evented.fire()`). A slow callback blocks delivery to subsequent subscribers on the same peer. Keep callbacks fast, or offload heavy work to a `setTimeout` or WorkerPool job.
**Retained events are per-publisher.** Each peer stores its own retained events. When a publisher goes offline, its retained events are lost for future subscribers. Retained events are not a persistence mechanism — use WebDAV for durable state.
**Topic naming convention.** Use colon-delimited segments, noun-first: `sensor:temperature:living-room`, `file:activity:upload`, `cursor:move`. This works naturally with wildcard patterns.