Loom — Real-Time Site Structure Visualizer

Problem

Sitemaps and crawl reports tell you a site's structure exists; they don't make you feel it. I wanted to watch a site's link graph assemble itself — orphan pages drifting in from a sitemap, hub pages pulling neighbors into orbit — instead of reading a table of URLs.

Architecture

Loom runs as three cooperating services, deliberately kept separate:

• App service — a custom Next.js server (server.ts) that serves the UI, hosts Socket.io, subscribes to Redis Pub/Sub, and broadcasts crawl events to every connected browser.
• Worker service — a BullMQ worker (worker.ts) that fetches HTML, parses links with Cheerio, applies depth and external-domain rules, updates Redis visited-state, queues follow-up jobs, and publishes node/link events back to the app.
• Redis — the shared broker and state store: backs the BullMQ queue, holds a per-crawl visited-URL set (SHA-256 hashed, scoped by crawlId so re-running the same seed does a clean crawl), and carries live events on a loom-events Pub/Sub channel.

The frontend renders the graph with react-force-graph-2d, with physics controls exposed live — repel force, link distance, hub gravity, node sizing by link degree — so high-authority pages visibly grow into planet-sized hubs as the crawl fills in around them.

Interesting decisions

• URL identity strips query strings and folds www. into the apex host, so /contact?plan=pro and www.example.com/contact collapse to the same graph node instead of fragmenting it.
• Sitemap-discovered orphan pages aren't just plotted passively — the worker queues them for crawling too, so orphan clusters can reveal their own internal linking instead of sitting as isolated dots.
• A "force purge" control exists separately from the normal stop button: it broadcasts a worker control event, aborts in-flight fetches, and deletes persisted BullMQ/Redis state, for when a crawl runs away and the normal stop isn't fast enough.
• Graph node objects are passed to the force-simulation without cloning on every render, specifically to preserve live simulation positions — re-creating nodes each render would make the physics jitter instead of settle.

Deployment

Ships as a multi-stage Docker image with separate ui and worker targets, deployed via Coolify — the same self-hosted pattern I use for most of the homelab.

Lessons learned

Keeping the crawl worker and the Socket.io gateway in genuinely separate processes (not just separate files) made the failure modes much easier to reason about — a runaway crawl can be killed without taking the UI down with it, which would not have been true if everything shared one Node process.