projects · hutfx · web · Pipeline Engine — website section 3 ("Model Builder") · ✎ edit

Pipeline Engine — website section 3 ("Model Builder")

Method (same as the website spec): the owner describes what it is and what it must do first. Tech choices and the page layout are decided only after the description is locked.

What it is (in the owner's words)

A pipeline is a process that goes through several stages. We need to build a pipeline tool that allows us to:

  1. choose a dataset
  2. process it through different modules — cleaning data, making graphics, some calculation, and many, many other things you can think of to process some data
  3. see the outcome in a viewer where we see the result
  4. at the end, save it as:
  5. a module — to be used for insertion into another pipeline (e.g. in its second stage)
  6. a research — just saved as research
  7. an export to many different other services — e.g. ML, create LLM, or just to send a signal to the device (many things we don't clarify yet)

The idea is that this web page with the pipeline needs to be used and reused for our services and processes — so basically we want to build something like the Unreal Engine interface or Blender: an infinite canvas with dots, where you create boxes, and inside each box you have some action/function, a sandbox to write Python scripts, or a build-block schema with IF/else blocks.

Flow

flowchart LR DS[Dataset
choose / upload] --> ST1[Module: clean data] ST1 --> ST2[Module: calculate] ST2 --> ST3[Module: graph / visualize] ST3 --> V[Result viewer] V --> SM[(Save as module
insert into another pipeline)] V --> SR[(Save as research)] V --> EX[Export to services
ML / LLM, device signal, API, ...]

(The middle stages are any chain of modules — the diagram shows the shape of a pipeline, not a fixed sequence.)

The 4 parts

1. Choose dataset

All four load paths, confirmed by the owner in the section 3 description (website spec): "select data or upload or read real time from something or it might come from the Data Router":

2. Process through modules

An infinite canvas with a dot grid (Unreal Engine / Blender node-graph style). You create boxes (nodes) and connect them; data flows through the pipeline, stage by stage.

Inside a box, one of:

Module catalogue (brainstorm — confirm / add / drop):

Stage Module What it does
Input Choose dataset Pick a dataset from Data Exchange
Input Upload Bring a new file into the pipeline
Processing Clean Remove duplicates, fill / fix missing values
Processing Filter Keep rows / columns matching a condition
Processing Transform Rename, reshape, join, split
Processing Calculate Arithmetic / statistics on the data
Processing Features Build feature sets
Processing Aggregate Group / summarize
Visualization Linear graph Plot data
Visualization Heatmap (and other sub-tools)
Logic IF / else Branch on a condition
Logic Merge / split Combine or route data streams
Sandbox Python script Custom Python for this step
ML Train model Fit a model on the processed data
Output Result viewer Show the result of a stage / of the pipeline
Output Save as module Package the pipeline as a reusable box
Output Save as research Store as a research record
Output Export to service ML / LLM creation, device signal, API, …

(…and "many, many other things you can think of to process some data" — keep adding here as they come up.)

Also confirmed by the owner in the section 3 description (website spec):

3. View the result

4. Save / export

Reuse across services

The pipeline page is a section of the website, but the engine itself is reused for hutfx's services and processes:

(to describe with the owner.)

Naming & terminology (owner's notes)

Tech (decided after the description is locked — in progress)

The owner started the tech conversation (canvas approach) while the description is still being finished — the decision below is a proposal until the description is locked.

Canvas approach — Svelte Flow (proposed, awaiting owner confirmation)

Svelte Flow (@xyflow/svelte) — the official Svelte port of React Flow by the xyflow team (same maintainers, node-based UIs since 2019):

Stack impact: hutfx has no Svelte today (Laravel + Blade + Alpine + Vite). Pattern: Svelte as a widget runtime — the pipeline page mounts a Svelte app into a <div> from Blade, with its own Vite entry (precedent: resources/js/dashboard.js for the Chart.js dashboard section). Adds svelte, @sveltejs/vite-plugin-svelte, @xyflow/svelte to package.json and one plugin line to vite.config.js; Alpine and the rest of the stack stay untouched. Honest downsides: smaller community than React Flow (fewer answers online), one more framework runtime on the page. The alternative (React Flow — same team, same API) would bring a React runtime instead: no meaningful win for us.

Open questions (answer before tech)

  1. Node shape — what does a box show? Ports (inputs / outputs)? Label?
  2. Execution — run the whole pipeline at once, or per node? Live, as data flows through?
  3. Where does execution run? On our server? — language: answered in the owner's section 3 description — everything runs in a sandbox: Python, R, Julia
  4. "Send a signal to the device" — what does it mean technically (protocol, payload)? — context: see part 4, device-to-device
  5. Versioning — does saving a pipeline overwrite it, or create a new version?
  6. Access — can all users use it? Per-user isolation of pipelines?
  7. Templates — answered (section 3): both — predefined pipelines as templates, or a blank canvas for advanced users
  8. Size — how big can a dataset be for in-pipeline processing?

Proposed milestones (after the description is locked)

# Milestone Status
1 Engine foundation: pipeline + module data model; canvas with boxes & connections; save / load Not started
2 First live pipeline: choose dataset + one processing module + result viewer Not started
3 Save as module: compose a pipeline into another pipeline Not started
4 First export to a service (one concrete target) Not started
5 Python sandbox + IF / else build-blocks Not started

Steps

Result

(Draft — description in progress with the owner.)

source: DOCS/hutfx/web/2026-08-17-pipelineengine.md · edit this page · rebuilt by build-wiki