The Pipelines page

Pipelines has five pages in the side: Jobs, Catalog, Runs, Logs and Models. The section opens for a person with see on Pipelines or on Models, and each page needs its own grant. Everything heavy runs as a job the engine queues: reading a dataset, pseudonymising, sorting, picking, making pictures, releases, backups and analyses. What runs is here as it runs; what ran stays.

Watch the jobs

1. Open Jobs. Now lists what runs and what waits, live from the engine’s event stream where it serves one (updated every second), and read every few seconds otherwise.
2. Each job is a card: what it is doing and to what, for whom, how far it is, and what it is waiting for. Steps queued after a job show as one dashed card under it.
3. Under Now, every job the engine lists, filtered by all, running, queued, done or failed (failed holds the cancelled).
4. A job that stopped says why in the words the verb itself printed. Press the command it offers to run it again, or Dismiss.
5. A pipeline run’s job opens its run: click the card, or press Open run.

The engine’s standing workers show apart as one line, “Workers: 3 ready”, that opens to each worker’s kind and last heartbeat. A worker silent for two minutes shows as not answering.

Pictures have a queue of their own. When a sort ends, the pictures of the scans it judged, then their 3D views, are queued there as jobs of their own, below the rest, so they never hold up a read, a sort, a pick or a score. A large dataset’s pictures are made several scans at a time.

Cancel

Cancel (or Drop, for a job still queued) goes by the verb’s grant. The card keeps its button and says why when you lack the grant.

Job	Cancelled with
a read, a pseudonymisation, a bring-in, a vaulting or purge of the originals, a map filed, a merge	work on Data
a fingerprint, a sort, a pick, a session build, a picture build	work on Pipelines
a release, a hand-over	work on Release
a backup, a backup check	work on Backups
a query run	work on Query

A running job stops at its next heartbeat and what is written stays written. A queued job is dropped from the queue.

Chains

A job may carry the steps queued after it. Each starts when the one before it ends done, under the grants and detail of the person who queued the first. A step that person may not queue stops the chain, and the job’s result says which step and why. Read a dataset is such a chain; see the chain for the engine’s side.

Read a job’s log

1. Open Logs. The jobs are listed newest first.
2. Choose one. Its log shows its end first and follows it while the job runs. Press Earlier to read back.
3. A job that ran a pipeline lists its run’s units under its log. Choose a unit to read its own log.

A log the engine will not show says why: none was kept, it is another host’s, it holds a sealed sample, or you lack the grant. A job’s log needs the work of the page its verb belongs to.

Run an analysis

The Catalog lists the pipelines the engine serves, each pinned by its image, with what it writes and its newest run. An engine seeds a starter catalog: N4, SynthStrip, SynthSeg, SAMSEG with lesions, segcsvd, MRIQC and FreeSurfer recon-all. Their images are large and are pulled on a run’s first use. segcsvd reads each session’s SynthSeg label map, so run SynthSeg first. FreeSurfer reads the lab’s licence as a secret.

1. Open Catalog. A row is one name. Its select offers the versions a new run may take, newest first. A version a later one replaces because it fixes it is not offered.
2. Press Run on a selection.
3. Over: type to find a saved selection by its name and words (“t1 3d” finds t1-3d-stacks). Each entry says its newest version, its size, who made it and when. Or choose a dataset or an open cohort, shown as “study-a · main T1w”: the run goes over its main scans by role, and a pipeline that needs several roles takes every one of them. Exact text such as selection:x@2 is taken as typed and checked by the engine.
4. Set the parameters. Each shows the range its descriptor allows.
5. Read the pre-flight before anything runs:
   • the units ready out of the whole, and the units that lack an input, folded by why (“2 sessions cannot run: no FLAIR picked”) or “All 4 sessions can run.”;
   • the stacks left out, the estimated time, the card, and the lane’s budget;
   • what the run needs of this machine (the converter, the runtime, the image, memory, the card, disk), each held or not in the engine’s words.
6. Press Run. It waits while the engine names a blocker. With no container runtime the page says why and offers no Run.

A run takes a selection of any size. For a very large one, --chunk runs it in parts of at most N stacks, in stack order, each part a run of its own, one after another. A handle splits the same way:

nils run <pipeline> --select selection:NAME@V --chunk 20000
nils run <pipeline> --handle <id> --chunk 20000

Under a worker the parts are one job, which stays running until the last part ends; a stop asked between two parts stops there. Taking a part up again (nils run --resume <run>, or nils jobs resume) ends that part and goes on with the parts after it that no run holds yet. System 1’s score at arrival runs a large batch in parts the same way.

Runs whose stacks do not meet go on side by side and share the lane’s one budget. A run that would take a stack a running run holds is refused before it starts, and the refusal names that run and its job.

An admin adds a pipeline (Add a pipeline, from its nils.job.yml), sets where pipelines run, and sets or unsets the secret files a pipeline declares, such as a licence, whose contents never show. The three need work on the install. The models’ pipelines come from the engine’s release: nils pipeline add --release NAME.

Two lines on the Catalog say how pipelines run, for a person with see on Pipelines:

• Runtime: what runs pipelines and which runtime is chosen. Change chooses the runtime, and how Apptainer keeps an image: one sif file, or a sandbox folder where the machine has no /dev/fuse.
• Starters: whether the engine seeds its starter catalog when it starts, and how many of the starters it holds. Seed now adds the ones lacking; the seeding at start is turned off or on beside it.

Changing either is work on the install, and every change is audited.

To have the assistant plan a run, open the Assistant and start a conversation with Plan a pipeline run; the run opens filled in here, for you to start.

Read a run

1. Open Runs, or press Open run on a job. A row opens its run wherever it is clicked.
2. The table says what the run made, never a bare done: “4 of 4 made”, “3 of 4 made” or “nothing made”, and under it the rest (“4 of 4 skipped: no FLAIR picked”).
3. The run’s page shows its units by state, each done, failed or skipped, with what a skipped one lacked. A failed run says why in one line, with the cause counted by kind.
4. Cancel while it runs; Resume once it stopped, which keeps what it finished.
5. The page also has the table of measures as the ask reads it (one row per scan at the identifying details level; below that, totals over groups, and the engine withholds a group of fewer than five scans), the run’s declared checks with their breaches linked to review items, and the files it made.
6. A unit with a mask offers View: the scan the mask’s grid is on opens with the mask drawn over it, read only, with one chip a part set (Lesions 8, Structures 36). Check results starts a check.
7. A run that suggested scans to curate says so under Suggested to curate: the values by axis, and Save as a selection, which a campaign starts from. A scan of a sample sealed now is left out, and counted.

When a run fails

1. Read the reason under the row. Where the engine names the failure it gives the kind (dependency, resource, input or pipeline), the cause, and one sentence of what to do.
2. A missing tool: an admin presses Install dcm2niix beside the reason, then Run again. The page says the version. If the install fails, it says the engine’s reason whole.
3. Where it failed because the engine cannot reach the releases, a line names the archive to fetch, with its sha256. Fetch it on a machine that can, and press Install from a file beside Try again; the engine checks it against the pinned sha256 before it installs it. Anyone else reads “Ask an admin to install dcm2niix.”
4. Run again on a failed run opens its pre-flight first and takes the run up again where it stopped.

Choose the model that answers a task

Models shows each task and slot with its models side by side, as cards. One model answers each task: the one chosen there.

A card says
tested on	what it was tested on, with the evidence run
speed	on the CPU and on a card, and what it needs of a card
equivalence	its CPU and card answers, stated and measured
trained on	the label set
complete	or what it lacks
certified	the graded run or the sealed sample that certified it, with its numbers, or that none names it

The chosen model is marked by its border, with why it answers: chosen by a person, a certificate, the one promoted last, or the newest admitted.

1. To change the model of a task, press Choose on the card of another admitted or promoted model, as an admin. A panel says what changes; give a why, which the audit keeps. Back to default, beside the task’s name once a person has chosen, opens a panel; press Use the default. Then the certified model answers, else the one promoted last, else the newest admitted.
2. To add a model, press Register, paste its card and press Register the card (models:work). The engine refuses in its own words, shown as they are. A card registered without some of its claims (the label set it was trained on, its tested claims, its speed) gains them at the keyboard with nils model add-claims MODEL --claims FILE: additions only, each checked as registration checks it, and a claim the card states already is refused.
3. Admit a registered model by a check, Promote it, or Retire it. Promoting retires nothing: several promoted models stay in a task and the site chooses. A retired model can be run again by name to reproduce a result.
4. A model’s own page (#pipelines/models/<id>) keeps its card, its encoders, its label set, each tested claim and its history. Record a certificate cites the graded run that measured the model: the run, the sha256 of its manifest, the commit that registered the plan, the bar, how many stacks it graded and how many were wrong. It is your own act under your sign-in and unseals nothing, so no second person is needed. The engine works the bound out again from the counts.

What a model proposes waits in Review, Proposals, staged until a person commits it.