The Data page

Data has four pages in the side: Datasets and cohorts, Browse, Pseudonyms and Clinical. It opens for a person who may see Data. Every act below needs work on Data unless it says otherwise; adding a root folder or a dataset needs work on Places as well. Add a dataset, the steps on a dataset’s card and the Pseudonyms page need engine 1.0.0-alpha.80 or later, and Clinical 1.0.0-alpha.88 or later.

Add a dataset

A dataset is a folder under a root folder. NILS looks inside the folder once, before anything is declared, and reads what the dataset is from its structure. It never asks you to say what the dataset is.

1. Open Datasets and cohorts and press Add a dataset.

2. With no root folder yet, the dialog asks for one first. Choose the folder your dataset folders are in, from the engine’s own locations or with Type a folder, and press Add. Setup’s Add a root folder step and Settings > Places do the same.

3. Find the folder. Type part of its name, or press Browse; the list shows fifty at a time and More shows the rest. With more than one root, choose the root first.

4. Choose the folder. The dialog says what it found: DICOM yes, no or unknown; an identified folder, yes or no; an anonymised folder, yes or no; the loose entries that hold DICOM; and what it would be.

5. Answer the one question the folder asks, if it asks one.

   The folder would be	The dialog asks
   Anonymised	PatientID holds: the subject code, or an ID type. Subjects: made from the ID, or from a map.
   Identified	PatientID holds an identifying ID: one that is the same everywhere (personnummer, national ID), or a hospital or study ID, with its type. For a hospital or study ID, also An ID with no subject code: generate its subject code from the ID (recommended), or wait for a map.
   Unknown	nothing now; the card’s button is Sort the files

6. Press Add. A folder with no DICOM asks Add anyway? first.

Adding queues no job and moves no file. The card says what is next. The rename of a derivatives/dcm-raw folder of the earlier NILS to dcm-anon is the one automatic change, and it is shown.

Warning: the ID of an identified dataset is never kept. A personnummer or any other ID that leads to a person becomes a subject code. NILS keeps its keyed lookup and nothing else.

What the structure means

Under the dataset’s folder	The card says	What reads it
only derivatives/dcm-original	Identified	the pseudonymiser, which writes derivatives/dcm-anon
only derivatives/dcm-anon	Anonymised	the registry
both	Identified, with its anonymised copy	the registry reads dcm-anon; the pseudonymiser reads dcm-original
anything else: DICOM beside derivatives/, or no tree at all	Unknown	nothing

An anonymised dataset is read only once it says what PatientID holds and how its subjects are found. The registry never points at an identified file: a read of dcm-original is refused.

Sort the files

1. Choose a dataset whose card says Unknown and press Sort the files.
2. Say what the loose entries are: Identified or Already anonymised. Press Next.
3. The dialog names how many entries would move and into which folder. Nothing has moved yet. Press Move them.

Each move is a rename inside the folder. Entry names show only at the everything detail level.

Set the IDs

Anonymised data says its IDs before it is read. Choose the dataset and press Set the IDs (or More, The IDs) to change them.

1. PatientID holds: the subject code, or a hospital or study ID of a type.
2. Subjects: Made from the ID, or From a map of subject codes to the IDs.
3. Folder names: the subject code, or the ID when PatientID holds one.
4. Press Save.

For identified data the same dialog sets what the pseudonymised copy’s PatientID gets; see the rules of a dataset.

Read a dataset

1. Choose the dataset. The primary button names the next step: Read, Sort, Review n, Read again or Read new files.

2. Press it. The step runs as a job, and the card’s log shows it with its progress and Stop.

3. Press Read new files after more files land. It queues the whole chain, one step after another:

   Step	What it does
   Pseudonymise	identified data only: the new originals into dcm-anon
   Read	what is new in dcm-anon into the registry
   Sort	fingerprint, then classify with the rules
   Pick	the main scans, where the dataset feeds a cohort
   Score	System 1 over the new scans, where its model is chosen

4. More, Read again reads the whole tree again.

Each step starts when the one before ended done, under the grants of the person who pressed. A step they may not queue stops the chain and the card says why: without work on Pipelines the chain stops after the read. A read that brings nothing new ends before the sort. On the engine the chain is the verb bring-in; see pseudonymise a dataset.

System 1 scores every dataset as it arrives: the chain’s last step runs it over the new scans, with no button. A large batch is scored in parts, one after another, in one job. Body part and post-contrast never run at arrival. They are optional: run them from the rail where the data need them.

The sort ends once the rules have judged the scans, so the pick and the score start at once. The pictures of the scans it judged, and then their 3D views, are made after it, as a job of their own on a queue of their own. While they are made, the page says Preparing pictures in one quiet line, with about how long is left; the job itself is on Pipelines.

A read that stopped

A read that was stopped, or failed part way, has not read every file. Its scans are not sorted as they stand: the card says “The last read stopped part way”, the rail draws the read as stopped, and the primary button is Read again.

1. Press Read again. It queues the chain from the read on: the read, which skips the files read already, then the sort, the pick where the dataset has one, and System 1’s score over the scans of both reads.
2. Follow it in the card’s log.

It never queues a pseudonymise step. Where that step waits, it comes first.

Where it is

A dataset’s detail shows its steps as a rail. Each says what it did and when: the time it ended and how long it took, “now” with about how long is left while it runs, or “next” while it waits in the queue. Run on body part, post-contrast and main scans needs work on Pipelines.

Step	It says	Run
Found	the files in the tree; “at least N” until a read counts the tree whole
Pseudonymised	identified data only: the files still to do, or the copy’s files once none waits, and the files held for a subject code	opens the step in place
Read	files read, refused and set aside; “stopped” when the newest read stopped part way
Sorted	scans sorted, sure, and needing a look
Body part	while it waits, what a run would add (“6 with no body part”, or “none needs it”); after a run, the scans the chosen model answered	Run
Post-contrast	while it waits, what a run would add (“12 T1 brain, contrast unknown”, or “none needs it”); after a run, the scans the chosen model answered	Run
Main scans	scans picked, and the borders	Run, and a Rules link to Main scans
Pictures	pictures made, or how far the job that makes them is
3D views	3D views made, or how far the job that makes them is

A step with no model served for it says “not available”. A Run that the engine would refuse stands disabled, with the engine’s reason behind a “?”. A model reads the images, which needs seeing records with identifying details.

The card on the list draws the same steps as a bar. While a dataset’s summary is still being read, a step the card cannot tell yet is drawn as not known yet, never as not done, and no count is guessed.

A count that stopped short of a large folder says “at least” wherever it shows: on the card, on the rail, in the pseudonymise step and in the folder picker. A whole count is the number alone. The engine counts a folder only for a short while; a read or a pseudonymise run counts it whole.

What it holds

The detail shows subjects, visits and scans as a funnel. A bar splits the scans into sure, need a look and not sorted; the hover names what the scans that need a look are asked. Under it: the kinds of scan, and the files with the counts below.

Count	It means
set aside	files that are not scans (screen captures, presentation states); asked of nobody
refused, why	files the reader refused; opens the batch page
gone from the folder	files the registry holds that the folder no longer has
left out of the read	files a person let go
a scan filed already elsewhere	n questions; opens Subjects

A line under the head says whether the dataset’s headers are as usual or damaged (“Headers damaged: sorting less reliable”), by its signs or because a person flagged them, with a why. The signs are read at the end of every sort: the share of unnamed scans, of scans with no word the rules know, and of scans by a maker never seen, each held to the rules’ threshold; one that reaches it flags the dataset. A person’s word stands over the signs. The line offers Flag as damaged, Mark as usual or Let the signs decide. On a flagged dataset System 1 stages its answer where the rules name nothing and it is confident, for a person to confirm one question at a time in Review. It never fills an answer by itself.

A scan copied into two datasets is one scan with a path in each. A dataset counts the scans its folder holds files of; only a cohort has members.

A batch opens its own page at #data/batch/<id>: five stages (pseudonymised, walked, digested, classified, reviewed) with a count under each, the files it refused by reason with Read the N again, and the jobs that ran on it. Read again and Sort N act on the batch.

Pseudonymise

Identified data is pseudonymised before anything reads it. The step opens in place under the rail.

1. Press Pseudonymise on the card, or choose the step on the rail. Three boxes show the originals (locked), the IDs and how many have a subject code, and the pseudonymised copy, which is what NILS reads. A count that stopped short of a large folder says at least.
2. Read the button. It says what happens next:
   • Pseudonymise and sort N files: the key codes every ID, or the dataset is set to generate a code for an ID that has none. Press it and go to step 7.
   • Pseudonymise and sort the rest: a run before was stopped, or a count stopped short, so how many are left is not known. A run again skips what is done. Press it and go to step 7.
   • Find the IDs: press it. The dataset’s thread runs once and holds the files whose ID has no subject code.
3. The step now says “Next: n IDs have no subject code yet” and offers two ways on.
4. Generate subject codes gives every waiting ID a code made from the ID itself, which is final and the same wherever that ID is read, and then runs the thread at once. Go to step 7.
5. Give them from a map opens the table of IDs: one row each by shape, never by value, with their files and their subject code or what they wait for. Give the codes:
   • Drop a CSV of ID and subject code on the box, or press Choose a file or Paste. The engine rehearses it and the rows fill as they match. Conflicting rows are named and nothing of the map is filed while one stands.
   • Or generate in the table: the button on a row codes that one, and Generate subject codes for these N codes the rest.
6. Press the button that says what it will do, Pseudonymise and sort N files. It files the map, then runs the dataset’s thread: pseudonymise, read, sort.
7. When it is done the step lists what every file got.

A run that wrote every copy it could is done: the step offers no button, the rail says the copy, and the dataset goes on to its read. A file the run refused as not DICOM is not left to do. A run that was stopped leaves the rest, and the card says “The rest waits to be pseudonymised”.

Right after a run, while the dataset’s summary is read, the step shows the counts it has, offers no button and says Counting what is left… (or that what is left could not be counted). The IDs box shows “…” until the held IDs are read, never 0.

In every file
PatientID	the subject code, or the subject’s ID of the type chosen
removed	the tags of five groups: patient, provider, trial, institution and IDs (145 in all), less the few whose row says why they stay (sex, weight and size as covariates, the scanner’s identity, what the rules read from the copy, cleaned of the file’s own IDs). The examination’s accession number and study ID go from every file, whatever a list says
kept	sex, weight and size (covariates, unless the dataset opts out), dates, times and UIDs
written	PatientAge, computed before the birth date goes
dropped	private tags except what the rules name, overlays and curves
marked	de-identified, and how, in the file’s own header

Show the IDs shows the values once and records it in the audit as read by you. It needs everything in records. Giving a map reads identifiers and needs the same. A pseudonymise run needs work on Data and the everything detail level; the engine’s side is pseudonymise a dataset.

The rules of a dataset

The step shows Standard rules. Press Change to change them:

Rule	Choices
PatientID holds (the originals)	the kind of ID, changed until anything is pseudonymised
The copy’s PatientID gets	the subject code, or a hospital or study ID. The copy then carries an ID that can lead to the person: choose it only where a study needs it.
An ID with no subject code	its files wait for a map, or it gets a generated subject code
Tags	the standard removal, or the dataset’s own: Choose tags
The originals, once done	kept, vaulted or purged

Choose tags lists every tag the pseudonymiser acts on, by group, each with what becomes of it, and the dataset’s own kept and removed ones. Four are never removed: PatientID, which becomes the code; PatientAge, which is computed; and SOPClassUID and SOPInstanceUID. A tag cannot be on both lists: keeping beats removing, so it is refused. The list is the engine’s own; see the tag list.

More, The originals acts on them. Vaulting moves them into a backup place you choose, with a reason. Purging deletes them, asks for a reason and the dataset’s name typed out, and says first what it would reach. A purge is refused while a file waits for a map, unless every original is verified in the pseudonymised tree, and for a dataset read in place. Both run as jobs. The engine’s side is what becomes of the originals.

More, Settings sets the cohort a dataset feeds: every subject a read adds joins it, and taking it away keeps every member. It needs work on Places too. More, Remove takes the dataset off this page; its folder and what NILS read from it stay.

Pseudonyms

Pseudonyms holds what spans datasets: the ID types, the maps, and, for an admin, one subject’s IDs.

Add an ID type

1. Press Add an ID type.
2. Give a name in lower case letters, digits and hyphens, and say in a few words what one of them is.
3. Under Leads to, choose A study’s subject, or A person, anywhere (a hospital’s record number). The second marks the type Leads to a person, and looking a subject up by one then needs work on Data and the everything detail level.

The table lists each type with how many IDs and subjects it holds, never a value. Two types are never kept: the ID that is the same everywhere, and a hospital’s reserve number. Each has a never kept tag, and its count reads as lookups (“4,836 lookups”): no number is stored, only a keyed lookup that knows the number again when it arrives.

Give a map

A map is a CSV with a header row and a row for each person. Filed once, it holds for every dataset: any of its IDs arriving in any dataset lands on the same subject.

1. Press Give a map.

2. The file. Choose the CSV; up to 100,000 rows at once. It is read in the browser and posted once to the engine; nothing of it is kept on this machine.

3. What each column is. The dialog shows each column by its header, the shape of its values and its counts, never a cell, and guesses what the column is from its header. Change it:

   A column is	What it does
   an identifier	filed on the row’s subject, under a type of the site or a new type made here with a description
   the ID the subject code comes from	the code derives from it, as a read derives it
   the subject code itself	names the subject as it is
   not read	read past

4. Press See what it will do. The engine reads the whole file and writes nothing: subjects known and new, identifiers filed, held files it would release, merges, and the first fifty conflicts with their rows.

5. Press File the map. It is offered only after a rehearsal that came back clean, and it files as a job. Held files it names are pseudonymised and read after.

A column headed for a hospital’s reserve number (reservnummer, reserve number and the like) is read as the reserve number type. No subject code is ever made from a reserve number, so the person’s code or another ID of theirs goes beside it, and a map whose column for the person is a reserve number is refused before it is sent.

Filing needs work on Data and the everything detail level. The engine’s side is identifiers and the map.

One subject’s IDs

An admin reads one subject’s IDs here.

1. Type the subject code and press Show. The panel lists the subject’s IDs by type, where each came from and whether NILS keeps it, with no value.
2. To see the kept values, press Show the values and give a reason, which the audit keeps. A value that reads as a personnummer is shown as its shape alone.
3. The linkages that name the subject are listed, each with Unlink. Link a subject records another subject as the same person, with its evidence, and merges it in.
4. Purge says first what would go, and is sent only with the subject’s code typed. Purge every subject needs “every subject” typed.

Shown values stay in the panel alone, and go when the code changes or the page closes. The engine’s side is one subject’s identifiers.

Browse

Browse is the dataset viewer over the whole registry. A card’s View opens it on one dataset or one cohort.

1. Open Browse. The Grid shows subjects as folders, with no pictures. Find one by search (/) or by the filters. The search takes a subject code, or a full ID at the everything detail level.
2. Press Enter on a subject to see its visits, and again on a visit to see its scans. Only the scans level shows pictures, in folders by data type that fold, each family outlined in its colour.
3. Esc goes up a level. g switches Grid and Browser (the tree and one scan) at the same place.
4. A level’s bar ends in Save as a selection and Make a cohort, over what the filters show. The dialog counts before it keeps anything.

The place is in the address, so back and forward work. The subjects’ search is held in memory only, since a person may type an identifier there. What shows follows the detail level of the person.

Cohorts

A cohort is a membership and nothing else: it names subjects from any dataset. A cohort is a card under the datasets.

1. Press New cohort.
2. Give a name (one word, no @, / or spaces), an owner, what it is for, and optionally the codes, one per line, that join at once. A code the registry does not know is named back, not added.

A subject joins when
a dataset feeds it	set on the dataset: every read adds the new subjects it brings
a query promotes them	on a Query card, Make a cohort: the subjects of the answer’s rows
a hand adds them	More, Add or take out by hand, with a reason; leaving closes the membership and erases nothing

Choose a cohort to see how it grew, where it is, where its subjects come from and its log. Its rail has the members’ scans sorted, body part, post-contrast and main scans; body part and post-contrast say, while they wait, how many scans a run would answer. The primary button is Review n, Release or Add subjects from a query. More also has Rename and Retire; a retired cohort keeps its members and history and Bring back returns it.

Main scans, visit by visit

The cohort’s Main scans, visit by visit shows one row a subject and one cell a session, for the role chosen as a key. A cell is picked, doubted or missing by its fill, and a person’s pick by its border. A cell opens Main scans.

A Layer colours each cell by what its scan is: body region, body part or post-contrast. A line says whether the step that answers the layer has run over the cohort, with a Run where it has not.

Clinical

Clinical holds the vocabulary of diseases and kinds of observation the registry knows, and brings clinical rows in from a file. It opens for a person who may see Data; loading a vocabulary and importing need work on Data.

1. Open Clinical. The table lists each kind of observation with its category and the value it takes, and marks the primary and the sensitive ones.
2. With no vocabulary yet, press Load a vocabulary: the pack’s own, or a file of the site’s own. What exists is updated by name, what is new is added, and nothing is removed.
3. Press Import a file. Choose the Mapping, which says what the rows are, how a row names its subject, the columns and how each is read, and what to do with a row that exists. Choose the Rows, a CSV with a header line.
4. Press Preview. It says what the import would give: rows added, already there, updated, superseded, for review and refused, the refused counted by reason. Nothing is written.
5. Press Import. The rows of the preview are written under your name, in one go.

A row the registry holds already is left as it is, so the same file twice changes nothing. A mapping that names its subjects by an ID needs the everything detail level, and is held back for anyone else before it is sent.

On the command line

The cohorts are nils clinical cohort list|show|make|rename|set|retire|add|remove and the doors under GET /api/cohorts. A promotion is nils ask promote and POST /api/ask/handles/{id}/promote. A retired cohort keeps its members and its history and leaves the summary and the query’s lists.

The vocabulary is loaded with nils clinical vocabulary load, and clinical rows are imported with nils clinical import, which previews what would change and then applies it under your name, as Clinical does.