Pseudonymise a dataset

The registry never points at an identified file. Data that arrives identified is pseudonymised into a tree of its own first, and only that tree is digested. This page is the engine’s side of it; the desk’s is the Data page.

Declaring a dataset

A source place is a root: a folder whose folders are datasets. nils setup adds the root alone. A folder under it becomes a dataset only when someone adds it, and only then is its structure read. How a dataset’s files arrive is never declared; the folder says it.

Under the dataset’s folder	State	What reads it
only derivatives/dcm-original	identified	the pseudonymiser, which writes derivatives/dcm-anon
only derivatives/dcm-anon (or the earlier NILS’s dcm-raw, renamed dcm-anon)	anonymised	the registry: every digest
both	identified, with its anonymised copy	the registry reads dcm-anon; the pseudonymiser reads dcm-original
anything else: DICOM beside derivatives/, or no tree at all	unknown	nothing

A folder that names a dataset’s derivatives/dcm-raw or dcm-anon itself is read as that tree, a legacy dataset. The registry never points at an identified file: a digest of the originals is refused, and nothing of an unknown dataset, a root, or a dcm-original is read or pseudonymised until it says what it is. The walk of a digest never goes into a derivatives/dcm-original below the folder it reads.

nils place folders source --search scans    # a page of the root's folders, found by name
nils place folder source scans              # look at one before it is added; nothing is written
nils place add-dataset source scans --identity scans-identity.yml --unmapped hold --cohort study-a

folders lists fifty a page (200 at most; --after NAME takes the next page) in one read of the root’s listing, and looks inside none of them. folder answers whether the folder holds DICOM and derivatives/, and the state it would have. add-dataset adds it under the folder’s name, or --name.

An unknown dataset’s entries holding DICOM go into the tree the person names, and only on the person’s word. Without it the answer says what would move, and nothing is written:

nils place set 7 --move-into originals --confirm-move   # identified data
nils place set 7 --move-into anon --confirm-move        # already anonymised

An anonymised dataset also says what its PatientID holds and how its subjects are found, before anything reads it:

nils place set 7 --patient-id id-type:study-id --subjects map
nils place set 7 --patient-id subject-code --subjects generated

Flag	What it declares
--move-into originals|anon, --confirm-move	where an unknown dataset’s loose entries go, and the person’s word that they may move
--patient-id subject-code|id-type:NAME	what PatientID holds in the pseudonymised tree: the subject’s code, or its value of an ID type, whose files wait until the subject has one. Changed only before anything is pseudonymised
--subjects map|generated	how an anonymised dataset’s subjects are found, which it must say: a map of subject codes to its IDs, given or already in the registry (a file whose ID no map names is held), or the subject code generator making each code from the ID
--copy-folder subject-code|id-type	what names each copy’s folder: the code, or the value PatientID holds
--identity FILE, --no-identity	the identity rule an identified dataset’s files are read under, as nils digest --identity-rule reads one
--unmapped hold|code	what a file whose identifier the linkage store does not know does: hold, or code it anyway
--cohort NAME, --no-cohort	the cohort every read of the dataset feeds
--picks after_sort|off	whether picking main scans follows a sort of the dataset
--keep-demographics, --no-keep-demographics	keep sex, weight and size when pseudonymising; kept unless said
--remove TAG, --keep TAG	a tag to remove beside the groups always removed, or to keep out of them, as gggg,eeee; both repeat, and a tag on both is refused

nils place set takes the same flags, and a flag not given keeps what is in force. nils place explore reads each dataset’s structure again and adds nothing; nils place layout NAME shows what a folder holds, and for each way its files may arrive the tree that is read and what a confirmed declaration would move there. A rule file is the identity block a digest reads:

identity:
  id_type: study-id
  from:
    - field: PatientID

Declaring renames a derivatives/dcm-raw to derivatives/dcm-anon, shown, and makes no other write under a source outside the pseudonymiser. Every move is a rename inside the folder, top-level entries only; nothing is copied and nothing is read. A folder that is a tree of another dataset is refused.

Wherever a location is named, @name is the dataset’s pseudonymised tree and @name/originals its originals, which a digest is refused. nils place list shows each dataset’s state and trees, and nils place bindings where each name points. On the door, a dataset is added with POST /api/places and {role: "source", root, folder}, and its fields go on POST /api/places and PUT /api/places/{id} under data:work beside places:work; the answer carries layout, what was found. GET /api/places/{id}/folders?q=&limit=&after= and GET /api/places/{id}/folders/{name} are folders and folder, POST /api/places/explore is explore, and POST /api/ingest/look answers layout for a path before anything is declared, naming no location in the answer. The desk’s side is Add a dataset.

The verb

nils pseudonymize @scans
nils pseudonymize @scans --dry-run
nils pseudonymize @scans --held

Only an identified dataset with its originals can be pseudonymised; any other dataset’s tree is read in place, and the verb says so. For every file of the originals it reads the header, resolves who the file is about through the dataset’s identity rule and the linkage store exactly as a digest does, and writes a copy into the pseudonymised tree, its pixel bytes streamed from the source untouched.

In the copy
PatientID	the subject’s code, or its value of the ID type the dataset chose
the patient group, 34 tags	removed, PatientName and the birth date among them
the trial group, 23 tags	removed
the provider group, 38 tags	removed
the institution group, 5 tags	removed
the IDs group, 47 tags	removed
accession number and study ID	removed from every file, whatever a list says
sex, weight and size	kept, unless the dataset says --no-keep-demographics
the scanner’s identity, and what the rules read from the copy	kept, the rules’ reading cleaned of the file’s own IDs, accession number, birth date and the words of its names
PatientAge	written, computed before the birth date goes
dates, times and UIDs	kept, so the same study brought in twice is one study, and a release applies the date policy
private tags	dropped, except what the pack’s release list names by creator and element
overlays and curves	dropped
the dataset’s own lists	its --remove tags removed and its --keep tags kept, on top; a tag on both lists is refused

Every file the pseudonymiser writes says so in its own header: PatientIdentityRemoved (0012,0062) is YES, DeidentificationMethod (0012,0063) names NILS, its version and the writer, DeidentificationMethodCodeSequence (0012,0064) has one item for each option the writer’s policy applies (DICOM PS3.16, CID 7050), and LongitudinalTemporalInformationModified (0028,0303) is UNMODIFIED. The option list is derived from the policy, never written by hand.

Four elements are never removed, whatever the dataset’s lists say: PatientID, which becomes the code; PatientAge, which is written; and SOPClassUID and SOPInstanceUID, the two that make a file a file. The engine serves the whole list itself, and the desk shows it with names; see the tag list and pseudonymise on the desk.

The copy goes where the file’s own facts put it, <code>/<study date>-<eight hex of the study UID>/<series number>-<description>/<instance number>.dcm, the numbers padded so a listing sorts and a clash taking a counter, and never under the original path, which is the sender’s. The description is the copy’s, cleaned of the file’s own identifiers, reduced to letters, digits and hyphens and cut to 40 characters; the number alone where it is empty or the dataset removed it. A tree begun with numbered folders keeps them, and an existing tree is never renamed.

Flag	What it does
--name N	the batch’s label; the dataset’s name and today’s date unless given
--pack PACK, --pack-dir DIR	the pack whose release list says which private elements are kept; mri unless given, and a pack asked for by name that is not found is an error
--held	read only the held files a map released or a person asked to code anyway
--dry-run	walk, read and resolve everything, print the report, write nothing
--workers N, --walk-threads N, --batch-rows N	as a digest’s
--json	the report as one JSON document, progress as JSON lines

A file the reader refuses, not DICOM or missing a UID, is refused with the same classes as a digest refuses it. An original the pseudonymised tree holds already is left as it is rather than written a second time, and the report says how many. The verb needs data:work at detail sensitive on the door, since it reads the identifiers it replaces.

The tag list

The policy above is the engine’s, so the engine states it. GET /api/pseudonymize/tags answers what the pseudonymiser does to the standard elements it acts on, and nothing that shows the list has to transcribe it:

In the answer
tags, count, categories	the 145 elements of the five categories (patient 34, trial 23, provider 38, institution 5, ids 47), in tag order, each written gggg,eeee as a dataset’s own lists are written, with its category, its fate, what becomes of it, and its reason; why says more than the plain removal
code	the element the subject’s code is written into, replaced rather than removed and in no category, so that it is never offered to keep
mandatory	the two that make a file a file, never removed whatever a list says, each with why
examination	the accession number and the study ID, removed from every file whatever a category or a dataset’s list says
covariates	sex, weight and size, kept unless the dataset opts out, with the name of the switch that opts out
fates, fate_meanings, reasons	the three words a fate can be and what each means, and the reasons a row has its fate
marks	the de-identification marks every file carries, for the pseudonymiser and for a release
version	the engine’s version, for a reader to key a cache to

A fate is one of three words: removed; replaced, for the age, computed before the birth date goes, and for the element the code is written into; or kept. A reason is category (removed because its category is removed), computed, covariate, code, mandatory, examination, rules (kept because the rules read it from the copy, cleaned of the file’s own identifiers) or device (the scanner’s identity, kept and claimed as the Retain Device Identity Option). The times, which are a release’s rather than the pseudonymiser’s, are not in it.

The door serves no tag names. The engine holds none, and the words a reader needs belong to whatever shows the list. A dataset’s own keep and remove lists are that dataset’s, and are served with the dataset on the sources and places doors. Any grant opens the door, which reads the constants of the binary and nothing of the registry.

Reading a tree that already holds codes

A digest of a pseudonymised tree reads the codes in it verbatim, under the subject-code type: any value of the code alphabet, whatever its length. A subject the pseudonymiser made is found by its code and never coded again, and so is a subject whose code was made elsewhere. A v0 cohort folder’s sixteen hex characters are read as the codes they are, under a registry that derives twelve among them: with v0’s map filed, a person keeps the code they had and is one subject, not two.

A value no subject holds stands as its own code where it has the shape this registry makes, its scheme’s alphabet and its display length. Any other value is no code of this registry, so a code is derived from it as from any identifier, and a value outside the alphabet is no identifier the rule can read at all: the study UID answers instead.

A twelve-digit personnummer is never read as a code, though the code alphabet spells its digits. Such a value does not answer the rule (an identity_unparsed diagnostic, with the shape and no value), and in a dataset’s tree the file is held, so a number never becomes a subject’s code or a sub- name in a release.

Held files

A file whose identifier the linkage store does not know does what the dataset says:

--unmapped	What happens
hold	the file is not written; it is recorded with the identifier’s shape, digits as 9 and letters as A, and a keyed lookup, counted on the batch and the dataset, and raised as one review item per dataset and shape (identity.unmapped)
code	the file gets a code derived from the identifier under the key. The subject is a subject like any other: the person chose it, the same ID gets the same code wherever it is read, and no question is raised. A map that later names the ID under another subject is refused as a conflict, which a merge answers

A map that names a held identifier releases its files, and POST /api/linkage/held/code marks a dataset’s held files to be coded anyway; nils pseudonymize @scans --held then reads those files and nothing else, and the next bring-in writes them. The identifier itself never leaves the process and is never written in the clear; GET /api/linkage/held?place= lists the shapes with counts, and revealing the identifiers is a door of its own. See identifiers and the map.

An anonymised dataset has no pseudonymise step, so its digest answers unmapped itself. It must say what its PatientID holds and how its subjects are found. Where it reads its subjects from a map (--subjects map), a file whose identifier the linkage store does not know is not filed: its row is quarantined under identity.unmapped with the shape of the identifier, no subject and no row of any other table is made for it, and one review item per dataset and shape says how many, opened and closed as the pseudonymiser’s are. A held file is read again by the next digest whatever it was asked, so a map filed since releases it without anyone asking for the quarantine to be retried. Where the subject code generator makes the codes (--subjects generated), every file is a subject. A file whose PatientID holds a value its declaration cannot read is held in the same way, and the next read files it once the declaration is corrected. The doors above read the pseudonymiser’s held files, so for such a dataset the way out is the map, or --subjects generated and a read again.

Resume and stopping

A run resumes by size and modification time, asking once per directory what an earlier run recorded. A file written before whose source is unchanged is checked for its output and left alone; an original the tree holds already is left as it is, so a v0 cohort folder’s files are not written a second time beside v0’s own; a held file is read again only once a map released it or a person asked for it to be coded anyway; a refused file stays refused until it changes; a changed file is read again and its old output let go. Every file is written as .part and renamed into place, hashed while it is written, and the tree is synced once at the end of the run, never once per file.

A stop travels through one signal: the walker lets its queue go, the workers finish the file in hand, the run records what was written, and the batch and the job end as cancelled. A written file stays written, and running again goes on from there, the way a stopped digest does. A run refused because another job holds the registry says which job, and since when.

The report

nils pseudonymize   name scans-2026-09-16   dataset scans
  originals        /data/example/scans/derivatives/dcm-original
  tree             /data/example/scans/derivatives/dcm-anon
  files            12,400 seen   12,380 written   0 unchanged   20 held   0 refused   0 skipped   0 walk errors
  subjects         310 seen   310 new   0 provisional
  removed          620,000 elements over 41 tags   37,140 private
  held by shape    AAA999 20
  in the tree      40 original(s) the tree held already, left as they are
  run              1.6 GB in 4 s   3,100 files/s   workers 8
  batch            17

Nothing in the report comes out of a file but tag numbers, refusal classes and the shapes of held identifiers. The run is a batch of kind pseudonymize, and the report is the job’s result, a dry run queued at the door included.

What becomes of the originals

The acts that do it:

nils place originals scans
nils place originals scans --vault --into archive --why "the study is closed"
nils place originals scans --purge --why "every original is in the tree"

With neither flag the verb only says what an act would do, which is what GET /api/places/{id}/originals answers under data:see:

originals of scans   kept
  files            12,400   1.6 GB
  verified         12,380 of 12,400 in the pseudonymised tree
  held             20 for want of a map
  purge            is refused: 20 files of the dataset scans are held for want of a map

Verified means the copy the pseudonymiser recorded is in the tree, at the size that was recorded, and its digest now is the digest that was recorded, hashed again for the answer rather than read off the row. So the survey costs what a purge costs, which is the point of it: nothing is destroyed on the strength of a row.

A vault moves the originals into the backup place --into names, the one role the engine never reads for data, under originals/<dataset> there, so that a backup place’s archives and a dataset’s originals never share a folder. A file moves by a rename where the destination is on the same filesystem, and otherwise by a copy whose digest is compared with the source’s before the source is removed, so no file is ever removed whose copy did not verify.

A purge deletes them, and is refused three ways:

Refused	Why
while any file of the dataset waits for a map	a held file’s original is the only copy of that person’s scans; file a map that names the identifiers, or code them anyway, then look again
unless every original is verified in the pseudonymised tree	checked again as the job runs rather than trusted from the rows; pseudonymise the dataset again, then look again
for a dataset read in place	its folder is its own pseudonymised tree, so there is nothing to verify the originals against

Each refusal names its count, and the door refuses in the same words the job does.

Both acts are jobs of kind originals, queued at POST /api/places/{id}/originals {do, into, why} under data:work at detail sensitive, since they move and delete identified files. Both resume, a file that has moved being simply not there the next time, and both stop at a heartbeat on a cancel with what was done kept; the result is {did, files, bytes, into, path, verified, seconds}. On success the dataset records what became of them, and a vault the place it used (originals_kept and originals_vault); until then it reads kept. Each act is audited, originals.vault or originals.purge, with its counts and the reason, which moves the epoch.

Neither act touches the pseudonymised tree, the registry’s rows or the linkage store: a person keeps their code and their history whatever becomes of the originals.

The chain

nils bring-in @scans
nils bring-in @scans --name 2026-09-16 --pack mri

bring-in queues the thread of a dataset as a chain: pseudonymize @scans --name N, then digest @scans --name N, then fingerprint, then classify, then a pick run where the dataset feeds a cohort, the pack declares picks and picking after a sort is not off, and last score, which queues System 1’s run over the batch’s new stacks where its model is chosen and this machine can run it. Each is queued when the one before it ends done. The pseudonymise step and the digest share the name, so the two batches are one thread. An anonymised dataset has no first step and starts with the digest. A step that cannot run is skipped and the result says why; a bring-in never fails because of it. A read that adds, changes and removes no file and changes no stack ends the chain before the sort, and its result says so (chain_ended), so a dataset read again with nothing new does not sort and pick the whole registry again. A worker runs them in turn: nils jobs work, or nils serve --worker.

Where an identified dataset’s pseudonymised tree holds files no digest has read, a v0 cohort folder on its first bring-in, a digest of the tree goes first, so the pseudonymiser knows what the tree holds and leaves those originals as they are. The answer and the job’s args.digest_first say how many files and why.

On the door, POST /api/jobs takes bring-in as a command, and any command may carry then, the command lines queued after it, each a list of words or one string:

{ "command": ["digest", "@scans", "--name", "spring"],
  "then": [["fingerprint"], ["classify", "--pack", "mri"]] }

Each step is queued under the principal, grants and detail of the person who queued the first. Its grant is checked when its turn comes; a step the person may not queue stops the chain, and the job’s result says which step and why, as chain_stopped. GET /api/jobs/{id} shows a job’s then and its chain, the job before and after it. bring-in is checked at the door by its first step, and pseudonymize needs data:work at detail sensitive.

A chain queued at the keyboard, whole with nils bring-in or a step at a time with nils jobs enqueue, records the keyboard’s reach: every grant at every detail, and sealed:see only where --unsealed-access gave a reason. So its score runs System 1. A job whose command line names no pack directory reads the packs the server serves.

The score at arrival

The score runs System 1 alone, on the pipeline lane, so it never holds up the next digest. A batch of more new stacks than one container of System 1 holds is queued in parts (--chunk), each part a run of its own in the one job: 1,250 stacks for each GB of memory the pipeline’s descriptor declares a unit needs, which is 5,000 for System 1. A pipeline whose units run apart is never split. A dataset’s step door and System 1’s run over the archive are queued the same way.

A System 1 model that brings its own code runs only where the run says it trusts that code (trust_code), and then only after the model’s files match the digest recorded for it. The score at arrival passes the parameters the pipeline’s descriptor declares for a run at arrival (x-nils.on-arrival: {params: {...}}). A site that trusts the model it chose for System 1 registers its System 1 pipeline with trust_code: "true" there; without it, the run at arrival stops at that check and says so, and the score can be run with --param trust_code=true by hand.

Body part and post-contrast never run at arrival. They are optional models a person runs where the data need them, from the dataset’s step (POST /api/datasets/{name}/steps/{step}/run), and the score’s result names them under optional. Each step’s hint on the dataset’s summary says how many scans its model would add an answer to. At the keyboard, --step queues one over a batch’s new stacks:

nils score @scans --name 2026-09-16 --step body_part
nils score @scans --name 2026-09-16 --step body_part --step post_contrast

With both, post-contrast is queued once the body part’s run ends done.

Pictures after the sort

A sort’s job ends done as soon as it has judged its scans and read their headers, so the pick and the score start at once. The worker that ran it then queues the previews of the scans it judged on the pictures lane (preview build --classified JOB), and their 3D views after them, under the sort’s own principal, grants and detail. A sort run at the keyboard queues its previews itself and says which job; --no-previews queues none.

A build over many scans makes several at once, each with its share of the workers: a quarter of the workers’ worth of scans for the previews (one to sixteen) and an eighth for the 3D views (one to eight), fewer where the scans are large. --at-once N, from 1 to 64, says how many, and --at-once 1 builds one after another:

nils preview build --dataset scans --at-once 4
nils pyramid build --select selection:NAME@V --at-once 4

A stop waits for the scans being built and keeps them. A picture a reader asks for while the 3D views are built is made between two scans, never beside a build writing the same scan.

The batch is the thread

GET /api/batches/{id} reports a batch’s five stages with their counts and jobs: pseudonymised (files, changed, held, refused), walked (files, new, changed, unchanged, refused), digested (stacks, sessions, subjects, moved), classified (stacks, of, unsure, pack, jobs) and reviewed (done, of, since). GET /api/timeline/batch/{id} is its timeline, and GET /api/sources shows each dataset’s trees with their files and bytes, what it holds for want of a map, the subjects whose files its tree holds whoever made them, each recent digest’s pseudonymised step with the jobs of its thread by stage under chain, and the rate this machine pseudonymised and digested at. A rate is taken from the last run of that step that read or wrote at least a hundred files, else the largest run there was, and carries the files it was measured over, since a resume run of ten files times the cost of starting a run rather than the machine.