Identifiers and the map

A subject is one person, under however many identifiers of however many types, from however many datasets. The identifiers live in the linkage store beside the registry, sealed under the registry’s key, and the registry itself holds only codes. This page is how identifiers are filed and read; pseudonymise a dataset is what reads them.

Identifier types

A type is the name an identifier is filed under. Five exist always: personnummer, an ID that is the same everywhere and leads to a person, reserve-number, the number a hospital gives a person whose personnummer it does not have, patient-id, study-instance-uid and subject-code, the last being what a merge files an alias’s code under, and what a digest of a pseudonymised tree reads the codes in that tree under. A code is read verbatim, any value of the code alphabet whatever its length, so a person a map named keeps the code they had; see pseudonymise a dataset. The rest are the site’s own, made from the list or on the fly, named in lower case letters, digits and hyphens.

nils linkage types
nils linkage id-type add study-id --description "the number a study gives a participant"

nils linkage types prints each type with how many identifiers and subjects it holds, and never a value. The doors are GET /api/linkage/types and POST /api/linkage/types {name, description}, which makes a type once by name.

A type says whether its IDs lead to a person wherever they are met (identifying): personnummer, reserve-number, patient-id and study-instance-uid do, and a hospital’s record number does when you say so. A study’s own ID does not.

nils linkage id-type add record-number --description "the hospital's record number" --identifying

Changing the flag on a type that exists, either way, needs data:work at detail sensitive, taking it away is asked for with unflag: true, a built-in type keeps it, and every change is audited as linkage.type.flag.

A personnummer is never kept

A value of type personnummer is read as its twelve digits before its code or its lookup is derived: separators and spaces are dropped, a ten digit number gets its century (a + meaning a hundred years or more), and a coordination number keeps its day as written. So one person gets one code however a source wrote the number. A value that is no personnummer (another shape, a date that is no date, a wrong check digit) does not answer the identity rule, which goes on to its next source, and refuses its row in a map import, naming the column and never the value.

NILS keeps the number nowhere, not in the registry, the linkage store or the audit, not even sealed: its identity is its keyed lookup alone. A held file’s row keeps its shape and lookup, and a map with a personnummer column is imported from the request itself, never written to a file for a job. Where a dataset’s identity rule files personnummer, the subject code generator codes every valid number, and its subject is a subject like any other.

Nor is a reserve number

A hospital’s reserve number is kept as a personnummer is: its identity is its keyed lookup alone, with nothing sealed in the linkage store or on a held file’s row. A map takes it as identifier:reserve-number, beside the person’s code or another identifier of theirs, in one written form (spaces and one separator dropped). A canonical:reserve-number column is refused, since no code is ever made from a reserve number, and no dataset writes one into PatientID.

Where a dataset’s rule reads PatientID as an ID that is the same everywhere, a PatientID that is no personnummer is read as a reserve number. A file whose number a map names is that person’s, so one person’s personnummer and reserve number land on one subject. A file whose number no map names is held under the number’s keyed lookup and shown by its shape, and a map row that names the number releases it. Coding such a file anyway leaves it held.

From an ID to its subject code

POST /api/linkage/resolve {id_type, value} turns one ID of a known type into the subject code it stands for with one lookup: {id_type, known: true, subject_code, scans, datasets}, or {id_type, known: false}. The value is looked up by its keyed hash as a digest and a map filed it, so no stored identifier is opened and the answer never holds the value back. A merged subject answers the subject it was merged into. Asked with subject-code, it answers whether some subject holds that code, whatever type the code was made from. A study’s own type is open to whoever may use the data (data:see or query:see) at detail quasi, since a subject code is quasi-identifying; a type flagged as leading to a person needs data:work at detail sensitive. Every lookup, found or not, writes an audit row.

The map

A map is a CSV with a header row, then one subject per row, whose columns are named for what they are:

Role	The column is
identifier:<type>	an identifier of that type, filed on the row’s subject
canonical:<type>	the identifier that stands for the person: the subject’s code derives from it under the registry’s scheme and key, exactly as a digest derives it
code	the subject’s code itself
ignore	read past

nils linkage import map.csv \
  --column "registry number=canonical:registry-id" \
  --column "study number=identifier:study-id" \
  --column "note=ignore" \
  --dry-run

Several identifier columns in one row are several identifiers of one subject, and an empty cell is skipped. A row resolves to one subject by its code, by its canonical identifier, or by an identifier already known, and adds the rest. The first shape, one identifier column and one code column, still works as --id-type, --id-column and --code-column, and a file of PatientID,subject_code imports as it is.

Flag	What it does
--column HEADER=ROLE	a column’s role, as above; repeats
--dry-run	print the report and write nothing
--make-types	make the types the columns name that the store has not got; without it, such a type is a conflict
--place NAME	the dataset the map is for; recorded on the job, never a filter
--json	the report as one JSON document

The map is validated whole and then applied, and it reports what it will do before anything is written:

The report
subjects	named, known and new
identifiers	filed, known and new, and the types made
held files released	files held whose identifier the map names, with how many each identifier released and under which type (held_released_by); a value held under one type and named in the map under another is released all the same. The next pseudonymize --held or bring-in writes them, and on a dataset read in place the next digest reads them
merges	rows that map an identifier of one existing subject to the canonical identifier of another: the first is merged into the second, and the report says so first
conflicts	an identifier filed on another subject than the row resolves to, two rows disagreeing, or a type the map does not make; each with its line, and nothing is written while one stands

The report never carries an identifier: counts, codes and reasons only, and neither does the audit. The map applies as one transaction on each store, so a map refused halfway leaves no subject behind it. Filed once, the map holds for every dataset: from then on any of those identifiers arriving in any dataset lands on the same subject. The import runs as a linkage import job whose result is the report.

On the door, POST /api/linkage/imports takes the same map as JSON, {place?, columns: [{header, role, id_type?}], rows, dry_run, make_types}. A dry run answers the report at once; otherwise the engine writes the map under the registry home, readable by itself alone and removed when the job ends, and queues the job. More than 100,000 rows in one call are refused; a larger map goes in as a file under a registered location, queued through POST /api/jobs.

A later merge

Two subjects found to be one person are merged, the alias into the canonical:

nils linkage merge <canonical code> <alias code> --why "the same person: two study numbers"

In one transaction every row of the alias that carries a subject is re-pointed to the canonical one: studies, series, fingerprints, memberships, clinical rows, picks, hand-over subjects, the members of handles and lists, and decisions at subject scope. A membership of the alias that duplicates an open one of the canonical is closed as left by the merge; the alias’s identities move; the alias’s code is filed on the canonical subject as an identifier of type subject-code; the alias row stays, marked merged, and leaves every listing and count; its provisional review item closes; and subject.merge is recorded in the audit with both codes, which moves the epoch. nils linkage link records the same-person link as before and then merges the same way. The door is POST /api/linkage/merge {canonical, alias, why}, a linkage merge job.

The held doors

A file whose identifier the linkage store does not know is held when its dataset says so, and the identifier’s shape is recorded, digits as 9 and letters as A. The pseudonymiser holds the files it would write; a dataset read in place has no pseudonymise step, so its digest holds them instead, and the doors below read the pseudonymiser’s own.

Door	Answers	Needs
GET /api/linkage/held?place=	the held files of a dataset by shape and type, with the count, when the first was seen and the batch; never a value	data:see
GET /api/linkage/held/ids?place=	the held IDs one row each, by shape: its type, its files, when it was first seen and how it stands (held, with no code yet; mapped, with the code a map gave it; generated, to be coded at the next run; or waits, for a value of the type PatientID gets), and the subjects the pseudonymised files carry; never a value	data:see
POST /api/linkage/held/code {place, ids?, run?}	marks the held IDs of the dataset (all, or those named) to be coded anyway: a code derived from the identifier under the key, final, the same wherever that ID is read. With run it queues the pseudonymise (or, for a dataset read in place, the digest) that writes them, and answers with the job	data:work
POST /api/linkage/held/reveal {place}	the identifiers themselves, by shape, each with its files; one read-audit row per identifier revealed, and one audit row with the count	data:work, detail sensitive

A dataset read in place holds its files as quarantined rows rather than in the table these doors read, so they answer nothing for it: the review item says how many, the map is the way out, and the next digest reads them again. See held files.

The review items are identity.unmapped, one per dataset and shape, brought up to date by each run and closed by a run that holds nothing under the shape, identity.provisional, one per subject an earlier engine coded without a map (a subject coded now is final and raises none), closed by the merge that folds it in, and identity.same_instance, one per source and pair of subjects, for a file whose instance UID the registry holds under another subject, study or series. A merge of the two subjects answers it. Where one subject holds the scan under another visit or series, a person keeps the file as another copy of the scan or leaves it out of the read; nothing is deleted.

One subject’s identifiers

What nils linkage show, link, unlink and purge do at the keyboard are doors as well, an admin’s. Each is a person’s act, and each is audited.

Door	What it does
POST /api/linkage/show {subject}	the subject’s identifiers by type, where each came from and whether the store keeps it, with no identifier opened, and the linkages that name the subject. With values: true and a why, the kept values, one read audit row each; a value that reads as a personnummer is answered as its shape alone (withheld)
POST /api/linkage/links {a, b, evidence}	records two subjects as one person, and queues the merge of b into a
POST /api/linkage/links/{id}/unlink	reverses a linkage
POST /api/linkage/purge	deletes the identifiers and linkages of one subject, or of every subject, only with the subject’s code (or every subject) typed in confirm; dry_run answers what would go

No personnummer and no reserve number is kept, so neither can be shown. The keyboard’s purge and the door’s are one function. The desk’s side is one subject’s IDs.

What needs detail sensitive

Reading an identifier needs data:work and the sensitive detail; what holds no identifier needs less. A job keeps the detail of the person who queued it.

Act	Needs
a subject’s identifiers shown, a link, an unlink, a purge	data:work and install:work, detail sensitive
applying the map, the reveal, the merge, pseudonymize	data:work, detail sensitive
the map’s dry run and its report, coding held files anyway, making a type	data:work
the held files as shapes and counts, the types with their counts	data:see

Groups and grants says who holds which detail.