The Review page · NILS documentation

One list of work: the questions the rules leave open, main scans, subjects, the rules, the rounds, and the campaigns people read, with their results.

Review is one section with six pages in the side: Work, Main scans, Subjects, Rules, Rounds and Label sets. It opens for a person with see on Review or on Campaigns. A rater who holds only campaigns sees Work, with the campaigns they read, and Label sets; the page never reads the queue for them. Deciding, proposing and adopting need work on Review; making and closing a campaign need work on Campaigns.

The rules decide every answer. A person’s work here is to answer the few questions the rules leave, to fix a rule instead of a scan, and to read what people and models disagree on.

Work

Work is the one list. Each row has a count and one button.

Kind of work	What a person does	Where
1. Answer a question	an axis, several axes, a form or a pair	a question’s grid, or the rate reader
2. Pick a main scan	choose the scan that stands for a role	Main scans
3. Check a result	accept, reject or flag a pipeline’s output	the check reader
4. Draw or correct a mask	edit a segmentation	not built yet
5. Settle a disagreement	choose between two answers	adjudication, or the A/B reader
Audit	read accepted scans blind; it never trains	shown only while a round’s audit is open

Work lists the kinds that are built, each as a heading with its rows under it.

The machine’s questions come first under each kind, with how many wait. The campaigns follow, each with its items by state and your own share (“34 of 300 yours”).
A question a campaign holds is counted once (“219 questions are held by campaigns, answered there”) and is answered there, never in the queue.
Closed campaigns fold into one line and keep every answer.

Answer the questions the rules leave

On Work, press Answer on a row of the machine’s questions.
The questions page lists each question in plain words, with its distinct scans and why it is asked: “Which base? 7 scans · The rules found no base for these scans.” Choose one.
The grid opens: a page of fifty scans, each picture framed in the colour of NILS’s answer. System 1’s answer is a second small mark, and the scans it agrees on step back. A staged answer is marked.
Press 1 or 2 to give one of the question’s two answers to the page, 0 for the axis’s whole list, - for no value.
Press space on a scan to set it apart. It takes the other key’s answer, or one of its own.
Press Enter. The page is answered in one move, with the scans set apart as exceptions, and the scans are sorted again at once. The next page opens, then the next question.

f opens a scan full size with its slices. ? shows each scan’s rule and the rules it was decided over. The arrows move. There is no dialog, no scope and no why: a decision is the scan’s own and outranks the rules until a newer rules version moves their answer from it.

A question drawn blind first shows none of NILS’s answers. Answer from the scan and its header; the answer posts as provisional, then both sides show and you keep or change it. The share of questions read blind first is set in Settings, Backups. Only blind answers train a model.

The same keys work in the dataset viewer: a scan that needs a look shows its question under it, and 1, 2, 0, - and Enter answer it alone.

Also waiting

The questions page (#review/questions) also holds what the grid does not take.

Page	What waits there
Picks	the occasions a pick run doubts, opened on a session board; a person’s pick, with a required why, stands through every later pick run. Keep on a border makes the run’s pick the person’s.
Proposals	what models proposed, from the value held now to the value proposed. Each group is committed or withdrawn alone; a model’s answer is never in force until a person commits it.
Asked	the legal candidates for a whole stack, most probable first

Each shows its count, zero too, and a line saying what would fill it.

Main scans

A main scan is the scan that stands for a role in a visit (T1w, FLAIR, T2w). Each dataset and each cohort has its own rules for choosing it, in versions.

Open Main scans and choose a dataset or a cohort. The button beside the title changes it.
Choose a role. Five numbers show how it stands: visits that have one, subjects that have one at some visit, how alike the picks are across the data and within each subject, and the series complete at every visit.

Choose Keep alike:

Choice	The pick
Within each subject	each subject keeps the kind it has at most visits
Across the data	every visit takes the first kind it has, in the rules’ order
Balanced	the data’s order first; a subject keeps one kind where one covers all its visits

Press Rules · version N to open the panel. It shows the effect first (“14 visits change their pick”), then the kinds in the order they are taken (each can be used or left out, and moved), what is allowed (contrast, 2D or 3D, body part, slice thickness), and how two scans of one kind in a visit are told apart.
Every change is a draft in the browser. The page asks the engine again at each change and shows the effect before anything is saved.
Give a why and press Save as version N. The saved rules queue the scope’s pick run, and one line follows it. Back to version N restores a version. The rules are also shown As text. Saving needs work on Pipelines.

The map has the kinds, where they come from (by scanner or dataset), the series, and the visits one by one. A visit opens its pick:

Press a visit, choose another scan and give a why. A person’s pick stands through every later pick run until it is withdrawn.
A Layer (Body region, Body part or Post-contrast) colours each visit by what its scans are. A line says whether the step that answers the layer has run over the scope, with a Run where it has not.

Where a scan is copied into two datasets, a line says how many visits are counted in each. A dataset’s Main scans step on Data runs the same picks.

Subjects

Subjects lists the identity questions by kind. Each says what settles it.

Question	What settles it
Two codes share one identifier	Merge, or Decide that they are two
Files held until the map names their identifier	Map them, on the dataset’s Pseudonymise step. Nothing is decided here.
Subjects an earlier engine coded without a map	Merge into the subject they stand for, or a map that names the identifier. A subject coded now is final and raises no question
The same scan filed twice	Merge where two subjects hold it. Where one subject holds it under another visit or series: Keep as another copy, or Leave out of the read

Merge two subjects takes the canonical subject, the alias and why. Every row of the alias moves to the canonical subject, the alias’s code is filed on it as an identifier, and the alias leaves every list. Merging and mapping read identifiers and need work on Data and everything in records; identifiers and the map says what moves. Leave out of the read deletes nothing: NILS reads the files again only when one of them changes.

Rules

Rules is where an error is fixed in the rules, so that one fix mends every scan it touches. It has four tabs, each in the address (#review/rules/proposals, quality, change, words). It opens on Proposals while one waits for a person, else on Quality. ?dataset=NAME narrows it to one dataset.

Proposals

Each proposed rule change is one approval card.

Read the effect line: “Moves 50 scans · base T2w → FLAIR, 47 · fixes 47, breaks 3”.
Read where it lands: the pack’s next version for everyone, or one site, dataset, scanner or batch.
Read every answer it breaks, with its scans in the caution colour’s frame. They are shown before any button.
Press Adopt, or Reject (it asks why). Details opens the stored effect report; Change it puts a one-operation proposal back in the form.
After Adopt, one line follows the re-sort until it ends and says what it did: the stored answers per axis, before and after.

Adopting needs work on Review and on Data, and is a person’s act: the assistant proposes and never adopts. A proposal is a pack edit or a scoped change, not both. Adopting a pack edit writes the pack’s next version, keeps the one it replaces whole beside it, and sorts the registry again; the version is then released as a rules release. Adopting a scoped change sorts its scope alone. A proposal rehearsed on an older pack, or before a change in force, says “rehearse again” and offers only that. A proposal that a later version of its overlay replaced says so and offers nothing.

Quality

The pairs of rules the pack cannot rank, the overrides by rule pair, and System 1’s disagreements (said as not served yet until it is). Each row has its count, pictures and the change that would mend it.

Change a rule

Choose an operation. The form is built from the patch contract’s schema, one for each typed operation:

Family	Operations
Words	add words, remove words
Order	move a rule, move a rule set, change a priority
Values	add a value, change a name
Rules	add a rule, silence an axis, hand an axis to its model
Picks	who competes for a role, raise a border no more, decide near ties, near-tie margin

Say where it applies, and why.
Press Show the effect. The registry is replayed with the change and nothing is written. The report shows the moves per axis with their datasets, sites, makes and pictures; the answers we know (right to wrong, wrong to right, the net); the names; the main scans by role; Review’s questions; and the pack’s own cases.
Press Propose it. It appears on Proposals.

Words

Words shows every axis the pack declares, with its values and the words that reach each. Not every axis is reached by a word, and the page says so. A word the site added is drawn apart from one the pack shipped.

Choose an axis and a value, and press Add a word.
Try it: the engine rehearses the word as an overlay on real stacks and shows what moves. Trying writes nothing.
Propose it (needs work on Review). The proposal waits on Proposals with its effect.
Adopt it there.

A word is matched against the pack’s normalised text of the series description and the protocol and sequence names. What stays the pack’s, as a new pack version: the flags, the physics windows, the review thresholds and the order the values are tried in.

Rules releases

The MRI rules ship on their own tag. Settings, Parts has a row for them, with an Update that takes the rules alone and starts nothing again. Stacks keep their old values until they are sorted again with the new pack: press Sort again on the row (Sort the scans again). Parts and keeping the engine current say more.

Rounds

A round is a rules version and a System 1 version. Rounds shows for each:

the share it accepted and the share it sent to a person, by kind;
that it sent nothing for a conflict, a low confidence, body part or post-contrast;
System 1’s coverage, and how many answers were given blind;
its audits: a seeded share of what it accepted, read blind by agent raters, with the accepted error and its exact bound.

Press Draw the audit. It makes the audit campaign and its items.
When the raters have read it and the campaign is closed, press Grade.

An audit campaign is said never to train, closes into no decision, and suggests nothing.

Make a campaign

On Work, press Make a campaign.
From: a saved selection, a result you kept, or Review items of an open kind. Each is chosen from a list; nothing is typed that a list can offer.

Question: choose the kind.

Kind	Work
One axis, Several axes	answer a question about each scan
Pick the stacks	pick a main scan for a role
A form, A file, Free text	answer a form, attach a file, write
Two stacks side by side	read pairs, such as a pre- and a post-contrast pair
A stack beside two anchors	read a candidate beside a reference pre and a reference post
Settle readings (A/B)	settle where independent voters split
Check results	check a pipeline’s result; opens New check

The values of an axis are keys from the pack. Choosing none asks every value.

Shown beside each item: nothing, the rules’ answer, or imported suggestions. With nothing, a read is blind: no answer is filled in and no evidence of the rules shows. Pictures only also hides the header text.
Its answers are for: Decide, Train, Measure or Audit. Train and Measure are read blind first, always.
Choose who rates and who adjudicates, from the people on the desk with work on Campaigns, and yourself.
Under Adjudication, close and lease, choose what a close writes: decisions in force, staged decisions, picks, a status on each result, or nothing.
For pairs, anchors and A/B, the dialog draws the items from the selections you chose and a seed, and its foot says what it would make (“Draws 118 pairs · 236 stacks · 6 left out.”) before anything is written. The campaign keeps the seed’s hash, never the seed.
Press Make. It opens for reading at once.

A rater reads only the campaigns they are named in. While a campaign is open a rater sees only their own answers. Making and listing raters need work on Campaigns; anyone who may make a campaign sees the names of the people who can rate it, and nothing more. The raters are named to the campaign’s maker and to a person with work on Review alone. Anyone else reads each other person as “rater 1”, “adjudicator 1” and so on, and their own answers under their own name.

Check a result

A check is a campaign whose item is one result of a pipeline, a mask, read over the scan it was made on and the visit’s main scans.

On Pipelines, Runs, open the run and press Check results. New check opens filled for that run. From Work, Make a campaign, Check results, choose the run first.
Choose which results (all, not yet checked, the units its declared checks flagged), the part set to look at (the whole result, each lesion, each structure), the scans beside, the reasons a rater may give, and who reads.
The foot says what the engine would make and what it leaves out. A result made from a scan of a sealed sample is left out first and counted, never named.
Press Start checking. The check reader opens.

In the reader, press 1 to accept, 2 to reject, 3 to 9 for a reason (the campaign’s own), Enter to send. A reason may ask for a point on a picture or a note (e). n and p bring the next and previous part into view on every panel, r the whole result, Backspace takes the last mark back. The mask keys are the viewer’s: o opacity, f fill, [ and ].

Two raters are compared on the verdict. A different verdict goes to the adjudicator, and an agreed flag is the union of their flags. A close writes a decision on the result and on each part judged, a person’s, and the status on the result. A rejected result is left out of a later run’s inputs unless its descriptor asks for any.

The readers

Open a campaign from Work with its button: Rate, Read, Settle, Adjudicate, Check. ? lists every key by what it does.

Rate

The reader fills in the answer the rules and the model agree on, and Enter confirms it. Where they differ, the legal candidates show with their probabilities. A choice fills in what the pack says it implies and greys what can no longer hold.
Every asked axis has a find box: type any name of a value (a vendor’s too: BRAVO finds MPRAGE). / finds a whole answer.
Can’t tell is never held to the pack’s implications and never becomes a decision. m or the Unsure button marks an answer for a second look; the adjudicator sees both.
Why the rules decided stands beside the picture, open: one line per asked axis with the value, the flags, the rule, the header values it read and the words it matched. A caution border marks an axis where the rules and System 1 differ.
A blind item keeps the file’s header text and hides NILS’s answers. h opens the whole header, less what names a person.
H switches the whole reader between compact and expanded, kept per person. The picture box has a fixed size on a big screen.
The next items are read ahead while one is read. The view (the stack, the three planes, or one plane) is the reader’s choice and stays.

Correct your own answer

While a campaign is open, My answers lists your answers, latest first, and opens any in the reader. u opens the last one again. A correction is written as a new answer that supersedes the earlier; the earlier is kept, and an export holds both. A closed campaign’s answers cannot be corrected.

Gallery

A campaign that asks one axis has a Gallery: a hundred items at once, each a small picture with the value suggested, who suggested it and how sure. A number key corrects one item, and Accept all as shown (Ctrl+Enter) sends every item’s own value. Each is still its own answer, marked as a batch answer. Sealed items and the share held back are never in a batch.

Pairs, anchors, A/B

Read pairs: two stacks of one subject, named left and right and nothing else. One key answers; the keys are on the page. Where the two stacks’ geometry matches, paging one pages the other.
Anchored: a candidate beside a reference pre and a reference post. Panels share one window; w gives each its own.
Settle: where voters split, the candidates show as A, B and C in an order drawn per item, with one reason each, never who gave it. n is neither, x is can’t tell. Once decided, 1 to 5 give a cause.

Change who rates

People change while a round is open. On a campaign that names its raters, its maker, or a person with work on Review who also works on Campaigns, can add and take off raters.

On the campaign’s page, press Change beside the raters.
The people who rate and the people the desk names are chips; a chip is on while that person rates. Press a chip to add or take off. Where the desk names no people, type a principal.
The foot says what will happen: “Adds rater-b. Takes off rater-a: their answers stay and what they hold goes back to the pool.” You must leave one rater.
Press Change raters. The page says who rates now and, in the engine’s words, any rater added who already read those stacks in another campaign. The change stands.

A campaign that names no raters offers nothing: anyone with work on Campaigns rates it, and naming one would shut out the rest. A closed campaign offers nothing either.

Adjudicate, close, export

When raters disagree an item goes to the adjudicator: Adjudicate, or Settle.
Close first shows what the close will write.
Export writes a label set: the answers as labelled data, each version in files of its own with the sha256 of its labels and whether it may train a model. It is read on Label sets.

Suggestions on a campaign’s page brings in answers suggested from outside NILS, from a tab-separated file; each is kept with its author and is never an answer. A sealed sample takes none.

A campaign’s results

A campaign’s page has two views, Items and Results. Results reads what the raters answered and what was settled.

On the campaign’s page, press Results.
Read the head: the items settled of all, the adjudicated, the exact agreement and kappa, the answers marked unsure, and, once the campaign is closed, the decisions or picks the close wrote.
By value: for each value, the answers given and the items settled, per axis on a question of several axes.
By rater: each rater’s answers, adjudications and unsure marks.
By item: each rater’s answer in a column of its own (? where unsure), the adjudication, the settled value, and what the close wrote, or the item’s state while the campaign is open.
Labels: the campaign’s label sets, each downloaded as its labels.tsv.

While the campaign is open, Results is offered to its maker (unless the maker rates in it), its adjudicators and a person with work on Review. Once it closes, it is offered to everyone in it. The raters are named as on the Items table: by name to the maker and to a person with work on Review, by number to anyone else. Results needs engine 1.0.0-alpha.88 or later; an older engine offers only Items.

Label sets

Label sets holds labelled data: the answers of a campaign, or the decisions in force on an axis, as files.

Open Label sets. The table lists each set with where it is from, its rows, whether it may train a model, and when it was made.
Choose a set to read its provenance (the handle it pinned, the epoch and the pack) and its labels as written.

Each version is in files of its own, with the sha256 of its labels. A set that holds any item of a sample sealed for certification is sealed, and never trains a model. A campaign’s Export makes a set. So does New label set, which writes the decisions in force on one axis:

Press New label set. It needs work on Review or on Campaigns, and see on Review.
Under Decisions on, choose the axis from the pack. The name is the axis unless you give one; a name taken gets its next version and nothing is written over.
Where there is more than one export place, choose Written to.
Press Write the label set. One row for each scan: the decision in force on the axis, by whoever made it. A decision not yet committed is left out, and so is a scan of a sealed sample; the page says how many.

The engine’s other options (the authors, a selection or a campaign to keep, staged decisions, development labels for training) stay at the keyboard: nils labels export.

On the command line

This stays a command; the desk has no button for it yet.

nils campaign repack <campaign> --dry-run
nils campaign repack <campaign>

repack moves an open axes campaign to the pack the engine serves now; the dry run writes nothing. The raters of an open campaign can also be changed with nils campaign raters <campaign> --add <principal> and --remove <principal>.

Sealed samples

A sealed sample is hidden from every door until a certificate unseals it. While a scan belongs to one, nothing any system said of it reaches anyone, an administrator included. Its file, header and pictures still open, and a rater reads it blind. Only a token with sealed:see reads a sealed stack, and no group gives it.