Infrastructure
Designed, built and run by us, on open source.
We designed this platform ourselves and put it together in house: two machines, a set of open-source services arranged in layers, and the procedures to run them. Asgard computes, Midgard stores, and nothing on either is reachable directly. This page shows how the layers fit together and which projects they are built on.
The layers
A member’s computer is known by its SSH key, registered by an administrator; a lost laptop is revoked in a minute. Browser logins use the kineuro account with a mandatory second factor. There are no passwords for the machines and no self-registration; from any network, always over TLS.
EdgeTraefikTraefik terminates TLS for every name with Let’s Encrypt certificates and routes each to its service. Only HTTPS is exposed; the machines, their management controllers and the storage network are not reachable from the Internet.
GatewayWarpgateWarpgate is the SSH and web bastion: key-only access, one target per role, every session recorded, repeated failures banned. No password can reach a machine.
IdentityAuthentikAuthentik is the single sign-on for everything in a browser, with a mandatory second factor; groups decide who may open what, and accounts are created only by an administrator.
Imaging data serviceNILSNILS is our own registry for imaging data: every scan digested and classified by a rule pack, every person pseudonymous, every question a stored document with a hash, every release described before it is made. It runs in its own container with the GPUs and the source data beside it, and the desk is behind the portal login.
Data bridgeBifrostBifrost is our own door for exchanging data with partners: a link per collaboration with direction, space, expiry and checksums, in the browser or with a small command for terabytes. It runs in its own virtual machine that sees one exchange area and nothing else; nothing enters a study until an administrator accepts it.
OverlayHeadscaleHeadscale (the open Tailscale control server) and our own relay connect the other sites and administrators’ devices without opening more ports.
HypervisorProxmox VEProxmox VE runs one Debian container per person, each with both GPUs visible, its own root, Docker inside, and the person’s home and data mounted from outside. A rebuild takes minutes and loses nothing.
GPUsNVIDIATwo NVIDIA RTX PRO 6000 Blackwell (96 GB each), shared at run time by everyone; the driver is shared, each environment pins its own CUDA.
Fast storageOpenZFSZFS on four NVMe drives (56 TB) for homes, scratch, working sets and the fast copy of the current data release.
Archive and resultsTrueNASTrueNAS with a ZFS pool of 32 hard drives (640 TB) in four RAIDZ2 groups: source data, dated BIDS releases, results and backups, with snapshots and monthly scrubs.
Four bonded 100 Gbit/s links between the two machines; NFS 4.2 carries the data, measured at over 4 GB/s in and 17 GB/s out.
BackupsProxmox Backup ServerProxmox Backup Server keeps nightly copies of the platform containers on Midgard; the imaging data has its snapshot history on Midgard itself.
Metrics and alertsPrometheusPrometheus collects from both machines’ management controllers, the GPUs, the hypervisor, every door and a watchdog that tracks updates; Alertmanager posts to Teams.
DashboardsGrafanaGrafana shows the platform’s health at a glance and in depth, behind the same login.
Website and portalAstroAstro renders these pages from files in a GitHub repository; members write news in Keystatic; the staff portal reads who you are from the login.
CodeGitHubOur software lives in the kineuro organization on GitHub.
The machines
Other sites: our legacy storage server and Fractal reach Asgard over the overlay only, on SSH, to push data into Midgard’s source pool. The bridge that moves their archives is the next piece of work.
Built on open source
Nothing on this platform is a black box. Every layer is free software maintained by people who share their work, and we are grateful to them. In turn we publish our own tools on GitHub, open where we can.
Debian
the operating system of every container
Linux
the kernel under all of it
Proxmox VE and Backup Server
hypervisor and backups
TrueNAS
the storage appliance
OpenZFS
the file system on both machines
Traefik
the edge
Let’s Encrypt
certificates for every name
Warpgate
the SSH and web gateway
Authentik
identity and second factor
Headscale
the overlay control server
Docker
containers inside containers
NVIDIA open kernel modules
the GPU driver
Prometheus and Alertmanager
metrics and alerts
Grafana
dashboards
Astro and Keystatic
this website and its editor
Python
the watchdog and most of the tooling
Bifrost
our own data bridge, published under MIT
Go
the bifrost command line tool
tus
resumable uploads in the browser
Logos are the projects’ own marks, used to credit them. Hardware by Supermicro, delivered and assembled with Nextron; hosted at Karolinska Institutet.