The Interrupt Controller That Isn't There

Apple Silicon gives you a hypervisor. It does not give you an interrupt controller you can own, and almost everything distinctive about amber follows from that one refusal.

Why any of this exists ¶

amber [20] is a small microVM monitor for arm64. It puts code in a hardware-isolated VM and runs it like a function call: spawn, run a command, throw the machine away. The motivating case is the one everybody now has: an agent, a build step, a snippet from a model, something you want to run without believing in it. A container is the usual answer and it shares a kernel with you. A VM does not, and on arm64 the hardware makes that isolation cheap enough to use per task.

Cheap enough, provided the spawn is fast. That is the number the whole design answers to, and it rules out the obvious implementation immediately. Booting Linux costs hundreds of milliseconds no matter how small you make the kernel; we build our own trimmed one, resin, which boots in about 60 ms, and even that is too slow to sit in front of a function call. The way out has been known since Firecracker’s snapshot work and the serverless cold-start papers that followed: boot once, freeze the booted machine, and start every subsequent sandbox by thawing the frozen one [1][2][3]. Restore is not boot. Restore is mapping a memory image and writing some registers, and it costs tens of milliseconds, most of which is not the VM at all.

So amber’s real product is snapshot and restore, and everything users touch is that one capability wearing different clothes: amber fork, the warm pool, amber exec <template> -- <cmd> at about 30 ms end to end.

Freezing a guest means capturing all of its state. RAM is a file. The vcpu registers have a getter each. The interrupt controller is neither, and on macOS it is the piece the platform will not hand back.

The restore that came back dead ¶

The first version of the snapshot path was written against Hypervisor.framework’s in-kernel vGIC [16], because that is the supported thing to build against and there was no reason to expect trouble. Capture wrote guest RAM, every vcpu’s general and system registers, and the device state. Restore mapped the memory image, replayed the registers, and re-entered the guest.

The guest came back. ps in the restored shell printed. Then nothing. No prompt, no output, no response to input, and no crash either. The vcpu was executing, happily, forever.

What had stopped was time. A Linux guest measures time with the architected virtual timer: a free-running counter, CNTVCT_EL0, and a compare register, CNTV_CVAL_EL0, that fires an interrupt when the counter passes it [4]. The kernel arms the compare, takes the interrupt, advances jiffies, runs the scheduler, arms it again. On HVF the counter is derived from the host’s mach_absolute_time() minus a per-vcpu offset, and the snapshot had been taken minutes or hours earlier on a different process. The compare value we restored was an absolute point on a timeline that no longer existed.

Both ways of handling it are wrong. Write the captured compare value onto the fresh timeline and it is already far in the past, so the timer reads as permanently expired and the guest spins servicing an interrupt that re-fires immediately. Skip it and the guest’s next deadline sits somewhere the counter will not reach for years. And underneath both, the decisive problem: with the in-kernel vGIC enabled, the framework owns delivery from the timer to the controller to the vcpu. Nothing amber does above the framework can put that tick back, because the part that would deliver it is on the other side of a boundary with no accessor.

That comment is still in the source, because it is the shortest statement of why the rest of this post exists:

//  - vGIC mode: do NOT write them. Pinning the stale compare value on a
//    fresh timeline wedges HVF's internal timer (stale CVAL fires
//    continuously -> busy-spin; a fresh one never fires), and HVF owns the
//    timer→GIC delivery anyway, so the periodic tick does not resume — the
//    known HVF gap that motivated the software GIC.

The choice at that point was between shipping a monitor that boots VMs and cannot fork them, or moving the interrupt controller into amber’s own address space.

What the GIC actually holds ¶

On arm64 the interrupt controller is the GIC, the Generic Interrupt Controller [5]. It sits between every device that wants attention and every CPU that could give it, and it owns the state that makes an interrupt mean anything. Which interrupt IDs are enabled. Which are pending. Which have been acknowledged but not yet completed, so a re-raise must not be delivered twice. What priority each carries, and what priority is currently running, which together decide whether a new interrupt preempts the handler already on the stack. Which CPU each one is routed to.

Interrupt IDs come in three flavours, and the split matters for everything below. SGIs (0 to 15) are software-generated, the IPIs one CPU sends another. PPIs (16 to 31) are private per-CPU lines, and the virtual timer is PPI 27. SPIs (32 and up) are shared peripheral interrupts, the devices, routed to a CPU by a target mask.

Restoring a snapshot means restoring all of that, and if the piece you cannot restore happens to be the one carrying PPI 27, the machine you bring back is intact in every respect except that its clock never advances again.

On the Linux backend, none of this is amber’s problem. KVM’s in-kernel vGICv3 handles delivery and the host kernel owns the timer. The part that matters most is that KVM exposes the controller’s state through device attribute groups, so a VMM can read it out and write it back [6]. Save and restore is an API there. On macOS it is not an API at all.

Why GICv2, when the host speaks v3 ¶

Moving the controller into userspace raises an immediate question: emulate which one? The obvious answer is GICv3, since that is what the hardware and the framework implement. amber emulates GICv2 instead, and the reason is a trap that does not exist.

GICv3 moved the CPU interface off the memory map and into system registers, the ICC_* family [7]. A guest acknowledges an interrupt by reading ICC_IAR1_EL1. To emulate that, a monitor must intercept guest system-register access, and Hypervisor.framework does not trap those accesses to the GIC interface. There is no hook, so there is no emulation. Apple’s in-kernel vGIC is the answer to exactly this gap, implemented below the boundary instead of exposed across it, which is a perfectly good decision that happens to be useless to us.

GICv2’s CPU interface is memory-mapped [5]. Every access to it is a stage-2 data abort, and HVF reports those with the faulting intermediate physical address and the syndrome register. That is a hook, and one hook is the difference between an architecture you can emulate and one you can only consume. Trap-and-emulate needs something to trap on; the requirement was stated formally in 1974 and it decided this design [8].

The cost is real: amber runs an older interrupt architecture than the hardware implements, because the older one is the one the host will let it see. The guest does not mind. Linux has had a GICv2 driver for as long as it has had arm64, and amber’s generated device tree tells it what it is getting [9]:

let intc = fdt.begin_node("intc@8000000")?;
fdt.property_string("compatible", "arm,cortex-a15-gic")?;
fdt.property_u32("#interrupt-cells", 3)?;
fdt.property_null("interrupt-controller")?;
fdt.property_array_u64(
    "reg",
    &[GIC_DIST_BASE, GIC_DIST_SIZE, GIC_CPU_BASE, GIC_CPU_SIZE],
)?;
fdt.property_u32("phandle", GIC_PHANDLE)?;

Two 64 KiB windows: distributor at 0x0800_0000, CPU interface at 0x0801_0000. The #interrupt-cells = 3 matters more than it looks. It fixes the shape of every interrupts property elsewhere in the tree, so the PL011 declares its line as <0 1 0x04> (SPI, number 1, level-high) and each virtio-mmio device gets its own SPI in the same encoding [10][18].

Inside the controller ¶

Every guest load or store into those two windows arrives as an exit with an IPA, and swgic_mmio decodes it into an offset and a register:

let (off, is_cpu) = if (GIC_DIST_BASE..GIC_DIST_BASE + GIC_DIST_SIZE).contains(&ipa) {
    (ipa - GIC_DIST_BASE, false)
} else if (GIC_CPU_BASE..GIC_CPU_BASE + GIC_CPU_SIZE).contains(&ipa) {
    (ipa - GIC_CPU_BASE, true)
} else {
    return Ok(false);   // not ours: fall through to the device model
};

Returning false rather than faulting is deliberate; the same exit handler serves the PL011 and the virtio devices, and the GIC is just the first claimant.

Behind the decode is a plain state machine, banked exactly the way the architecture says: INTIDs 0 to 31 and the whole CPU interface live per CPU, SPIs are shared, and ITARGETSR carries the routing mask. Writes to it clamp to the CPUs that actually exist, which is the kind of detail that only shows up as a bug much later:

// Compute the mask in u16 then truncate: with 8 cpus, 1u16<<8 == 256,
// so `(… as u8) - 1` would overflow (panic in debug, wrap in release).
let cpu_mask = ((1u16 << self.cpus.len()) - 1) as u8;

The delivery decision is one predicate, and getting it exactly right is most of the work of a correct controller:

fn highest_pending(&self, cpu: usize) -> Option<(u32, u8)> {
    if !self.dist_enabled { return None; }
    let cap = self.running_prio(cpu);
    let pmr = self.cpus[cpu].pmr;
    let mut best: Option<(u32, u8)> = None;
    for i in 0..NUM_INTID {
        if !self.enabled_at(cpu, i) || self.active_at(cpu, i) || !self.is_pending(cpu, i) {
            continue;
        }
        let p = self.priority_at(cpu, i);
        // PMR and running-priority both gate strictly (lower value passes).
        if p >= pmr || p >= cap { continue; }
        if best.is_none_or(|(_, bp)| p < bp) { best = Some((i as u32, p)); }
    }
    best
}

Four conditions and two gates. Enabled, not already active, pending; priority strictly better than both the mask register and whatever is currently running. A controller that treats either gate as non-strict delivers an interrupt that should have waited, which shows up weeks later as a guest that stalls under load and cannot be reproduced on demand.

Acknowledging is a read with side effects, which is the part of GICv2 that makes it a state machine rather than a register file. GICC_IAR returns the winner, marks it active, consumes the edge latch, and pushes its priority onto the running stack so that only something better can preempt it:

fn acknowledge(&mut self, cpu: usize) -> u32 {
    match self.highest_pending(cpu) {
        None => SPURIOUS,
        Some((intid, prio)) => {
            let i = intid as usize;
            if i < PRIVATE {
                let bank = &mut self.cpus[cpu];
                bank.active[i] = true;
                if bank.edge[i] { bank.pending[i] = false; }
            } else {
                self.active[i] = true;
                if self.edge[i] { self.pending[i] = false; }
            }
            self.cpus[cpu].running.push(prio);
            if i < 16 { intid | ((self.cpus[cpu].sgi_src[i] as u32) << 10) } else { intid }
        }
    }
}

GICC_EOIR deactivates the INTID and pops one level back off that stack. The edge/level distinction is the subtle half: an edge-triggered interrupt latches when it arrives and the latch is consumed on acknowledge, while a level-triggered one is pending exactly as long as its input line is high. Devices are level. SGIs are edge by nature. The timer is level, which is why the injection path re-drives it from the guest’s own compare register on every entry instead of latching it once.

One write gets special treatment. GICD_SGIR is how a guest sends an IPI, and an IPI posted to a vcpu that is currently executing guest code would sit unnoticed until that vcpu happened to exit for some other reason:

// An SGI (IPI) posted to another vcpu that is executing guest code is only
// noticed at its next exit; kick everyone so it lands now.
if off == 0xf00 {
    drop(g);
    let hs = self.shared.handles.lock().unwrap().clone();
    unsafe { hv_vcpus_exit(hs.as_ptr(), hs.len() as u32) };
    return Ok(true);
}

The whole file makes no hypervisor call. It is pure state, so it unit-tests without a VM, and it serializes, which is the reason it exists at all:

/// Serialize the full controller state to a flat blob (snapshot). The software
/// GIC has no opaque host object, so its state round-trips as plain bytes — and
/// because we deliver interrupts ourselves, the restored timer just works.
pub fn capture(&self) -> Vec<u8> {
    let mut v = Vec::with_capacity(...);
    v.push(Self::BLOB_MAGIC);   // 0xa2
    v.push(Self::BLOB_V2);
    v.push(self.cpus.len() as u8);
    v.push(self.dist_enabled as u8);
    ...

The magic byte earns its place: pre-SMP blobs began with a 0-or-1 dist_enabled byte, so 0xa2 is unambiguous against the old format and a template captured by an older amber fails loudly instead of restoring as nonsense.

Owning the timer, because nothing else will ¶

Turning off the in-kernel vGIC has a second consequence, and this is the one that turns the constraint from a tax into leverage.

With no in-kernel controller, HVF also stops absorbing WFI. The guest’s idle instruction surfaces as an exception with EC == 0x01 instead of being handled below the boundary, which means amber decides what idle means: park the vcpu thread, work out when the next timer is due, wake it up. Owning idle and owning the timer are the same job, and doing that job is what lets a restored snapshot tick at all.

The injection hook runs before every guest entry. It keeps HVF’s own virtual timer masked, reads the guest’s CNTV_CTL_EL0 to see whether the guest’s own deadline has passed, drives PPI 27 in this vcpu’s bank from that, and raises the IRQ line only if the controller has something deliverable:

if !self.vtimer_masked {
    check(hv_vcpu_set_vtimer_mask(self.handle, true), "vtimer mask")?;
    self.vtimer_masked = true;
}
// Virtual timer output: enabled (bit0), not masked (bit1), fired (bit2).
let ctl = self.get_sys(HV_SYS_REG_CNTV_CTL_EL0).unwrap_or(0);
let due = ctl & 0b001 != 0 && ctl & 0b010 == 0 && ctl & 0b100 != 0;
let pend = {
    let mut g = gic.lock().unwrap();
    g.set_level(self.cpu, VTIMER_INTID, due);
    g.irq_pending(self.cpu)
};
// HVF auto-clears the pending interrupt after each run, so only the `true`
// case needs a syscall — skipping the common `false` case halves the cost.
if pend {
    check(
        hv_vcpu_set_pending_interrupt(self.handle, HV_INTERRUPT_TYPE_IRQ, true),
        "set pending irq",
    )?;
}

Reading ISTATUS out of the guest’s control register rather than tracking a deadline ourselves is what keeps this honest across a restore: the guest’s own notion of when the timer is due is the only one that survives the process boundary, so it is the one we consult.

Idle then has a real deadline instead of a poll interval. pending_timer_ns converts the guest’s compare value into host nanoseconds, and the run loop parks for that long with a cap:

let cap = if virtio.lock().unwrap().iter().any(|d| d.mmio.wants_poll()) {
    1_000_000      // a network device is awaiting host replies: 1 ms
} else {
    50_000_000     // otherwise park up to 50 ms
};
let ns = match vcpu.pending_timer_ns() {
    Ok(Some(n)) => n.min(cap),
    _ => cap,
};
if ns > 0 {
    std::thread::park_timeout(std::time::Duration::from_nanos(ns));
}

An idle guest costs nothing here. It parks until its own next tick, and the host scheduler never sees it, which is what makes sixty idle forks per gigabyte tolerable as a steady state rather than a benchmark.

The arithmetic that brings the clock back ¶

With delivery in our hands, restore stops being impossible and becomes a change of coordinates. At capture, the guest’s counter read mono - vtimer_offset. On restore, in a different process at a different instant of host time, pick an offset that makes the counter read that same value now, and let it advance from there:

let now = unsafe { libc::mach_absolute_time() };
let captured_cntvct = cpu.mono.wrapping_sub(cpu.vtimer_offset);
let new_offset = now.wrapping_sub(captured_cntvct);
check(hv_vcpu_set_vtimer_offset(self.handle, new_offset), "set vtimer offset")?;
check(hv_vcpu_set_vtimer_mask(self.handle, false), "clear vtimer mask")?;

Then, and only then, the guest’s CNTV_CVAL and CNTV_CTL go back:

let restore_cntv = self.swgic.is_some();
for &(id, v) in &cpu.sysregs {
    if !restore_cntv && (id == HV_SYS_REG_CNTV_CTL_EL0 || id == HV_SYS_REG_CNTV_CVAL_EL0) {
        continue;
    }
    unsafe { hv_vcpu_set_sys_reg(self.handle, id, v) };
}

Ordering is load-bearing. The compare value is an absolute point on the counter’s line, so writing it before the offset is correct pins it against the wrong line. That was the bug that ate several evenings. And the same two registers must be skipped on the vGIC path, where writing them wedges HVF’s internal timer. One boolean, two backends, opposite decisions about the same pair of registers.

From the guest’s point of view nothing happened. It armed a timer 400 µs ago and the timer is still 400 µs away, in a process that did not exist when it was armed.

Where the bill arrives ¶

Emulating a controller above the hypervisor has a price, and it lands in one specific place.

Interrupts are injected from the pre-entry hook, and the hook only runs when the vcpu exits. A guest spinning in a compute loop that waits on jiffies never exits. The kernel’s own raid6 and crypto self-tests during boot are exactly this shape: process blocks until the tick count changes, then report a rate. The tick cannot change, because the only thing that would deliver it is waiting for the loop to finish. The machine wedges with nothing broken.

The fix is a thread that forces the exit the guest will not take on its own:

// ~500 Hz: enough for jiffies to advance through a benchmark, not a flood.
let ms: u64 = env::var("AMBER_PREEMPT_MS").ok().and_then(|s| s.parse().ok()).unwrap_or(2);
std::thread::spawn(move || loop {
    std::thread::sleep(Duration::from_millis(ms));
    let hs = shared.handles.lock().unwrap().clone();
    if !hs.is_empty() {
        unsafe { hv_vcpus_exit(hs.as_ptr(), hs.len() as u32) };
    }
});

hv_vcpus_exit forces every registered vcpu out of guest execution; the CANCELED arm of the exit handler does no work beyond falling through, which re-runs the injection hook and re-enters. The same forced exits pay a second debt by bounding cross-vcpu SGI latency to a couple of milliseconds. Interrupt latency is one of the places where ARM virtualization has historically been weakest [11], and forcing exits at 500 Hz trades a slice of guest throughput for a bound on it.

The other standing cost is the round trip itself. On the vGIC path, a guest acknowledging an interrupt reads a system register and stays in the guest. Here it takes a stage-2 abort, exits to userspace, walks amber’s decode, takes a mutex, mutates a few arrays, writes a general-purpose register back through the framework, steps the PC, and re-enters. That is the tax on every IAR and every EOIR, on an interrupt-heavy workload, forever. It is the reason --no-default-features still builds the vGIC variant: if you never need to fork, you should not pay for the ability.

The seam ¶

None of this leaks upward. amber-core holds the boot path, the device model, the snapshot format, and the run loop. It names no hypervisor, and it talks to hardware through two traits, the same split KVM drew between a thin accelerator and a userspace device model [17]. The asymmetry between backends lives in the type system rather than in branches: Hypervisor declares kick, set_yield, capture_gic and the rest as default no-ops, KVM overrides what it needs, HVF overrides more, and the backend-neutral loop calls both unconditionally because each is a no-op on the other host [12].

Once you are trapping HVC for the GIC’s sake, the power-state calls come along too. PSCI arrives through the SMC Calling Convention as an HVC exception [13][14], and CPU_ON, the call a Linux guest uses to bring up a secondary core [15], is serviced by posting an entry point into a slot the target vcpu’s parked thread is waiting on:

0x8400_0003 | 0xc400_0003 => {
    let target = (self.get_x(1)? & 0xff) as usize;
    let entry = self.get_x(2)?;
    let ctx = self.get_x(3)?;
    let (lock, cv) = &self.shared.cpu_on;
    let mut slots = lock.lock().unwrap();
    let rc: i64 = match slots.get(target) {
        Some(CpuOn::Off) => { slots[target] = CpuOn::Posted { entry, ctx }; cv.notify_all(); 0 }
        Some(_) => -4,   // ALREADY_ON
        None => -2,      // INVALID_PARAMETERS
    };
    ...
}

SMP stops at eight vcpus, because GICD_SGIR’s target list is eight bits wide. That is the architecture’s ceiling, not an arbitrary one, and amber reports it as such rather than pretending to scale past it.

What the detour bought ¶

What the nine hundred lines bought, first of all, is the thing that was missing at the start: a live VM on Apple Silicon can be captured and brought back with its periodic timer still running. Everything else follows from that one property, because once a booted machine can be frozen and thawed reliably, a sandbox stops being something you boot and becomes something you copy.

Copying it is cheap for a reason that has nothing to do with the GIC and everything to do with being able to use it. A fork maps the template’s memory image with MAP_PRIVATE, so the base pages are shared through the page cache across every fork of that template and each one pays only for the pages it dirties, which in practice is about 16 MiB resident. That is what puts roughly sixty idle sandboxes in a gigabyte before RAM, rather than CPU or spawn latency, becomes the binding constraint. The daemon keeps some of them pre-restored in a warm pool and hands one over in tens of milliseconds, which is how amber exec gets a command running in a fresh sandbox in about 30 ms on an M1 Pro, measured from CLI invocation to exit code rather than from some point inside the monitor. SMP came along almost incidentally: a monitor that already emulates the distributor is already holding the machinery that delivers IPIs, so PSCI bring-up and cross-core interrupts are serviced by the same file that services the timer.

None of that was reachable through the faster, better-supported path. There is a version of this project that took the in-kernel vGIC, wrote a thinner backend, deleted those nine hundred lines, and shipped a monitor that boots quickly and cannot fork. It would have been a nicer codebase and a worse tool.

So was it worth it ¶

Yes, with a bill attached, and the bill is worth stating plainly rather than leaving the reader to guess.

The software GIC costs a userspace round trip on every interrupt acknowledge and every end-of-interrupt, plus a thread forcing 500 exits a second whether or not anything is happening. An interrupt-heavy guest pays for that continuously, and nothing about the design makes it go away. It also pins amber to GICv2, which caps SMP at eight vcpus, and it means the correctness of every guest’s timer, priority masking, and IPI delivery rests on code I wrote rather than on code Apple or the kernel maintainers wrote. That is a real risk and the reason the controller is pure state with its own tests.

Against that: on Apple Silicon, without it, there is no fork, no warm pool, and no 30 ms sandbox, because there is no restore that keeps time. The feature that makes amber worth using does not exist on the supported path. So the tax is not overhead bolted onto a working product, it is the price of the product existing at all, and for a monitor whose entire purpose is spawning disposable sandboxes fast it is a trade I would make again.

What amber is not, yet, is finished. The published numbers are HVF-only; the KVM backend runs the full pipeline but has been validated under emulation rather than on real arm64 KVM hardware, so I do not claim performance figures for it. The networking is a userspace netstack with outbound TCP, DNS, and inbound forwards, which is enough for the workloads amber targets and is not kernel-speed. busybox and musl are still borrowed Alpine artifacts. Those are ordinary young-project gaps, and they are separable from the thing this post is about.

The constraint was never that Hypervisor.framework lacked a feature. It has a good one, and for most VMMs it is the right one. The trouble was that the feature has a shape amber cannot use, and the only way to establish that was to build on it and then watch a restored guest sit there, perfectly alive and entirely out of time.

---

References ¶

[1] Agache, Alexandru; Brooker, Marc; Iordache, Alexandra; Liguori, Anthony; Neugebauer, Rolf; Piwonka, Phil; Popa, Diana-Maria. “Firecracker: Lightweight Virtualization for Serverless Applications.” 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ‘20), 2020. The microVM design amber narrows to arm64, and the snapshot-and-resume model behind fast spawn.

[2] Du, Dong; Yu, Tianyi; Xia, Yubin; Zang, Binyu; Yan, Guanglu; Qin, Chenggang; Wu, Qixuan; Chen, Haibo. “Catalyzer: Sub-millisecond Startup for Serverless Computing with Initialization-less Booting.” ASPLOS ‘20, 2020. Restoring a checkpointed sandbox instead of booting one, and why the checkpoint’s device state is the hard part.

[3] Ustiugov, Dmitrii; Petrov, Plamen; Kogias, Marios; Bugnion, Edouard; Grot, Boris. “Benchmarking, Analysis, and Optimization of Serverless Function Snapshots.” ASPLOS ‘21, 2021. Where snapshot-restore time actually goes, including the working-set paging amber’s copy-on-write mapping depends on.

[4] Arm Ltd. Arm Architecture Reference Manual for A-profile architecture (ARM DDI 0487), chapter on the Generic Timer. CNTVCT_EL0, CNTV_CVAL_EL0, and the CNTV_CTL_EL0 ENABLE/IMASK/ISTATUS bits the injection hook polls on every guest entry.

[5] Arm Ltd. ARM Generic Interrupt Controller Architecture Specification, version 2.0 (ARM IHI 0048B.b). The memory-mapped distributor and CPU interface amber emulates: GICD/GICC layout, IAR/EOIR acknowledge-and-complete, priority masking, the running-priority stack, and the SGI target list that caps SMP at eight.

[6] The Linux Kernel. “GICv3 Device (KVM),” Documentation/virt/kvm/devices/arm-vgic-v3.rst. The KVM_DEV_ARM_VGIC_GRP_* attribute groups that let a VMM read and write in-kernel controller state. This is the save/restore interface that has no macOS equivalent.

[7] Arm Ltd. Arm Generic Interrupt Controller Architecture Specification, GIC architecture version 3 and version 4 (ARM IHI 0069). The ICC_* system-register CPU interface that replaces the memory-mapped one, and therefore the trap Hypervisor.framework does not provide.

[8] Popek, Gerald J.; Goldberg, Robert P. “Formal Requirements for Virtualizable Third Generation Architectures.” Communications of the ACM, vol. 17, no. 7, 1974. Trap-and-emulate as a requirement on the architecture: sensitive operations must be interceptable, or they cannot be virtualized above the boundary.

[9] The Linux Kernel. “ARM Generic Interrupt Controller” devicetree binding (Documentation/devicetree/bindings/interrupt-controller/arm,gic.yaml) and the irq-gic driver. The arm,cortex-a15-gic compatible string amber advertises and the two-window reg layout the driver expects.

[10] The Linux Kernel. “Open Firmware and Devicetree, interrupt mapping,” #interrupt-cells and the three-cell <type number flags> encoding used by every interrupts property in amber’s generated tree, including the PL011 and each virtio-mmio device.

[11] Dall, Christoffer; Li, Shih-Wei; Lim, Jin Tack; Nieh, Jason; Koloventzos, Georgios. “ARM Virtualization: Performance and Interrupt Latency.” 43rd International Symposium on Computer Architecture (ISCA ‘16), 2016. Where interrupt latency comes from on ARM, and what it costs to resolve it above the hypervisor rather than inside it.

[12] Dall, Christoffer; Nieh, Jason. “KVM/ARM: The Design and Implementation of the Linux ARM Hypervisor.” ASPLOS ‘14, 2014. The in-kernel vGIC and virtual timer on the Linux side, and why keeping both below the userspace boundary is what makes that backend thin.

[13] Arm Ltd. SMC Calling Convention (ARM DEN 0028). How firmware calls are encoded in x0 and delivered as HVC/SMC, the exception amber decodes before dispatching PSCI.

[14] Arm Ltd. Power State Coordination Interface (ARM DEN 0022). PSCI_VERSION, CPU_ON, SYSTEM_OFF, and the return codes a monitor must produce for a Linux guest to bring up secondary cores and shut down cleanly.

[15] The Linux Kernel. “Booting AArch64 Linux,” Documentation/arch/arm64/booting.rst. The register and device-tree contract satisfied before the first guest instruction, and the enable-method = "psci" path that routes secondary bring-up through [14].

[16] Apple Inc. “Hypervisor framework,” Apple Developer Documentation, developer.apple.com/documentation/hypervisor. hv_vcpu_run, hv_vcpus_exit, hv_vcpu_set_pending_interrupt, hv_vcpu_set_vtimer_mask, hv_vcpu_set_vtimer_offset, and the hv_gic_* family added in macOS 15.

[17] Kivity, Avi; Kamay, Yaniv; Laor, Dor; Lublin, Uri; Liguori, Anthony. “kvm: the Linux Virtual Machine Monitor.” Proceedings of the Linux Symposium, 2007. The split amber’s Hypervisor trait mirrors: a minimal accelerator underneath, device emulation in userspace above.

[18] OASIS. Virtual I/O Device (VIRTIO) Specification, version 1.2, 2022. The MMIO transport for the block, rng, net, balloon, and vsock devices whose interrupt lines are the SPIs the software controller routes.

[19] “Constraints as Method.” This site, 2026, /posts/2026-05-15-constraints-as-method. The general version of this post’s argument: a platform limit that forces a design can be worth more than the feature it withheld, provided you work out which limit you are actually up against.

[20] lupodevelop/amber. The monitor this post is about, Apache-2.0. The software controller is crates/amber-hvf/src/gicv2.rs, the injection hook and virtual-timer handling are in crates/amber-hvf/src/lib.rs, and the backend-neutral run loop is crates/amber-core/src/vm.rs.