path: root/README.md

# zig-p4 — an ESP32-P4 toolchain that is just Zig

```
zig build            # compile, link, and emit a flashable image
zig build flash      # ...then write it to the chip and run it
zig build run        # flash, then print the console (ordered; `flash monitor` is not)
zig build monitor    # reset the board and print its console
zig build reset      # just pulse the reset line
zig build size       # where every byte of the image went
zig build test       # host tests: image builder, and the register layer's field arithmetic
zig build diff       # the hardware oracle: this HAL vs ESP-IDF's, on the die (needs -Doracle)
zig build elf        # stop at the ELF, for disassembly
```

No CMake, no ninja, no `idf.py`, no `esptool`, no external linker. Zig's own LLD does the link
against a linker script this `build.zig` generates; the image builder and the serial flasher are
ordinary Zig code in `tools/`, imported straight into `build.zig`, so they leave no artefacts of
their own. What lands in `zig-out` is one file: the image.

## Why it exists

| | ESP-IDF blink | earlier Zig proof-of-concept | this |
|---|---|---|---|
| flashed image | 184,112 B | 66,176 B | **1,216 B** (432 B for `examples/minimal.zig`) |
| build, cold | 23 s, 1,072 ninja edges | 1.3 s | **1.37 s** |
| build, warm | ~1 s | 0.07 s | **0.088 s** (cache hit, no work) |
| build, one file edited | ~1 s | 0.07 s | **0.121 s** |
| flash + verify + run | ~2 s (esptool + stub) | ~2 s | **0.277 s** |
| host dependencies | ESP-IDF 663 MB + toolchain 3.4 GB + Python | Zig + system LLD + esptool | **Zig** |
| artefacts per build | ~1,100 files | 2 | **1** |

The 66 KB → 1 KB step is the interesting one. `esptool` refuses to put two flash-mapped segments
inside the same 64 KiB MMU window (`bin_image.py:832-838`) and pads the image out to the next
window, which for a small program is 64 KiB of zeros. The chip only requires that each mapped
segment satisfies

```
(offset of its data within the image) % 64 KiB == (its load address) % 64 KiB
```

(`esp_image_format.c:903-909`), and the MMU is perfectly happy to point two entries — or the same
entry twice — at one flash page; stock ESP-IDF already aliases a page that way on every boot.
`tools/image.zig` therefore packs both segments into one window and the pad disappears. Verified on
ESP32-P4 rev v1.3 silicon.

## Requirements

* Zig 0.16.0. That is the whole list.
* Membership of whatever group owns the serial port (`uucp` on Arch, `dialout` on Debian).
* An ESP-IDF second-stage bootloader and partition table already in flash at `0x2000` and `0x8000`.
  This toolchain builds and flashes *applications*; the bootloader is still Espressif's. See
  "What is still ESP-IDF" below.

## Options

Everything is a `b.option`, so `zig build -h` lists them all.

| option | default | meaning |
|---|---|---|
| `-Dapp=<path>` | `src/main.zig` | application root source |
| `-Dport=<path>` | `/dev/ttyUSB0` | serial port |
| `-Dbaud=<enum>` | `b921600` | flashing baud. `b2000000` does not work on this board's CH340 |
| `-Dled=<u8>` | `20` | GPIO the demo blinks (20 = JP1 pin 17) |
| `-Doffset=<u32>` | `0x10000` | flash offset of the app partition |
| `-Dflash-size=<enum>` | `16MB` | fitted flash, written into the image header |
| `-Dmin-rev`/`-Dmax-rev` | `100`/`199` | silicon revision window. The pre-v3 P4 needs 100..199 |
| `-Ddescriptor=<enum>` | `minimal` | 184-byte descriptor, or `full` for the 256-byte one `esptool image-info` can parse |
| `-Dstack=<u32>` | `8192` | stack size; the generated linker script follows |
| `-Dverify=<bool>` | `true` | ask the ROM for an MD5 of what it stored and compare |
| `-Delf=<bool>` | `false` | also install the ELF |
| `-Doptimize=<mode>` | `ReleaseSmall` | firmware default, not Debug (Debug costs ~780 B here) |
| `-Dseconds=<u32>` | `5` | how long `monitor` listens |

## Layout

```
build.zig              the toolchain: target, generated linker script, and four custom steps
tools/image.zig        ELF -> ESP image. Header, segments, congruence filler, checksum, SHA-256
tools/image_test.zig   8 host tests, one per rule the ROM bootloader enforces
tools/rom.zig          SLIP framing + the ROM loader protocol. No software stub
tools/serial.zig       termios2 raw mode, arbitrary baud, DTR/RTS reset dance
src/soc.zig            comptime register model: GPIO, IOMUX, mask-ROM entry points, cycle counter
src/appdesc.zig        esp_app_desc_t, linked as its own object so it cannot be optimised away
src/main.zig           demo: prints what it can prove, then blinks
examples/minimal.zig   the floor: 432 B, blinks and nothing else
```

## What adversarial review found

Five reviewers went at this in two rounds; sixteen findings were applied. The ones worth knowing
about, because each is a trap the next person will hit too:

* **The ROM's status byte is at `data[len-4]`, not `data[len-2]`.** The four-byte trailer is a
  ROM-versus-stub difference (esptool `loader.py:653-655`). Reading the wrong byte made *every* ROM
  error read as success: a rejected `FLASH_DATA` block was never retried and `zig build flash`
  printed a cheerful success line over a dead image. Only the MD5 pass caught it. Now the offset is
  right, the length check is mandatory, and `-Doffset=33554432` (past the end of a 16 MB part)
  fails with `write failed: CommandFailed` instead of claiming victory.
* **Congruence modulo the MMU page is necessary but not sufficient.** The first solver satisfied it
  by shifting a segment forward, which could leave both mapped segments in one *vaddr* page while
  their data sat in two different *flash* pages. The bootloader writes one MMU entry per vaddr page,
  so the second mapping replaced the first and the app booted with every constant reading as zero -
  no error anywhere. `-Ddescriptor=full` triggered it. The solver now anchors every mapped segment
  to a single `flash_address - load_address`, and `validate` checks the invariant directly.
* **The target enables `f` but nothing enabled the FPU.** `mstatus.FS` is Off after reset and
  ESP-IDF only turns it on lazily from a trap handler, which this image does not have, so the first
  `f32` multiply in application code was an unhandled illegal instruction. `_start` now sets FS, and
  the demo prints a float computed with hardware `fmul.s` as proof.
* **`-Dmin-rev` was feeding the descriptor's eFuse *block* revision**, an unrelated field, so
  `-Dmin-rev=150` produced an image the bootloader refuses with `Image requires efuse blk rev >= v0.50`.
* **`flash` and `monitor` are unordered top-level steps** and the build runner runs independent
  steps concurrently, so `zig build flash monitor` could pull the chip out of download mode
  mid-write. They now share a mutex - and because a mutex cannot express *order* (measured: monitor
  won 3 times out of 3, printing the old firmware before the new one was written), `zig build run`
  exists as the ordered version.
* **A cache manifest that does not hash the builder is worse than no cache.** The first version
  hashed only the ELF and the options, so editing `tools/image.zig` produced a cache hit and
  shipped the previous image - and `--watch` never noticed the edit at all. The manifest now covers
  the builder sources too.
* **`ALIGN(64)` does not reserve a hole.** The image builder needs 8 spare bytes before the second
  mapped segment for its header; for one rodata length in eight, 64-byte alignment leaves 0 or 4,
  and the build failed with `MappedSegmentsTooClose`. The generated script now ends `.flash.rodata`
  with `. = ALIGN(. + 8, 64) - 8;`, verified over a sweep of rodata sizes.
* **A constant the compiler folds proves nothing.** The FPU check computed `7 * 1.5 + 0.25` from a
  literal, so LLVM folded it and the image contained no float instructions at all: the test would
  have passed on a board whose FPU was still off. It now loads through a volatile pointer, and the
  image really does contain `fcvt.s.wu`, `fmul.s`, `fadd.s`, `fcvt.wu.s`.

## Three things that bit, and are now encoded in the code

1. **`standardOptimizeOption` hands out Debug builds.** With `preferred_optimize_mode` set it
   exposes `-Drelease` and still defaults to Debug, which for this target means panic machinery and
   formatting code inside a 500-byte image. `build.zig` takes `-Doptimize` and defaults to
   `ReleaseSmall` explicitly.
2. **An `@import`ed descriptor disappears under ReleaseSmall.** A `comptime _ = descriptor;`
   reference is enough in Debug; in release the constant folds away, `.flash.rodata` vanishes,
   `KEEP` has nothing to keep, and the image boots with one mapped segment at the wrong offset.
   The descriptor is now a separate object linked unconditionally.
3. **`tcsetattr` cannot set the baud.** `TCSETS`' struct has no `ispeed`/`ospeed`, so assigning
   those fields silently does nothing and the port stays at whatever rate it had — a 1.5 KB flash
   took 230 ms instead of 60. `tools/serial.zig` uses `TCGETS2`/`TCSETS2` with `BOTHER`, and
   declares `struct termios2` itself because std's version is 60 bytes where the kernel's is 44,
   which makes std's ioctl numbers wrong.

## What is still ESP-IDF

The second-stage bootloader at `0x2000` and the partition table at `0x8000`. Both are ordinary
flash contents and neither is rewritten here. The bootloader is what enforces the three rules this
toolchain obeys:

* exactly two segments in the mapped range (`bootloader_utility.c:842`)
* every segment length a multiple of 4 (`esp_image_format.c:857`)
* mapped segments congruent modulo the MMU page (`esp_image_format.c:903-909`)

`tools/image.zig` re-checks all three after building an image, so a violation is a build error
rather than a board that resets in a loop.

## The HAL: registers from ESP-IDF, sequences in Zig

A toolchain that can only blink an LED is a demo. The rest of the chip needs a hardware layer, and
the ESP32-P4 has a lot of chip: 96 `*_ll.h` headers in ESP-IDF v6.0.2 holding **3,081** inline
functions over **5,170** registers and **20,052** fields. Hand-transcribing that is not a plan —
the hand-written predecessor of `src/hal/gpio.zig` had a wrong matrix constant with a comment
warning about exactly that mistake.

So the register layer is not written at all. `build.zig` runs `zig translate-c` over a generated C
file that `#include`s every one of ESP-IDF's own `*_reg.h` headers, and the result is imported as a
module:

```zig
const regs = @import("regs");   // 87,373 constants, straight from IDF's macros
```

That takes 0.26 s and costs about 0.16 s of parse on top of a 1.7 s cold build; Zig only analyses
the declarations actually referenced, so the unused 87,000 are free. Every address, shift and mask
in this HAL is therefore not a re-derivation of ESP-IDF's number — it *is* ESP-IDF's number, as
evaluated by clang. `*_struct.h` is deliberately unused: translate-c demotes each of those register
structs to `opaque {}` ("has bitfield"), so the C bitfields buy nothing.

`src/mmio.zig` is the ~200 lines that turn flat constants into checked accessors:

```zig
const conf0 = mmio.Reg.at(regs.LEDC_CH0_CONF0_REG);
const timer_sel = mmio.Field.of(regs.LEDC_TIMER_SEL_CH0_S, regs.LEDC_TIMER_SEL_CH0_V);
conf0.modify(.{ timer_sel.is(2) });     // read-modify-write, preserving the rest
```

Fields are built from the `_S`/`_V` pair and never from `_M`, which is not a style rule: **153 `_M`
macros are broken C inside ESP-IDF itself** (`INTERRUPT_CORE0_LP_RTC_INT_MAP_M` expands
`CORE0_LP_RTC_INT_MAP_V`, dropping the prefix). Nothing in C ever expanded them, so nobody noticed;
`translate-c` surfaces them as poisoned declarations. `Field.of` also rejects a pre-shifted mask at
comptime, so passing `_M` by hand is a build error rather than a wrong bit position.

`modify` is the default and `write` is the exception, because `write` zeroes what it does not name
and 46.7% of the registers a low-level driver touches have a field whose reset value is not zero.

### The build refuses to hide what it could not translate

`translate-c` exits 0, prints nothing, and still emits `pub const X = @compileError(...)` for every
macro it could not handle — invisible until a driver names one. So a build step counts them and
fails if the number grows past a recorded 524, whose composition is written down: 333 register
addresses whose `DR_REG_*_BASE` **ESP-IDF references and never defines anywhere**, 153 broken `_M`
masks, and 38 function-like macros with Zig equivalents. Three of those missing bases were
recovered from `esp32p4.peripherals.ld` — which cross-checks against `reg_base.h` on all 65
peripherals both files name, 0 disagreements — putting 176 registers back. The rest stay
unreachable on purpose rather than reachable at a guessed address.

### The oracle: differential testing against ESP-IDF, on the die

Register numbers being right does not make a *sequence* right. So ESP-IDF's own `*_ll.h` functions
are compiled by Zig's clang into the same image as this HAL, and `zig build diff` drives each
operation both ways on the chip and compares the register block afterwards:

```
MARK DIFF_CFG gpio_ll_uses_rom_api=0 expect=0
MARK DIFF ok    gpio.set_level(1) 400 words identical
...
MARK DIFF_TOTAL cases=26 failures=0
```

It found two real bugs in this HAL on its first run. `GPIO_FUNC0_OEN_SEL` reads backwards from its
name — 1 means "use `GPIO_ENABLE_REG`", 0 means "use the peripheral's own output enable" — so
`matrixOut` had it inverted and then set the matching `GPIO_ENABLE` bit to compensate. It worked, by
the wrong mechanism, leaving the pad latently output-enabled. The first version of the harness also
only compared 112 words and so saw the symptom without the cause; the window now reaches the matrix
configuration registers at +0x558.

Four things the harness has to get right, each measured on this board rather than assumed:

* **A snapshot can have side effects.** `UART_FIFO_REG` is at offset 0x000 of every UART block — the
  first word a "read the whole block" loop touches — and reading it pops the RX FIFO. The header
  annotates it `RO`. Peripherals declare offsets that must not be read.
* **A block cannot be restored by writing its snapshot back.** ~10% of this chip's fields act when
  written; writing one saved word back to a UART's offset 0 transmits a character. Restore is the
  peripheral's reset bit, or a deliberate configure function.
* **A clock-gated block reads stale data, silently** — the last value latched, not zeros, so two
  meaningless snapshots can compare equal. The bus clock is checked before every comparison.
* **Equal registers do not prove equal sequences.** LEDC commits shadow registers through a
  self-clearing bit that leaves no trace afterwards.

### What is ported

| peripheral | what it covers | differential cases |
|---|---|---|
| `hal.gpio` | 57 pins across both banks, IO MUX pads, pulls, drive strength, open drain, the GPIO matrix in and out | 46 |
| `hal.timg` | TIMG0/1, two timers each: dividers, direction, auto-reload, alarms, the latch-then-read counter, and MWDT behind its write-protect key | 29 |
| `hal.uart` | UART0-4: the fractional baud divider, data format, FIFOs, loopback, pin routing, and the `_SYNC`/`REG_UPDATE` commit | 19 |
| `hal.intr` | the CLIC (not a PLIC): the 122-source interrupt matrix, per-line enable/trigger/priority, the memory-mapped threshold, a 48-entry vector table | 20 |
| `hal.ledc` | Q10.8 timer dividers, channels, duty, idle level, the shadow-register commits, pin routing | 28 |
| `hal.i2c` | I2C0/1 master: the full ten-register timing set plus its out-of-block clock divider, a typed command list, FIFOs, transaction status | 43 |
| `hal.clkrst` | peripheral clock gates and resets, under an interrupt-masked read-modify-write guard | 8 |
| `hal.systimer` | the two 52-bit counters, through their update/valid handshake | — |
| `hal.rwdt` | the RTC and super watchdogs, which the bootloader leaves armed | — |

**193 cases, 0 failures** on the die.

That number was 177 until an adversarial audit of the harness pointed out that about seventy of the
cases could not fail for any implementation error, which is worth more than the passing count was.
Two structural causes, both now fixed:

* **A window that missed the registers under test.** Every pad-configuration function writes the IO
  MUX at `0x500E1004 + 4*pin`, and GPIO's compared window ended at `0x500E063F` - 0xC00 bytes short.
  Eight operations across two pins were comparing two identical snapshots of a register file none of
  them touch. `iomux_suite` is the same cases against the right window.
* **Restores written with the code under test.** The harness runs restore, IDF, snapshot, restore,
  ours, snapshot. If restore calls the HAL, run B starts from whatever IDF just wrote, and a HAL
  function that does nothing at all compares equal - so each suite was blind to exactly the failure
  it existed to catch. `clkrst`'s restore called `setClockEnabled`, which is the function whose
  wrong-register bug started this whole line of work. Every restore is now built from register
  macros or from ESP-IDF's own LL, never from ours.

`src/hal` and `src/mmio.zig` are 4,590 lines; the oracle that checks them is 4,107.

`examples/halcheck.zig` exercises the first three on hardware directly. One number out of it is worth
keeping: the CPU runs at **89,995 kHz**, measured by counting cycles against the systimer's fixed
16 MHz over 50 ms rather than estimated — two independent clocks, not one clock and an assumption.

### Three traps now encoded in the code

**Peripheral clocks are already on.** `esp_system/port/soc/esp32p4/clk.c:200` says so and the reset
defaults agree, so the hazard is not gating but atomicity: every gate and reset bit shares a register
with unrelated peripherals, and ESP-IDF makes an unguarded call *uncompilable* by referencing an
identifier it never defines. `clkrst.maskInterrupts()` is the replacement for the spinlock IDF uses.

**Resetting a timer group re-arms flash-boot watchdog protection**, which reboots the board a moment
later with nothing on the console to explain it. `resetPeripheral(.timg0)` clears it as part of the
reset.

**The bootloader leaves the RTC watchdog running**, and expects the application to take it over —
which `esp_system` does and a bare image never did. Every demo in this repo had been resetting on a
ten-second cycle since the first one, invisibly, because no run had ever lasted eight seconds. It
surfaced when the differential grew past 64 cases and stopped fitting inside one watchdog period,
where it looked exactly like "the newest suite crashes the board". `hal.rwdt.disable()` is the fix
and `hal.rwdt.armed()` is the thing to print at startup.

## Reproducing the report's RF experiment

`04-report` section 5 is the only real SDR measurement in that document: a 25 MHz carrier on GPIO20,
OOK-keyed with `0x4200`, detected at +12.96 dB and decoded back from raw IQ. `examples/rf.zig` is that
emitter driven by this HAL instead of ESP-IDF, and the report's own capture and decode tools are used
unmodified. Full write-up and the numbers: `captures/RF-REPRODUCTION.md`.

The emitter reproduces exactly. The report derives its carrier from the divider — 1-bit resolution
off 80 MHz can only give `80e6 × 256 / (410 × 2)` = 24,975,609 Hz, never a round 25 MHz — and this
HAL computes divider 410 and 24,975,610 Hz from IDF's own Q10.8 arithmetic. The source clock was then
measured rather than assumed: 80,104,000 Hz, timed against the systimer. The differential harness
gained a case that runs the whole bring-up both ways in one image, and the LEDC block comes out
**96 words identical**; an edge count on the pad agrees to 2 parts in 112,500 at every rate the
instrument can measure.

**The detection does not reproduce, and not because of this toolchain.** ESP-IDF's own binary was
reflashed and driven by the report's own tool with the report's parameters:

| firmware | ON − OFF |
|---|---|
| report, §5.2 | **+12.96 dB** |
| ESP-IDF, re-run today | −0.063 dB |
| this toolchain, today | +0.05 dB |

The receive chain checks out — an FM carrier sits 31.8 dB above the span median, against the report's
30.9 dB — so the instrument is fine and both firmwares now give the same null. The report says its
emission reached the dongle through whatever wire happened to be on JP1 pin 17; that coupling is
gone. A measurement whose apparatus is incidental coupling is not reproducible, and this one is not.
That is a correction to the report rather than a difference between toolchains.

Wi-Fi, BLE and 802.15.4 stay out of scope by construction: the P4 has no radio, and those went over
an ESP32-C6 across SDIO under `esp_hosted` plus a prebuilt coprocessor binary. Reproducing them means
porting `esp_hosted`, not porting a HAL.

## The vendor ISA extensions, from a stock Zig

The P4's cores carry Espressif's `xespv2p1` vector unit ("PIE") and the `xesploop1p0` hardware
loops, which upstream LLVM does not know. That is really two gaps:

1. **The optimiser will never choose one.** No cost model, no intrinsics, no autovectorisation.
   That ceiling does not lift — but Espressif's own fork does not autovectorise into this unit
   either, which is why `esp-dsp` is hand-written assembly.
2. **The assembler cannot spell one.** Inline assembly does *not* help; it goes through the same
   integrated assembler: `asm volatile ("esp.vld.128.ip q0, a0, 16")` →
   `error: <inline asm>:1:2: unrecognized instruction mnemonic`. This is what stops Zig from
   assembling ESP-IDF's FreeRTOS context switch (`portasm.S`, 34 instructions under
   `#if SOC_CPU_HAS_PIE`).

Gap 2 closes without forking a compiler: `.insn` accepts any word, so generate the words offline
and pin the registers the fixed encoding names. `tools/encode.sh` is that oracle — it drives
Espressif's GAS at build-authoring time, never at build time, and prints a `.insn` line per
mnemonic. All 34 of `portasm.S`'s instructions encode, and the vld/vst/qacc words match what
Espressif's objdump reads out of an ESP-IDF build of this board byte for byte.

Watch the version: the extension is versioned and the versions disagree.
`esp.vld.128.ip q0, a0, 16` is `0x0311009f` under `xespv` (which defaults to 1.0) and `0x0201223b`
under `xespv2p1`. The P4 wants the latter — the pair in every IDF image's `.riscv.attributes` here
is `xesploop1p0_xespv2p1`.

`examples/pie.zig` runs the whole thing on the die: enable the unit (`csrw 0x7f2, 1`), a 16-byte
vector load/store round trip, sixteen 8-bit MACs in one `esp.vmulas.s8.qacc`, and a 256-element
dot product both ways for a price tag on gap 1:

```
MARK PIE_MAC lanes=16 distinct=0000ffff expect=0000ffff sum_ok=1
MARK PIE_DOT scalar=3045 in 2390 cyc, vector=3045 in 1004 cyc, agree=1
```

Same answer, 2.4× fewer cycles, naive port — and the compiler would never have written it.
What you give up: no operand checking, registers pinned by hand (`"{a0}"`), and the q registers
are invisible to LLVM, so keeping one live across two `asm` blocks is only sound while nothing
else in the image touches them.

## Known limits

* The CPU runs at whatever the bootloader left it at — measured 90 MHz, not 360. Raising it means
  programming the PLL and the MSPI timings; nothing here does that yet.
* No PSRAM init, no cache tuning, no interrupt controller setup, no FreeRTOS. That is the point of
  the floor, but it means `esp_wifi` and friends are not available: the radio on this board is an
  ESP32-C6 reached over SDIO by ESP-IDF's `esp_hosted`, which is a much larger dependency.
* The flasher speaks the ROM protocol only. No software stub, so no compressed writes; for a 1 KB
  image that costs nothing, and for a 4 MB one it would cost about a factor of two.
* Linux only: `tools/serial.zig` uses `termios2` and Linux ioctl numbers directly.