diff options
| author | Gabriel Schneider <[email protected]> | 2026-08-25 12:40:53 -0300 |
|---|---|---|
| committer | Gabriel Schneider <[email protected]> | 2026-08-25 12:46:51 -0300 |
| commit | f5f8068fac59b4f16046c2022c2fc7c7e447ef4c (patch) | |
| tree | 2731a3ed4e51cae09e184e25778eded5fc37d1f5 /README.md | |
| download | esp32p4-f5f8068fac59b4f16046c2022c2fc7c7e447ef4c.tar.gz esp32p4-f5f8068fac59b4f16046c2022c2fc7c7e447ef4c.zip | |
zig-p4: pure-Zig ESP32-P4 toolchain
build.zig generates the linker script and drives Zig's own LLD; tools/image.zig
turns the ELF into a flashable image and tools/{rom,serial}.zig speak the mask
ROM loader over the UART. No CMake, ninja, idf.py, esptool, or external linker.
src/soc.zig is a comptime register model over ESP-IDF's own *_reg.h headers;
src/hal/ adds peripheral sequences; src/io/ implements std.Io for the chip;
src/oracle/ diffs this HAL against ESP-IDF's on the die.
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 378 |
1 files changed, 378 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..6e15730 --- /dev/null +++ b/README.md @@ -0,0 +1,378 @@ +# zig-p4 — an ESP32-P4 toolchain that is just Zig + +``` +zig build # compile, link, and emit a flashable image +zig build flash # ...then write it to the chip and run it +zig build run # flash, then print the console (ordered; `flash monitor` is not) +zig build monitor # reset the board and print its console +zig build reset # just pulse the reset line +zig build size # where every byte of the image went +zig build test # host tests: image builder, and the register layer's field arithmetic +zig build diff # the hardware oracle: this HAL vs ESP-IDF's, on the die (needs -Doracle) +zig build elf # stop at the ELF, for disassembly +``` + +No CMake, no ninja, no `idf.py`, no `esptool`, no external linker. Zig's own LLD does the link +against a linker script this `build.zig` generates; the image builder and the serial flasher are +ordinary Zig code in `tools/`, imported straight into `build.zig`, so they leave no artefacts of +their own. What lands in `zig-out` is one file: the image. + +## Why it exists + +| | ESP-IDF blink | earlier Zig proof-of-concept | this | +|---|---|---|---| +| flashed image | 184,112 B | 66,176 B | **1,216 B** (432 B for `examples/minimal.zig`) | +| build, cold | 23 s, 1,072 ninja edges | 1.3 s | **1.37 s** | +| build, warm | ~1 s | 0.07 s | **0.088 s** (cache hit, no work) | +| build, one file edited | ~1 s | 0.07 s | **0.121 s** | +| flash + verify + run | ~2 s (esptool + stub) | ~2 s | **0.277 s** | +| host dependencies | ESP-IDF 663 MB + toolchain 3.4 GB + Python | Zig + system LLD + esptool | **Zig** | +| artefacts per build | ~1,100 files | 2 | **1** | + +The 66 KB → 1 KB step is the interesting one. `esptool` refuses to put two flash-mapped segments +inside the same 64 KiB MMU window (`bin_image.py:832-838`) and pads the image out to the next +window, which for a small program is 64 KiB of zeros. The chip only requires that each mapped +segment satisfies + +``` +(offset of its data within the image) % 64 KiB == (its load address) % 64 KiB +``` + +(`esp_image_format.c:903-909`), and the MMU is perfectly happy to point two entries — or the same +entry twice — at one flash page; stock ESP-IDF already aliases a page that way on every boot. +`tools/image.zig` therefore packs both segments into one window and the pad disappears. Verified on +ESP32-P4 rev v1.3 silicon. + +## Requirements + +* Zig 0.16.0. That is the whole list. +* Membership of whatever group owns the serial port (`uucp` on Arch, `dialout` on Debian). +* An ESP-IDF second-stage bootloader and partition table already in flash at `0x2000` and `0x8000`. + This toolchain builds and flashes *applications*; the bootloader is still Espressif's. See + "What is still ESP-IDF" below. + +## Options + +Everything is a `b.option`, so `zig build -h` lists them all. + +| option | default | meaning | +|---|---|---| +| `-Dapp=<path>` | `src/main.zig` | application root source | +| `-Dport=<path>` | `/dev/ttyUSB0` | serial port | +| `-Dbaud=<enum>` | `b921600` | flashing baud. `b2000000` does not work on this board's CH340 | +| `-Dled=<u8>` | `20` | GPIO the demo blinks (20 = JP1 pin 17) | +| `-Doffset=<u32>` | `0x10000` | flash offset of the app partition | +| `-Dflash-size=<enum>` | `16MB` | fitted flash, written into the image header | +| `-Dmin-rev`/`-Dmax-rev` | `100`/`199` | silicon revision window. The pre-v3 P4 needs 100..199 | +| `-Ddescriptor=<enum>` | `minimal` | 184-byte descriptor, or `full` for the 256-byte one `esptool image-info` can parse | +| `-Dstack=<u32>` | `8192` | stack size; the generated linker script follows | +| `-Dverify=<bool>` | `true` | ask the ROM for an MD5 of what it stored and compare | +| `-Delf=<bool>` | `false` | also install the ELF | +| `-Doptimize=<mode>` | `ReleaseSmall` | firmware default, not Debug (Debug costs ~780 B here) | +| `-Dseconds=<u32>` | `5` | how long `monitor` listens | + +## Layout + +``` +build.zig the toolchain: target, generated linker script, and four custom steps +tools/image.zig ELF -> ESP image. Header, segments, congruence filler, checksum, SHA-256 +tools/image_test.zig 8 host tests, one per rule the ROM bootloader enforces +tools/rom.zig SLIP framing + the ROM loader protocol. No software stub +tools/serial.zig termios2 raw mode, arbitrary baud, DTR/RTS reset dance +src/soc.zig comptime register model: GPIO, IOMUX, mask-ROM entry points, cycle counter +src/appdesc.zig esp_app_desc_t, linked as its own object so it cannot be optimised away +src/main.zig demo: prints what it can prove, then blinks +examples/minimal.zig the floor: 432 B, blinks and nothing else +``` + +## What adversarial review found + +Five reviewers went at this in two rounds; sixteen findings were applied. The ones worth knowing +about, because each is a trap the next person will hit too: + +* **The ROM's status byte is at `data[len-4]`, not `data[len-2]`.** The four-byte trailer is a + ROM-versus-stub difference (esptool `loader.py:653-655`). Reading the wrong byte made *every* ROM + error read as success: a rejected `FLASH_DATA` block was never retried and `zig build flash` + printed a cheerful success line over a dead image. Only the MD5 pass caught it. Now the offset is + right, the length check is mandatory, and `-Doffset=33554432` (past the end of a 16 MB part) + fails with `write failed: CommandFailed` instead of claiming victory. +* **Congruence modulo the MMU page is necessary but not sufficient.** The first solver satisfied it + by shifting a segment forward, which could leave both mapped segments in one *vaddr* page while + their data sat in two different *flash* pages. The bootloader writes one MMU entry per vaddr page, + so the second mapping replaced the first and the app booted with every constant reading as zero - + no error anywhere. `-Ddescriptor=full` triggered it. The solver now anchors every mapped segment + to a single `flash_address - load_address`, and `validate` checks the invariant directly. +* **The target enables `f` but nothing enabled the FPU.** `mstatus.FS` is Off after reset and + ESP-IDF only turns it on lazily from a trap handler, which this image does not have, so the first + `f32` multiply in application code was an unhandled illegal instruction. `_start` now sets FS, and + the demo prints a float computed with hardware `fmul.s` as proof. +* **`-Dmin-rev` was feeding the descriptor's eFuse *block* revision**, an unrelated field, so + `-Dmin-rev=150` produced an image the bootloader refuses with `Image requires efuse blk rev >= v0.50`. +* **`flash` and `monitor` are unordered top-level steps** and the build runner runs independent + steps concurrently, so `zig build flash monitor` could pull the chip out of download mode + mid-write. They now share a mutex - and because a mutex cannot express *order* (measured: monitor + won 3 times out of 3, printing the old firmware before the new one was written), `zig build run` + exists as the ordered version. +* **A cache manifest that does not hash the builder is worse than no cache.** The first version + hashed only the ELF and the options, so editing `tools/image.zig` produced a cache hit and + shipped the previous image - and `--watch` never noticed the edit at all. The manifest now covers + the builder sources too. +* **`ALIGN(64)` does not reserve a hole.** The image builder needs 8 spare bytes before the second + mapped segment for its header; for one rodata length in eight, 64-byte alignment leaves 0 or 4, + and the build failed with `MappedSegmentsTooClose`. The generated script now ends `.flash.rodata` + with `. = ALIGN(. + 8, 64) - 8;`, verified over a sweep of rodata sizes. +* **A constant the compiler folds proves nothing.** The FPU check computed `7 * 1.5 + 0.25` from a + literal, so LLVM folded it and the image contained no float instructions at all: the test would + have passed on a board whose FPU was still off. It now loads through a volatile pointer, and the + image really does contain `fcvt.s.wu`, `fmul.s`, `fadd.s`, `fcvt.wu.s`. + +## Three things that bit, and are now encoded in the code + +1. **`standardOptimizeOption` hands out Debug builds.** With `preferred_optimize_mode` set it + exposes `-Drelease` and still defaults to Debug, which for this target means panic machinery and + formatting code inside a 500-byte image. `build.zig` takes `-Doptimize` and defaults to + `ReleaseSmall` explicitly. +2. **An `@import`ed descriptor disappears under ReleaseSmall.** A `comptime _ = descriptor;` + reference is enough in Debug; in release the constant folds away, `.flash.rodata` vanishes, + `KEEP` has nothing to keep, and the image boots with one mapped segment at the wrong offset. + The descriptor is now a separate object linked unconditionally. +3. **`tcsetattr` cannot set the baud.** `TCSETS`' struct has no `ispeed`/`ospeed`, so assigning + those fields silently does nothing and the port stays at whatever rate it had — a 1.5 KB flash + took 230 ms instead of 60. `tools/serial.zig` uses `TCGETS2`/`TCSETS2` with `BOTHER`, and + declares `struct termios2` itself because std's version is 60 bytes where the kernel's is 44, + which makes std's ioctl numbers wrong. + +## What is still ESP-IDF + +The second-stage bootloader at `0x2000` and the partition table at `0x8000`. Both are ordinary +flash contents and neither is rewritten here. The bootloader is what enforces the three rules this +toolchain obeys: + +* exactly two segments in the mapped range (`bootloader_utility.c:842`) +* every segment length a multiple of 4 (`esp_image_format.c:857`) +* mapped segments congruent modulo the MMU page (`esp_image_format.c:903-909`) + +`tools/image.zig` re-checks all three after building an image, so a violation is a build error +rather than a board that resets in a loop. + +## The HAL: registers from ESP-IDF, sequences in Zig + +A toolchain that can only blink an LED is a demo. The rest of the chip needs a hardware layer, and +the ESP32-P4 has a lot of chip: 96 `*_ll.h` headers in ESP-IDF v6.0.2 holding **3,081** inline +functions over **5,170** registers and **20,052** fields. Hand-transcribing that is not a plan — +the hand-written predecessor of `src/hal/gpio.zig` had a wrong matrix constant with a comment +warning about exactly that mistake. + +So the register layer is not written at all. `build.zig` runs `zig translate-c` over a generated C +file that `#include`s every one of ESP-IDF's own `*_reg.h` headers, and the result is imported as a +module: + +```zig +const regs = @import("regs"); // 87,373 constants, straight from IDF's macros +``` + +That takes 0.26 s and costs about 0.16 s of parse on top of a 1.7 s cold build; Zig only analyses +the declarations actually referenced, so the unused 87,000 are free. Every address, shift and mask +in this HAL is therefore not a re-derivation of ESP-IDF's number — it *is* ESP-IDF's number, as +evaluated by clang. `*_struct.h` is deliberately unused: translate-c demotes each of those register +structs to `opaque {}` ("has bitfield"), so the C bitfields buy nothing. + +`src/mmio.zig` is the ~200 lines that turn flat constants into checked accessors: + +```zig +const conf0 = mmio.Reg.at(regs.LEDC_CH0_CONF0_REG); +const timer_sel = mmio.Field.of(regs.LEDC_TIMER_SEL_CH0_S, regs.LEDC_TIMER_SEL_CH0_V); +conf0.modify(.{ timer_sel.is(2) }); // read-modify-write, preserving the rest +``` + +Fields are built from the `_S`/`_V` pair and never from `_M`, which is not a style rule: **153 `_M` +macros are broken C inside ESP-IDF itself** (`INTERRUPT_CORE0_LP_RTC_INT_MAP_M` expands +`CORE0_LP_RTC_INT_MAP_V`, dropping the prefix). Nothing in C ever expanded them, so nobody noticed; +`translate-c` surfaces them as poisoned declarations. `Field.of` also rejects a pre-shifted mask at +comptime, so passing `_M` by hand is a build error rather than a wrong bit position. + +`modify` is the default and `write` is the exception, because `write` zeroes what it does not name +and 46.7% of the registers a low-level driver touches have a field whose reset value is not zero. + +### The build refuses to hide what it could not translate + +`translate-c` exits 0, prints nothing, and still emits `pub const X = @compileError(...)` for every +macro it could not handle — invisible until a driver names one. So a build step counts them and +fails if the number grows past a recorded 524, whose composition is written down: 333 register +addresses whose `DR_REG_*_BASE` **ESP-IDF references and never defines anywhere**, 153 broken `_M` +masks, and 38 function-like macros with Zig equivalents. Three of those missing bases were +recovered from `esp32p4.peripherals.ld` — which cross-checks against `reg_base.h` on all 65 +peripherals both files name, 0 disagreements — putting 176 registers back. The rest stay +unreachable on purpose rather than reachable at a guessed address. + +### The oracle: differential testing against ESP-IDF, on the die + +Register numbers being right does not make a *sequence* right. So ESP-IDF's own `*_ll.h` functions +are compiled by Zig's clang into the same image as this HAL, and `zig build diff` drives each +operation both ways on the chip and compares the register block afterwards: + +``` +MARK DIFF_CFG gpio_ll_uses_rom_api=0 expect=0 +MARK DIFF ok gpio.set_level(1) 400 words identical +... +MARK DIFF_TOTAL cases=26 failures=0 +``` + +It found two real bugs in this HAL on its first run. `GPIO_FUNC0_OEN_SEL` reads backwards from its +name — 1 means "use `GPIO_ENABLE_REG`", 0 means "use the peripheral's own output enable" — so +`matrixOut` had it inverted and then set the matching `GPIO_ENABLE` bit to compensate. It worked, by +the wrong mechanism, leaving the pad latently output-enabled. The first version of the harness also +only compared 112 words and so saw the symptom without the cause; the window now reaches the matrix +configuration registers at +0x558. + +Four things the harness has to get right, each measured on this board rather than assumed: + +* **A snapshot can have side effects.** `UART_FIFO_REG` is at offset 0x000 of every UART block — the + first word a "read the whole block" loop touches — and reading it pops the RX FIFO. The header + annotates it `RO`. Peripherals declare offsets that must not be read. +* **A block cannot be restored by writing its snapshot back.** ~10% of this chip's fields act when + written; writing one saved word back to a UART's offset 0 transmits a character. Restore is the + peripheral's reset bit, or a deliberate configure function. +* **A clock-gated block reads stale data, silently** — the last value latched, not zeros, so two + meaningless snapshots can compare equal. The bus clock is checked before every comparison. +* **Equal registers do not prove equal sequences.** LEDC commits shadow registers through a + self-clearing bit that leaves no trace afterwards. + +### What is ported + +| peripheral | what it covers | differential cases | +|---|---|---| +| `hal.gpio` | 57 pins across both banks, IO MUX pads, pulls, drive strength, open drain, the GPIO matrix in and out | 46 | +| `hal.timg` | TIMG0/1, two timers each: dividers, direction, auto-reload, alarms, the latch-then-read counter, and MWDT behind its write-protect key | 29 | +| `hal.uart` | UART0-4: the fractional baud divider, data format, FIFOs, loopback, pin routing, and the `_SYNC`/`REG_UPDATE` commit | 19 | +| `hal.intr` | the CLIC (not a PLIC): the 122-source interrupt matrix, per-line enable/trigger/priority, the memory-mapped threshold, a 48-entry vector table | 20 | +| `hal.ledc` | Q10.8 timer dividers, channels, duty, idle level, the shadow-register commits, pin routing | 28 | +| `hal.i2c` | I2C0/1 master: the full ten-register timing set plus its out-of-block clock divider, a typed command list, FIFOs, transaction status | 43 | +| `hal.clkrst` | peripheral clock gates and resets, under an interrupt-masked read-modify-write guard | 8 | +| `hal.systimer` | the two 52-bit counters, through their update/valid handshake | — | +| `hal.rwdt` | the RTC and super watchdogs, which the bootloader leaves armed | — | + +**193 cases, 0 failures** on the die. + +That number was 177 until an adversarial audit of the harness pointed out that about seventy of the +cases could not fail for any implementation error, which is worth more than the passing count was. +Two structural causes, both now fixed: + +* **A window that missed the registers under test.** Every pad-configuration function writes the IO + MUX at `0x500E1004 + 4*pin`, and GPIO's compared window ended at `0x500E063F` - 0xC00 bytes short. + Eight operations across two pins were comparing two identical snapshots of a register file none of + them touch. `iomux_suite` is the same cases against the right window. +* **Restores written with the code under test.** The harness runs restore, IDF, snapshot, restore, + ours, snapshot. If restore calls the HAL, run B starts from whatever IDF just wrote, and a HAL + function that does nothing at all compares equal - so each suite was blind to exactly the failure + it existed to catch. `clkrst`'s restore called `setClockEnabled`, which is the function whose + wrong-register bug started this whole line of work. Every restore is now built from register + macros or from ESP-IDF's own LL, never from ours. + +`src/hal` and `src/mmio.zig` are 4,590 lines; the oracle that checks them is 4,107. + +`examples/halcheck.zig` exercises the first three on hardware directly. One number out of it is worth +keeping: the CPU runs at **89,995 kHz**, measured by counting cycles against the systimer's fixed +16 MHz over 50 ms rather than estimated — two independent clocks, not one clock and an assumption. + +### Three traps now encoded in the code + +**Peripheral clocks are already on.** `esp_system/port/soc/esp32p4/clk.c:200` says so and the reset +defaults agree, so the hazard is not gating but atomicity: every gate and reset bit shares a register +with unrelated peripherals, and ESP-IDF makes an unguarded call *uncompilable* by referencing an +identifier it never defines. `clkrst.maskInterrupts()` is the replacement for the spinlock IDF uses. + +**Resetting a timer group re-arms flash-boot watchdog protection**, which reboots the board a moment +later with nothing on the console to explain it. `resetPeripheral(.timg0)` clears it as part of the +reset. + +**The bootloader leaves the RTC watchdog running**, and expects the application to take it over — +which `esp_system` does and a bare image never did. Every demo in this repo had been resetting on a +ten-second cycle since the first one, invisibly, because no run had ever lasted eight seconds. It +surfaced when the differential grew past 64 cases and stopped fitting inside one watchdog period, +where it looked exactly like "the newest suite crashes the board". `hal.rwdt.disable()` is the fix +and `hal.rwdt.armed()` is the thing to print at startup. + +## Reproducing the report's RF experiment + +`04-report` section 5 is the only real SDR measurement in that document: a 25 MHz carrier on GPIO20, +OOK-keyed with `0x4200`, detected at +12.96 dB and decoded back from raw IQ. `examples/rf.zig` is that +emitter driven by this HAL instead of ESP-IDF, and the report's own capture and decode tools are used +unmodified. Full write-up and the numbers: `captures/RF-REPRODUCTION.md`. + +The emitter reproduces exactly. The report derives its carrier from the divider — 1-bit resolution +off 80 MHz can only give `80e6 × 256 / (410 × 2)` = 24,975,609 Hz, never a round 25 MHz — and this +HAL computes divider 410 and 24,975,610 Hz from IDF's own Q10.8 arithmetic. The source clock was then +measured rather than assumed: 80,104,000 Hz, timed against the systimer. The differential harness +gained a case that runs the whole bring-up both ways in one image, and the LEDC block comes out +**96 words identical**; an edge count on the pad agrees to 2 parts in 112,500 at every rate the +instrument can measure. + +**The detection does not reproduce, and not because of this toolchain.** ESP-IDF's own binary was +reflashed and driven by the report's own tool with the report's parameters: + +| firmware | ON − OFF | +|---|---| +| report, §5.2 | **+12.96 dB** | +| ESP-IDF, re-run today | −0.063 dB | +| this toolchain, today | +0.05 dB | + +The receive chain checks out — an FM carrier sits 31.8 dB above the span median, against the report's +30.9 dB — so the instrument is fine and both firmwares now give the same null. The report says its +emission reached the dongle through whatever wire happened to be on JP1 pin 17; that coupling is +gone. A measurement whose apparatus is incidental coupling is not reproducible, and this one is not. +That is a correction to the report rather than a difference between toolchains. + +Wi-Fi, BLE and 802.15.4 stay out of scope by construction: the P4 has no radio, and those went over +an ESP32-C6 across SDIO under `esp_hosted` plus a prebuilt coprocessor binary. Reproducing them means +porting `esp_hosted`, not porting a HAL. + +## The vendor ISA extensions, from a stock Zig + +The P4's cores carry Espressif's `xespv2p1` vector unit ("PIE") and the `xesploop1p0` hardware +loops, which upstream LLVM does not know. That is really two gaps: + +1. **The optimiser will never choose one.** No cost model, no intrinsics, no autovectorisation. + That ceiling does not lift — but Espressif's own fork does not autovectorise into this unit + either, which is why `esp-dsp` is hand-written assembly. +2. **The assembler cannot spell one.** Inline assembly does *not* help; it goes through the same + integrated assembler: `asm volatile ("esp.vld.128.ip q0, a0, 16")` → + `error: <inline asm>:1:2: unrecognized instruction mnemonic`. This is what stops Zig from + assembling ESP-IDF's FreeRTOS context switch (`portasm.S`, 34 instructions under + `#if SOC_CPU_HAS_PIE`). + +Gap 2 closes without forking a compiler: `.insn` accepts any word, so generate the words offline +and pin the registers the fixed encoding names. `tools/encode.sh` is that oracle — it drives +Espressif's GAS at build-authoring time, never at build time, and prints a `.insn` line per +mnemonic. All 34 of `portasm.S`'s instructions encode, and the vld/vst/qacc words match what +Espressif's objdump reads out of an ESP-IDF build of this board byte for byte. + +Watch the version: the extension is versioned and the versions disagree. +`esp.vld.128.ip q0, a0, 16` is `0x0311009f` under `xespv` (which defaults to 1.0) and `0x0201223b` +under `xespv2p1`. The P4 wants the latter — the pair in every IDF image's `.riscv.attributes` here +is `xesploop1p0_xespv2p1`. + +`examples/pie.zig` runs the whole thing on the die: enable the unit (`csrw 0x7f2, 1`), a 16-byte +vector load/store round trip, sixteen 8-bit MACs in one `esp.vmulas.s8.qacc`, and a 256-element +dot product both ways for a price tag on gap 1: + +``` +MARK PIE_MAC lanes=16 distinct=0000ffff expect=0000ffff sum_ok=1 +MARK PIE_DOT scalar=3045 in 2390 cyc, vector=3045 in 1004 cyc, agree=1 +``` + +Same answer, 2.4× fewer cycles, naive port — and the compiler would never have written it. +What you give up: no operand checking, registers pinned by hand (`"{a0}"`), and the q registers +are invisible to LLVM, so keeping one live across two `asm` blocks is only sound while nothing +else in the image touches them. + +## Known limits + +* The CPU runs at whatever the bootloader left it at — measured 90 MHz, not 360. Raising it means + programming the PLL and the MSPI timings; nothing here does that yet. +* No PSRAM init, no cache tuning, no interrupt controller setup, no FreeRTOS. That is the point of + the floor, but it means `esp_wifi` and friends are not available: the radio on this board is an + ESP32-C6 reached over SDIO by ESP-IDF's `esp_hosted`, which is a much larger dependency. +* The flasher speaks the ROM protocol only. No software stub, so no compressed writes; for a 1 KB + image that costs nothing, and for a 4 MB one it would cost about a factor of two. +* Linux only: `tools/serial.zig` uses `termios2` and Linux ioctl numbers directly. |
