Architecture: how dau-build composes work from Hydra config

This page explains why dau-build is built the way it is — declarative specs, typed ccflow models, and a Hydra config tree — and what that structure buys you. It is background reading. For the commands themselves, see the command reference; for step-by-step goals, the how-to guides.

The shape of the tool

dau-build turns a declarative description of an FPGA build into concrete artifacts: generated SystemVerilog, artifact bundles, backend handoff manifests, Tcl scripts, and command plans. Everything the tool does is expressed as a typed ccflow.CallableModel — a SimulateTask, a SynthesizeTask, a BuildVivadoArtifactsTask, and so on — and every one of those models is built from configuration, not hand-constructed in Python.

That configuration is a Hydra config tree under dau_build/config. Running the tool is therefore always the same two-phase act: compose a config from groups and overrides, then instantiate and run the model it describes. The CLIs are thin front-ends over that one idea.

The reason for this design is that an FPGA build has many axes that vary independently — which task, which board, which backend, which spec — and those axes are reused across dozens of operations. Encoding each axis as a Hydra config group lets any combination compose without a bespoke argument parser per command, and lets a downstream package add new options without editing dau-build.

Config groups are directories

Each subdirectory of dau_build/config is a Hydra config group: task, step, spec, board, backend, platform, design, callable. Selecting an option is an override <group>=<option>, where the option is the file’s path relative to the group directory. A file at config/task/tasks/sim/simulate.yaml is selected as task=tasks/sim/simulate. The names are path-style because the groups nest; short aliases are deliberately not supported, so a name always says where its file lives.

Each option file opens with a # @package <key> directive that decides where its content lands in the composed config. Tasks and steps declare # @package model, so a selected task becomes the model that gets run. Boards, backends, specs, and platforms declare their own singular key. The base config config/base.yaml lists every group as optional … null, so nothing is selected until you override it, and a run picks exactly one of task= or step= to fill model. See the config group reference for the full list.

Because groups are just directories and options are just files, the config tree is the catalog. Adding a task is adding a file; the name→class registry is derived by globbing the tree (_model_types_from_config_group), so there is no second place to register it.

The override syntax

dau-build composes the config tree and runs the result, so a single command carries three kinds of override. task=<path> (or step=<path>) selects the runnable and composes it into the model key. <group>=<option> selects a config group — spec=specs/identity, board=boards/dau/dpv1, backend=backends/vivado. And model.<field>=value sets a field on the selected model. The model. prefix is not decoration: because the task is composed into the model key, its fields are addressed under model..

Names are path-style (tasks/sim/simulate, backend=backends/yosys) because they are paths into the config tree — the same reason adding a task is adding a file. Passing --explain composes the overrides without running, printing the resolved config so you can see exactly what they produced.

ccflow owns ordering and caching

A composed model is run by ccflow, not by dau-build directly. The callable group wires a MultiEvaluator of a GraphEvaluator and a MemoryCacheEvaluator. Composite tasks declare their prerequisites through Flow.deps, and the GraphEvaluator walks that dependency graph so the pieces run in the right order; the MemoryCacheEvaluator prevents a shared dependency from executing twice in a single process. This is why, for example, build-vivado-artifacts can depend on staging and validation without dau-build hand-coding the sequence — the ordering lives in the model graph, and the evaluator owns it.

The search path makes the tree extensible

dau-build registers its own config tree on the Hydra search path through a hydra.lernaplugins entry point (pkg:dau_build.config). Any installed package can do the same for its own pkg:<name>.config, and its groups then compose uniformly alongside the packaged ones — new boards, backends, designs, or tasks without touching dau-build.

The mechanism is honored by the lerna search-path bridge (lerna on PyPI); the entry point is inert without it, so a package that relies on cross-package composition depends on lerna directly. The private dau package is the working example: it registers pkg:dau.config, which adds task=dpv1-shell, and with dau installed dau-build task=dpv1-shell resolves that task with no --config-dir overlay. dau-build never imports dau — extension flows one way, through the search path, which keeps the public tool free of private dependencies. See Extending dau-build for how to do this yourself.

Plans first, execution on the host

Most build and hardware tasks default to execute: false and emit plans: generated Tcl, backend manifests at build_status=planned, staged work directories, and ordered command sequences — all without invoking a vendor toolchain or touching hardware. Passing execute=true runs the privileged action.

This separation is deliberate. It lets the whole pipeline be composed, inspected, validated, and tested on a developer machine with no Xilinx tools and no board attached, and confines the privileged, irreversible steps (running Vivado, programming JTAG, rescanning PCIe) to an explicit opt-in on the machine that actually has the hardware. The validators (validate-vivado-artifacts) check that a plan is internally consistent — manifest, Tcl, command plan, and output paths all agree — before anyone spends an hour of synthesis on it.

Board, platform, and backend are three separate things

These three groups are easy to conflate; they answer different questions.

  • platform (PlatformDefinition) is the hardware board as data: the part number, the resource budget, the memory, and the host link — including the full XDMA personality. For dpv1 the personality is the 47 proven bring-up XCI parameters, quoted verbatim and order-preserved so the generated Vivado CONFIG.* block is byte-for-byte identical to the known-good core. A hand-picked subset of those parameters leaves the device memory-dead on hardware, which is why the definition insists on carrying all of them exactly. fits() checks a design’s resource use against this budget.

  • board (BoardConfig) is a small build-config view — name, platform, shell — where platform and shell here are backend labels (e.g. vivado-xdma, xdma-ddr) threaded into Vivado manifests, not the hardware PlatformDefinition.

  • backend (BackendConfig) is the synthesis toolchain, as a label.

They are independent groups, composed together into a task’s ResolvedBuildConfig only where a task needs them. Keeping the hardware truth (platform) separate from the backend labels (board, backend) means the same physical dpv1 definition can be reused regardless of which backend produced a bitstream.

Backends: Vivado and yosys

Two synthesis engines are real: Vivado (the FPGA bitstream flow) and yosys (open-source synthesis). The engine is chosen the hydra-first way — it is the backend config group. Each option instantiates a polymorphic SynthesisEngine model, and SynthesizeTask just delegates to engine.synthesize(...). There is no engine Literal and no if engine == dispatch branch; adding an engine is adding a model and a yaml.

  • backend=backends/vivado (the default) composes a VivadoEngine, which writes a Vivado handoff — vivado_backend.py generates text artifacts (overlay Tcl, build Tcl, a key=value manifest, a command plan) but never spawns Vivado; that happens later in the execute=true build tasks and in hardware_plan.py. Vivado is not present in CI, so this path is plan-only there.

  • backend=backends/yosys composes a YosysEngine, which runs synthesis. yosys_backend.py generates a yosys script and executes it, so the generated top is actually elaborated and synthesized — a real check, not a plan. yosys is open-source and installs in CI, which is the point: it turns synthesis into something the test suite can exercise on every run.

Because the engine is a composed model, it is fully hydra-configurable. The YosysEngine’s SystemVerilog frontend is a field, set like any other override: backend=backends/yosys backend.frontend=slang (or +backend.<field>=...). frontend=verilog uses yosys’s built-in read_verilog -sv (enough for dau-build’s own synthesizable sources); frontend=slang uses the yosys-slang plugin’s read_slang, the same slang engine as the project’s pyslang parser, for the full SV surface (packages, interfaces) the private cores use.

The engine model carries only name/invocation up to the resolved-config view (ResolvedBuildConfig normalizes it to that label), so an engine’s own fields never leak into the build-config reporting. A third engine (a nextpnr place-and-route flow, another vendor) is a new SynthesisEngine subclass and a yaml; the config-group plumbing, the open registry, the _target_ indirection, and the search-path extension are all engine-agnostic, so it can even come from a package without touching dau-build — Extending dau-build walks through exactly what a new engine requires, using yosys as the worked example.