Python Drives Hardware: PYNQ Overlays

In Chapter 20 you produced a bitstream with a peripheral at an address on the ARM's bus. To use it you could write a Linux kernel driver in C, or a bare-metal program. PYNQ (Python productivity for Zynq) takes a different route: it runs Linux and Jupyter on the ARM cores and lets you load a bitstream and talk to its IP blocks from a Python notebook. The bitstream plus its description is called an overlay — like a library you import, except that importing it rewires the chip.

Every code block below in this chapter runs on the board, in a notebook — not in this web page. They import the pynq package, which only exists on a Zynq running the PYNQ image. The one program that runs here, in Section 8, is a model of the timing.

Creating an Overlay object does two things: it programs the PL with the bitstream, and it parses the .hwh file to learn what is inside. That second part is what makes PYNQ convenient — you never type an address by hand. The ip_dict is a dictionary with one entry per memory-mapped IP block:

from pynq import Overlay

ol = Overlay('calc.bit')       # finds calc.hwh next to it, programs the PL
print(ol.is_loaded())          # True once the bitstream is in the PL

for name, info in ol.ip_dict.items():
    print(f"{name:20s} base=0x{info['phys_addr']:08X} "
          f"range=0x{info['addr_range']:X}  type={info['type']}")
# axil_calc_0          base=0x43C00000 range=0x10000  type=<vendor>:user:axil_calc:1.0

The base address is whatever Vivado's address editor chose in Chapter 20, and the range is the size of the window (64 KB here even though we only use 16 bytes). The type string is the IP's VLNV identifier (vendor, library, name, version) that you set when packaging it. The exact value depends on your packaging settings, so print it rather than guess it.

MMIO (memory-mapped I/O) is PYNQ's class for accessing a window of physical addresses. Under Linux, user programs cannot touch physical addresses directly; MMIO maps the window into the Python process (through /dev/mem) and then each read/write becomes one 32-bit load or store, i.e. one AXI-Lite transaction on M_AXI_GP0. Offsets are in bytes from the base, exactly as in the register map of Chapter 20:

from pynq import MMIO

CTRL, A, B, RESULT = 0x0, 0x4, 0x8, 0xC          # byte offsets (Chapter 20)

info = ol.ip_dict['axil_calc_0']
calc = MMIO(info['phys_addr'], info['addr_range'])

calc.write(A, 1234)
calc.write(B, 5678)
calc.write(CTRL, 0)                 # OP = add
print(calc.read(RESULT))            # 6912

calc.write(A, 40000)
calc.write(B, 50000)
calc.write(CTRL, 1)                 # OP = multiply
print(calc.read(RESULT))            # 2000000000

# The overlay also gives you an object per IP with the same read/write:
ol.axil_calc_0.write(A, 7)
print(ol.axil_calc_0.read(A))       # 7

For anything more than a quick test, wrap the register map in a small driver class. If a class derived from DefaultIP declares a bindto list with the IP's VLNV string, PYNQ uses it automatically for every matching block when the overlay is loaded:

from pynq import DefaultIP, Overlay

class CalcDriver(DefaultIP):
    """Python driver for the axil_calc peripheral of Chapter 20."""
    # Copy this string from ol.ip_dict['axil_calc_0']['type'] (vendor:library:name:version)
    bindto = ['user.org:user:axil_calc:1.0']

    CTRL, A, B, RESULT = 0x0, 0x4, 0x8, 0xC

    def __init__(self, description):
        super().__init__(description=description)

    def add(self, a, b):
        self.write(self.A, a); self.write(self.B, b); self.write(self.CTRL, 0)
        return self.read(self.RESULT)

    def mul16(self, a, b):
        self.write(self.A, a & 0xFFFF); self.write(self.B, b & 0xFFFF); self.write(self.CTRL, 1)
        return self.read(self.RESULT)

ol = Overlay('calc.bit')       # define the class BEFORE loading the overlay
print(type(ol.axil_calc_0))    # <class '__main__.CalcDriver'>
print(ol.axil_calc_0.mul16(300, 400))   # 120000

Now measure what one register access costs from Python. This tells you a lot about where the time goes:

import time

N = 10_000
t0 = time.perf_counter()
for i in range(N):
    calc.write(A, i)
t1 = time.perf_counter()
print(f"{(t1 - t0) / N * 1e6:.2f} us per MMIO write from Python")

Expect a few microseconds per call. The AXI-Lite transaction itself takes far less; most of the time is the Python interpreter and the MMIO method call. (Our model in Section 8 assumes 2 µs — replace it with your measurement.) Hundreds of thousands of register writes per second is plenty for control, and hopeless for moving data.

Polling a status register in a loop works, but it keeps the CPU busy. Hardware can instead raise an interrupt: a wire from the PL to the ARM's interrupt controller (IRQ_F2P on the processing system). In PYNQ designs the IP's interrupt output is connected through an AXI Interrupt Controller block to the PS, and the .hwh records the connection. PYNQ then gives the IP an interrupt attribute with an asyncio-style wait():

import asyncio

# Requires the IP's interrupt pin to reach the PS through an AXI Interrupt
# Controller in the block design; PYNQ then exposes it as ip.interrupt.
async def wait_for_done(ip):
    await ip.interrupt.wait()          # the notebook is free while we wait
    print('done, status =', ip.read(0x4))

asyncio.get_event_loop().run_until_complete(wait_for_done(ol.my_ip_0))

Our axil_calc has no interrupt (it finishes in one cycle), and the DMA engine of the next section is usually used with a blocking wait(). Interrupts matter when hardware takes a long or unpredictable time — waiting for an external event, or a long computation — and you want the notebook to stay responsive.

To feed a hardware pipeline with megabytes of data, the CPU must stop handling words one at a time. A DMA (direct memory access) engine is a small hardware block that reads or writes memory on its own. Xilinx's AXI DMA IP converts between memory and AXI-Stream: its MM2S channel (memory-mapped to stream) reads a buffer from DDR through an HP port and emits it as a stream; its S2MM channel (stream to memory-mapped) takes a stream and writes it to DDR. The CPU only programs a few registers: buffer address, length, go.

Two problems arise that MMIO never had. First, the DMA works with physical addresses and cannot follow the page tables of Linux's virtual memory, so a buffer must be one physically contiguous block. An ordinary NumPy array is not. PYNQ's allocate() reserves contiguous memory and returns it as a NumPy array (a PynqBuffer) that also knows its physical_address.

Second, cache coherence. If the buffer is cacheable, when Python writes into it the new values may sit in the ARM's cache and not yet be in DDR; the HP port reads DDR directly, so the DMA would read stale data. In the other direction, after the DMA writes results into DDR, the CPU may still hold old copies in its cache. The fixes are buf.flush() (write cache lines back to DDR) before the hardware reads, and buf.invalidate() (discard cached copies) before the CPU reads what hardware wrote. For buffers from allocate(), PYNQ's DMA driver takes care of this for you when you call transfer() and wait(); you only need to call them yourself when your own hardware masters the buffer.

import numpy as np
from pynq import Overlay, allocate

ol  = Overlay('loopback.bit')            # AXI DMA with MM2S -> S2MM looped through a FIFO
dma = ol.axi_dma_0

N = 4096
src = allocate(shape=(N,), dtype=np.uint32)   # physically contiguous, DMA-able
dst = allocate(shape=(N,), dtype=np.uint32)
src[:] = np.arange(N, dtype=np.uint32)        # use it like any NumPy array
print(hex(src.physical_address))              # the address the DMA engine will use

dma.recvchannel.transfer(dst)   # arm the receiver first (S2MM)
dma.sendchannel.transfer(src)   # then start sending (MM2S)
dma.sendchannel.wait()
dma.recvchannel.wait()

print(np.array_equal(src, dst))  # True
src.freebuffer(); dst.freebuffer()   # contiguous memory is a limited resource

The loopback overlay assumed here is the simplest possible DMA design: AXI DMA with both channels enabled and its MM2S stream connected straight back to S2MM through an AXI-Stream FIFO, MM2S/S2MM memory ports on HP0, control on GP0. Build it as an exercise with the flow of Chapter 20; Chapter 22 replaces the FIFO with an accelerator. Arm the receive channel before starting the send channel, so that the returning stream has somewhere to go.

Moving \(N\) bytes costs a fixed overhead \(t_0\) (Python calls, driver, cache maintenance, waiting) plus the time at the stream bandwidth \(B\):

A 32-bit stream at 100 MHz carries at most \(4 \times 10^8\) B/s = 400 MB/s. The model below compares that with word-by-word MMIO and, for scale, with one NumPy pass over the same data on the ARM. All constants are assumptions in the right range for a PYNQ-Z2 driven from Python; replace them with your own measurements. Two lessons survive any reasonable choice of numbers: DMA beats MMIO for anything beyond a few hundred bytes, and a DMA call only reaches its bandwidth for buffers of tens to hundreds of kilobytes.

Notice the dashed line. Even a perfect DMA only moves data at a speed comparable to NumPy processing it on the ARM for a simple operation. That is the central question of Chapter 22: an accelerator only wins if it does much more work per byte than the CPU can.