The core · features 010 and 013

On a ULX3S

The SoC, core and devices together, as a bitstream for a ULX3S board with Lattice's ECP5-85F, built with the open toolchain alone: Yosys, nextpnr-ecp5, ecppack and openFPGALoader. Its program sits in block RAM from the bitstream, its console is the board's USB serial port, and sysk boots and runs its shell on it in a simulation of the board from power-up.

What is in it

Almost all of the board is sysr. boards/ulx3s/ulx3s.sysr is the board's top: the SoC (virt, from cores/rv64im/soc.sysr) with its parameters, the serial input through two flip-flops (it is asynchronous to the clock), the LEDs, and the ESP32's GPIO0 held high so the radio module does not reset the board. Its parameters are the RAM's size, the UART's rate and mtime's rate, and clock-hz says what the clock is; tools/platform.rkt reads all of them to generate platform/ulx3s (Devices and the platform).

LEDs: 0 is lit once the test device has halted, 1 shows the line out, 2 the line in, 3 to 6 the halt code's low bits, and 7 is a heartbeat, bit 24 of a free-running counter (about 0.45 Hz).

• top.v is the only hand-written logic: the pins under the names of the board's constraint file, the ECP5's PLL (an EHXPLLL, settings from ecppll: 25 MHz / 5 × 3 × 40 = 600 MHz inside, / 40 = 15 MHz), and a reset while FIRE1 is pressed or the PLL has not locked, through two flip-flops. sysr's generated registers start at their reset values from configuration, so nothing else needs a reset pulse.
• ulx3s.lpf places those pins: the 25 MHz oscillator, the FTDI chip's two serial lines, eight LEDs, seven buttons, GPIO0 (sites from the board's ulx3s_v20.lpf).
• hello.s is the default program: a banner, then every line typed comes back upper-cased.
• tb_board.v simulates the board top as it starts on the board, never reset, at 15 MHz, playing the far end of the serial line at 130 cycles per bit (make ulx3s-sim).

From sources to a bitstream

make ulx3s runs the chain below; make ulx3s-prog loads the result into the FPGA over USB (until power-off). Each step is a Makefile rule, quoted from the Makefile as it is. Step through it.

PlayBackStep

With sysk in the RAM, nextpnr-ecp5 reports, per the board's README: 19,146 LUT4 (23% of the 85F), 2,014 flip-flops, 192 of 208 DP16KD block RAMs, 36 multipliers and one PLL, and 16.7 MHz after routing against the 15 MHz needed. The core does one instruction a cycle with nothing pipelined, so its longest path runs from the block RAM's output (the instruction) through decode and execute into the next access, about 60 ns, two thirds of it routing across the 192 RAM blocks. A faster clock needs a pipelined core, not a better placement. Place and route takes about 20 minutes.

The program in block RAM: eight byte lanes

The core's RAM is eight memories, one per byte of a doubleword, so a store can write any of its bytes (b0 to b7 in rv64im.sysr's ram). Yosys maps each to block RAM, and the bitstream can hold their initial contents. tools/lanes.rkt turns a program, an assembly file or an ELF executable, into eight hex files, one per lane, each with ram-words lines; byte k of the image goes to lane k mod 8, line k div 8. sysr build --mem-init b0=.../b0.hex … puts them into the generated Verilog as $readmemh. A program larger than the RAM is an error. Type some bytes to see where they land; the default is hello.s's banner, read from its .word line.

Bytes from 0x80000000

Select a byte.

Two choices made 384 KB fit and meet timing. Each lane is one copy: the fetch port now sees bytes stored in the same cycle through a bypass outside the RAM, which Yosys recognizes, where reading the old data would have taken two copies of each lane. And 384 KB is 49,152 doublewords, which Yosys builds from 16384 × 1-bit blocks, so each bit of a doubleword is one of three blocks behind a small multiplexer; at 400 KB only 2048 × 9 blocks fit, each bit is one of 25, and routing did not finish. The RAM may be any number of doublewords: since feature 013 an access past its end is an access fault in the core, and the RAM itself reads word 0 there and writes nothing, so a kernel that runs out of RAM traps instead of overwriting its start.

Clock and timing

The board's oscillator is 25 MHz, but place and route found the single-cycle core's longest path too slow for it, so top.v's PLL makes 15 MHz. The UART's div is 130: 15,000,000 / 130 is 115,385 bits per second, 0.2% above 115,200, well within what a serial port accepts.

sysk's scheduler counts time in mtime ticks, and QEMU's virt machine counts mtime at 10 MHz. So that a 1 ms quantum is 1 ms on the board too, the core counts mtime on tick-num of every tick-den cycles, evenly: a register tick adds tick-num each cycle, and when the sum reaches tick-den it subtracts tick-den and mtime counts. The board sets 2 of 3, and the core proves tick-below: tick stays below tick-den.

tick-num tick-den PlayStepReset

sysk on the board, in simulation

Feature 013 put sysk in the board's block RAM. Two things stood in the way, and both are now in the core: its RAM had to be a power of two that wrapped around (now any size, with an end that faults), and its timer counted the 15 MHz clock (now 2 of every 3 cycles). sysk (its version 028, with the shell) fits in 384 KB once its stacks are 4 KB instead of 16 KB; the deepest any of its stacks went on the ISS was 1.5 KB.

make ulx3s ULX3S_PROG=../sysk/build/sysk.elf
make ulx3s-sim ULX3S_PROG=../sysk/build/sysk.elf ULX3S_INPUT='ls\nhello\n\004' ULX3S_CYCLES=2000000
make sysk-ulx3s ULX3S_PROG=../sysk/build/sysk.elf    # the core against the ISS, to the exit

As recorded in the board's README and feature 013: in the board's configuration (384 KB, mtime at 2 of every 3 cycles, 130 cycles per bit), sysk boots, answers ls with sys/ and hello with unknown command hello, and exits with the pass code after 53 timer interrupts, at cycle 862,075, 57 ms at 15 MHz. The core agrees with the ISS there on everything: registers, CSRs, the devices, all 384 KB of memory, every I/O store, every byte on the serial line. The board top, simulated in Icarus from power-up with its input on the serial pin from cycle 25,000, prints the same and halts at cycle 889,227.

What has not been done

• Running it on a board. Everything above is a build or a simulation.
• SDRAM. The ULX3S has 32 MB, which needs caches, a memory bus that can wait, a boot ROM, and bidirectional pins in sysr, which it does not have yet. Then sysk's stacks and heap can be QEMU's sizes again.
• A faster clock, which needs a pipelined core and, with it, a new timing model in the ISS.