Building an x86 full-system simulation with the gem5 standard library

One of the key ideas behind the gem5 standard library is to allow users to simulate, big, complex systems, with minimal effort. This is done by making sensible assumptions about the nature of the system to simulate and connecting components in a manner which “makes sense.” While this takes away some flexibility, it massively simplifies simulating typical hardware setups in gem5. The overarching philosophy is to make the common case simple.

In this tutorial we will build an X86 simulation, capable of running a full-system simulation, booting an Ubuntu operating system, and running a benchmark. This system will utilize gem5’s ability to switch cores, allowing booting of the operating system in KVM fast-forward mode and switching to a detailed CPU model to run the benchmark, and use a MESI Two Level Ruby cache hierarchy in a dual-core setup. Without using the gem5 library this would take several hundred lines of Python, forcing the user to specify details such as every IO component and exactly how the cache hierarchy is setup. Here, we will demonstrate how simple this task can be with using the gem5 standard library.

As we focus on X86, we must must build the gem5 X86 binary:

scons build/X86/gem5.opt -j <number of threads>

To start, create a new Python file. We will refer to this as x86-ubuntu-run.py.

To begin we add our import statements:

from gem5.utils.requires import requires
from gem5.components.boards.x86_board import X86Board
from gem5.components.memory.single_channel import SingleChannelDDR3_1600
from gem5.components.cachehierarchies.ruby.mesi_two_level_cache_hierarchy import MESITwoLevelCacheHierarchy
from gem5.components.processors.simple_switchable_processor import SimpleSwitchableProcessor
from gem5.coherence_protocol import CoherenceProtocol
from gem5.isas import ISA
from gem5.components.processors.cpu_types import CPUTypes
from gem5.resources.resource import Resource
from gem5.simulate.simulator import Simulator
from gem5.simulate.exit_event import ExitEvent

As in other Python scripts, these are simply classes/functions needed in our script. They are all included as part of the gem5 binary and therefore do not need to obtained elsewhere.

A good start is to use the requires function to specify what kind of gem5 binary/setup is required to run the script:

requires(
    isa_required=ISA.X86,
    coherence_protocol_required=CoherenceProtocol.MESI_TWO_LEVEL,
    kvm_required=True,
)

Here we state that we need gem5 compiled to run the X86 ISA and support the MESI Two Level protocol. We also require the host system to have KVM. NOTE: Please ensure your host system supports KVM. If your system does not please remove the kvm_required check here. KVM will only work if the host platform and the simulated ISA are the same (e.g., X86 host and X86 simulation).

This requires call is not required but provides a good safety net to those running the script. Errors that occur due to incompatible gem5 binaries may not make much sense otherwise.

Next we start specifying the components in our system. We start with the cache hierarchy:

cache_hierarchy = MESITwoLevelCacheHierarchy(
    l1d_size="32KiB",
    l1d_assoc=8,
    l1i_size="32KiB",
    l1i_assoc=8,
    l2_size="256KiB",
    l2_assoc=16,
    num_l2_banks=1,
)

Here we setup a MESI Two Level (ruby) cache hierarchy. Via the constructor we set the L1 data cache and L1 instruction cache to 32 KiB, and the L2 cache to 256 KiB.

Next we setup the memory system:

memory = SingleChannelDDR3_1600(size="2GiB")

This is quite simple and should be intuitive: A single channel DDR3 1600 setup of size 2GiB. Note: by default the SingleChannelDDR3_1600 component has a size of 8GiB. However, due to a known limitation with the X86Board, we cannot use a memory system greater than 3GiB. We therefore must set the size.

Next we setup the processor:

processor = SimpleSwitchableProcessor(
    starting_core_type=CPUTypes.KVM,
    switch_core_type=CPUTypes.TIMING,
    num_cores=2,
)

Here we are utilizing the gem5 standard library’s special SimpleSwitchableProcessor. This processor can be used for simulations in which a user wants to switch out one type of core for another during a simulation. The starting_core_type parameter specifies which CPU type to start a simulation with. In this case a KVM core. (Note: If your host system does not support KVM, this simulation will not run. You must change this to another CPU type, such as CPUTypes.ATOMIC) The switch_core_type parameter specifies which CPU type to switch to in a simulation. In this case we’ll be switching from KVM cores to TIMING cores. The final parameter, num_cores, specifies the number of cores within the processor.

With this processor a user can call processor.switch() to switch to and from the starting cores and the switch cores, which we will demonstrate later on in this tutorial.

Next we add these components to the board:

board = X86Board(
    clk_freq="3GHz",
    processor=processor,
    memory=memory,
    cache_hierarchy=cache_hierarchy,
)

Here we use the X86Board. This is a board used to simulate a typical X86 system in full-system mode. As a minimum, the board needs the clk_freq, processor, memory, and cache_hierarchy parameters specified. This finalizes our system design.

Now we set the workload to run on the system:

command = "m5 exit;" \
        + "echo 'This is running on Timing CPU cores.';" \
        + "sleep 1;" \
        + "m5 exit;"

board.set_kernel_disk_workload(
    kernel=Resource("x86-linux-kernel-5.4.49",),
    disk_image=Resource("x86-ubuntu-18.04-img"),
    readfile_contents=command,
)

The X86Board’s set_kernel_disk_workload function requires a kernel and disk_image to be set. Both these are obtainable from the gem5 resources repository. Therefore, via the Resource class, we specify x86-linux-kernel-5.4.49 for the Kernel (a Linux kernel, version 5.4.49, compiled to X86) and x86-ubuntu-18.04-img for the disk image (a disk image containing Ubuntu 18.04, for X86). The Resource class will automatically retrieve these resources if they are not already present on the host system. Note: If a user wishes to use their own resource (that is, a resource not prebuilt as part of gem5-resources), they may follow the tutorial here

The x86-ubuntu-18.04-img has been designed to boot the OS, automatically login, and run m5 readfile. The m5 readfile will read a file and execute it. The contents of this file are specified via the readfile_contents parameter. Therefore the value of readfile_contents will be executed on system startup. Note: readfile_contents is an optional argument. If it is not specified in set_kernel_disk_workload the simulation will exit after boot. This behavior is specific to the x86-ubuntu-18.04-img disk image and is not true for all disk images.

In this tutorial the script first runs m5 exit. This temporarily exits the simulation allowing us to switch the CPUs from KVM to TIMING. Then, when the simulation is resumed, the echo and sleep statements are executed (on the TIMING CPUs) and m5 exit is called again, thus exiting and completing the simulation. Users may inspect m5out/system.pc.com_1.device to see the echo output.

Finally, we specify how the simulation is to be run with the following:

simulator = Simulator(
    board=board,
    on_exit_event={
        ExitEvent.EXIT : (func() for func in [processor.switch]),
    },
)
simulator.run()

The important thing to note here is the on_exit_event argument. Here we can override default behavior. The m5 exit command triggers an EXIT exit event in the Simulator module. By default this exits the simulation run completely. In our case we want the first m5 exit call to switch processors from KVM to TIMING cores.

The on_exit_event parameter is a Python dictionary of exit events and Python generators. In this tutorial we are setting ExitEvent.Exit to the generator (func() for func in [processor.switch]). This means the processor.switch function is called on the first yield of the generator (that is, on the first instance of m5 exit). After this the generator is exhausted and the Simulator module will return to the default Exit exit event behavior.

This completes the setup of our script, to execute the script we run:

./build/X86/gem5.opt x86-ubuntu-run.py

You can see the output of the simulator in m5out/system.pc.com_1.device.

Below is the configuration script in full. It mirrors closely the example script at configs/example/gem5_library/x86-ubuntu-run.py in the gem5 repository.

from gem5.utils.requires import requires
from gem5.components.boards.x86_board import X86Board
from gem5.components.memory.single_channel import SingleChannelDDR3_1600
from gem5.components.cachehierarchies.ruby.mesi_two_level_cache_hierarchy import (MESITwoLevelCacheHierarchy,)
from gem5.components.processors.simple_switchable_processor import SimpleSwitchableProcessor
from gem5.coherence_protocol import CoherenceProtocol
from gem5.isas import ISA
from gem5.components.processors.cpu_types import CPUTypes
from gem5.resources.resource import Resource
from gem5.simulate.simulator import Simulator
from gem5.simulate.exit_event import ExitEvent

# This runs a check to ensure the gem5 binary is compiled to X86 and supports
# the MESI Two Level coherence protocol.
requires(
    isa_required=ISA.X86,
    coherence_protocol_required=CoherenceProtocol.MESI_TWO_LEVEL,
    kvm_required=True,
)

# Here we setup a MESI Two Level Cache Hierarchy.
cache_hierarchy = MESITwoLevelCacheHierarchy(
    l1d_size="32KiB",
    l1d_assoc=8,
    l1i_size="32KiB",
    l1i_assoc=8,
    l2_size="256kB",
    l2_assoc=16,
    num_l2_banks=1,
)

# Setup the system memory.
# Note, by default DDR3_1600 defaults to a size of 8GiB. However, a current
# limitation with the X86 board is it can only accept memory systems up to 3GB.
# As such, we must fix the size.
memory = SingleChannelDDR3_1600("2GiB")

# Here we setup the processor. This is a special switchable processor in which
# a starting core type and a switch core type must be specified. Once a
# configuration is instantiated a user may call `processor.switch()` to switch
# from the starting core types to the switch core types. In this simulation
# we start with KVM cores to simulate the OS boot, then switch to the Timing
# cores for the command we wish to run after boot.
processor = SimpleSwitchableProcessor(
    starting_core_type=CPUTypes.KVM,
    switch_core_type=CPUTypes.TIMING,
    num_cores=2,
)

# Here we setup the board. The X86Board allows for Full-System X86 simulations.
board = X86Board(
    clk_freq="3GHz",
    processor=processor,
    memory=memory,
    cache_hierarchy=cache_hierarchy,
)

# This is the command to run after the system has booted. The first `m5 exit`
# will stop the simulation so we can switch the CPU cores from KVM to timing
# and continue the simulation to run the echo command, sleep for a second,
# then, again, call `m5 exit` to terminate the simulation. After simulation
# has ended you may inspect `m5out/system.pc.com_1.device` to see the echo
# output.
command = "m5 exit;" \
        + "echo 'This is running on Timing CPU cores.';" \
        + "sleep 1;" \
        + "m5 exit;"

# Here we set the Full System workload.
# The `set_workload` function for the X86Board takes a kernel, a disk image,
# and, optionally, a the contents of the "readfile". In the case of the
# "x86-ubuntu-18.04-img", a file to be executed as a script after booting the
# system.
board.set_kernel_disk_workload(
    kernel=Resource("x86-linux-kernel-5.4.49",),
    disk_image=Resource("x86-ubuntu-18.04-img"),
    readfile_contents=command,
)

simulator = Simulator(
    board=board,
    on_exit_event={
        # Here we want override the default behavior for the first m5 exit
        # exit event. Instead of exiting the simulator, we just want to
        # switch the processor. The 2nd 'm5 exit' after will revert to using
        # default behavior where the simulator run will exit.
        ExitEvent.EXIT : (func() for func in [processor.switch]),
    },
)
simulator.run()

To recap what we learned in this tutorial:

• The requires function can be used to specify the gem5 and host requirements for a script.
• The SimpleSwitchableProcessor can be used to create a setup in which cores can be switched out for others.
• The X86Board can be used to setup full-system simulations. Its set_kernel_disk_workload is used specify the kernel and disk image to use.
• The set_kernel_disk_work accepts a readfile_contents argument. This is used to set the contents of the file to be read via gem5’s m5 readfile function. With the x86-ubuntu-18.04-img this is processed as a script to be executed after the system boot is complete.
• The Simulator module allows for the overriding of exit events using Python generators.