Running Bao Hypervisor on gem5

Nov 12, 2024 • Hidemasa Kawasaki¹, Soramichi Akiyama¹.

¹ Ritsumeikan University

This article presents a methodology for executing the Bao Hypervisor [1] on gem5 via the VExpress_GEM5_Foundation platform, enabling executions of two unmodified Linux virtual machines simultaneously.

It was tested on “x86_64 Ubuntu 22.04”.

Obtain and Build Software

Install the tools required to run Bao Hypervisor on gem5.

sudo apt update
sudo apt install git tree scons gcc g++ python3-dev \ 
make automake device-tree-compiler \
bison flex libssl-dev

Create a directory to store build outputs:

mkdir -p resources/binaries resources/semihosting

Download and setup the toolchain

Download the latest bare-metal cross-compile toolchain “aarch64-none-elf” from the Arm Developer’s website. Tested on “arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.

Once the cross compiler for aarch64 has been downloaded, the file should be extracted.
Then, define the cross compiler prefix to the CROSS_COMPILE environment variable.

The following example demonstrates the installation of “arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.tar.xz”.
When setting the environmental variable, be careful not to forget a “-” at the end.

# Downloaded "arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.tar.xz" 
# to current directory, then following command should be executed.
tar xvf arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf.tar.xz
export CROSS_COMPILE=$(pwd)/arm-gnu-toolchain-13.3.rel1-x86_64-aarch64-none-elf/bin/aarch64-none-elf-

Set up base environment and build automation

Obtain the git repository for building Bao Hypervisor and Linux. Then, build it using the make command.

git clone https://github.com/h1demasa/bao-demos.git
cd bao-demos
make -j $(nproc)

Once the build is completed, copy the generated binary to the directory that gem5 will use.

cp wrkdir/imgs/fvp-a/fip.bin ../resources/binaries/fip.bin
cp wrkdir/imgs/fvp-a/bl1.bin ../resources/binaries/bl1.bin
cp wrkdir/imgs/fvp-a/linux+linux/bao.bin ../resources/semihosting/bao.bin

Before proceeding, return to the parent directory and check that your resources directory has the following structure:

cd ../
tree resources/

resources
    binaries
        bl1.bin
        fip.bin
    semihosting
        bao.bin

gem5 (simulator)

To run Bao Hypervisor with gem5, build gem5.

As the Performance Monitors Control Register is not implemented in gem5, numerous warning logs are generated when Bao is running. These logs affect gem5’s execution performance. To prevent these warning logs, apply the following patch to gem5.

git clone https://github.com/gem5/gem5.git -b v24.0.0.1
cd gem5
git apply ../bao-demos/platforms/gem5/gem5.patch

Build gem5 with the above changes:

scons -j $(nproc) build/ARM/gem5.opt
cd -
make -C gem5/util/term

Run and Connect to the Simulation

Now you can run the simulation:

export M5_PATH=resources/ && \
	gem5/build/ARM/gem5.opt gem5/configs/example/arm/baremetal.py \
	   --workload ArmTrustedFirmware \
	   --num-cores 4 \
	   --mem-size 4GB \
	   --machine-type VExpress_GEM5_Foundation \
	   --semi-enable --semi-path resources/semihosting

And connect to it:

gem5/util/term/m5term 3456 # Trusted Firmware-A, U-boot and Bao Hypervisor
gem5/util/term/m5term 3457 # VM 1 (Linux)
gem5/util/term/m5term 3458 # VM 2 (Linux)

It is recommended that you observe the various stages of the bootloader and the text indicating the Bao Hypervisor boot process, which can be accessed via port 3456. Similarly, the Linux kernel boot sequence can be observed via port 3457 and port 3458. Once the boot process is completed, A login shell is started. You can access it via ports 3457 and 3458. The username and password for this login are “root”.

In my environment, it took an hour for the Linux login shell to start up.
Please wait while enjoying a cup of coffee.

Demo

This is a demo video of running Bao Hypervisor on gem5.

[1] José Martins, Adriano Tavares, Marco Solieri, Marko Bertogna, and Sandro Pinto. “Bao: A Lightweight Static Partitioning Hypervisor for Modern Multi-Core Embedded Systems”. In Workshop on Next Generation Real-Time Embedded Systems (NG-RES 2020). Schloss Dagstuhl-Leibniz-Zentrum für Informatik. 2020. https://drops.dagstuhl.de/opus/volltexte/2020/11779/