gem5art announcement
gem5art: a gem5 Experiment Framework
Hoa Nguyen, Ayaz Akram, Mahyar Samani, Krithiga Murugavel, Trivikram Reddy, Marjan Fariborz, Pouya Fotouhi, Jason Lowe-Power
We are excited to announce the gem5art project, a framework aiming to tackle the issue of reproducibility of gem5 experiments and improve the overall testing and experimental infrastructure of gem5.
The framework is written in Python3 and utilizes MongoDB for creating experiments database. gem5art is available for downloading through Python Package Index (PyPI, which is also known as pip).
To get started, you can download gem5art via pip with the following commnads,
pip install gem5art-artifact gem5art-run gem5art-tasks
The Need for a Framework for gem5 Experiments
The gem5 simulator provides rich support for a variety of hardware models and configurations with high complexity. However, this complexity comes with a compromise. Having such a tremendous amount of possibilities of using gem5 results in a lack of a standard approach for running gem5 experiments, which in turn results in a steep learning curve for using gem5 as well as makes reproducing a gem5 experiment a non-trivial process.
Setting up gem5 full system simulation is one example of a highly non-trivial processes. A typical setup for a full system experiment requires the following inputs: a compiled Linux kernel, a disk image containing a Linux distribution and the desired benchmark suite that is compiled, a compiled gem5 executable and a gem5 system configuration. Each of the components of the above experiment is sensitive to time and environmental changes, and some of them require a significant amount of time and attention to correctly produce the input.
For example, gem5 requires a very specific Linux kernel configuration. Another example is creating a working disk image with an installed benchmark. A frequent approach is to use qemu and manually typing commands to create such a disk image; this approach often requires several hours and lots of human attention and is error-prone. Another factor affecting the reproducibility of an experiment is the complier, which is frequently updated over time. Therefore, it is desirable to keep the artifacts that we used for experiments.
Unlike many other areas of science, we have the luxury of keeping carbon copies of the inputs to our experiments along with a certain degree of determinism in the experiment process. Thus, it is desirable to have a systematic approach for producing and documenting a gem5 experiment; more importantly, it is essential to have a standard approach for running gem5 experiments that allows throughout testing and reproducibility, both for ourselves and other researchers.
gem5art: Artifact, Reproducibility and Testing Framework for gem5
gem5art framework contains features simplifying the process of conducting structured gem5 experiments: allowing user to add important documentation (i.e. meta-data) to each component of a gem5 experiment, creating a database backing up the inputs and outputs of experiments, providing a way to detect duplicated experiments, providing sample Linux configurations, and providing a template for automating disk image creation.
Figure 1: An example gem5art work flow for setting up a gem5 full system experiment.
Each tile indicates a component of the experiment, where the components are represented as Artifact objects in a gem5art launch script.
The launch script corresponding to the setup can be found here.
The setup consists of the base repository, the gem5 repository and the Linux repository as the inputs; while the output component is results.
The base repository contains the scripts (e.g. the launch script, the scripts for installing benchmark while building disk images) and the configs (e.g. gem5 configs, Linux kernel configs, Packer configs for building disk images).
The gem5 repository and the Linux repository are the git repositories containing source code of the corresponding projects.
The results component indicates the zipped file containing all outputs of the gem5 run.
Other than the inputs and outputs, the intermediate artifacts are the gem5 binary, the m5 binary, the vmlinux binary and the disk images.
The steps to create the intermediate objects are expected to be documented in the gem5art launch script.
Each artifact is backed up to the database when it is registered: for git repositories, the git hashes are stored, while for other artifacts, the files are stored.
Both git hashes and files are stored in the database.
It is essential to document and backup every component of an experiment, and this improves reproducibility of an experiment. We define every input and output as a component (or an artifact as we call it in gem5art) of an experiment. By adding more meta-data (such as created/modified date/time, version of a binary/source code, commands/steps used to produce inputs/outputs) to every component, we subsequently have a system that automatically keeps track of the experiment instead of having to do it manually. As a result, we have well-defined inputs, and thus, a well-defined experiment. Adding meta-data also allows detecting duplicated experiments (which share the same input components), we avoid re-run experiments systematically instead of having to remember which experiments have been run, thus avoiding waste of time and hardware resource. gem5art also backs up the artifacts so that we can retrieve a binary or a git hash of a git repository used for an experiment.
gem5art also provides support for gem5 full system experiments setup. As mentioned, it requires a significant amount of modification to the default Linux configuration in order to have a Linux kernel that works with gem5. Our development repository provides known working configurations for a several version of Linux kernel. We also provide templates for creating disk images containing several commonly used benchmark suites, such as NAS Parallel Benchmarks, SPEC 2017, PARSEC. Please follow the tutorials for more information.
Preliminary Results
By thoroughly documenting the process of running gem5 full system experiments, we simplify the process and allow more users to contribute to gem5. Over the span of 6 months, starting from the gem5art template for the boot test experiments, we have developed and documented the use cases gem5art framework with many different benchmark suites including NAS Parallel Benchmarks, PARSEC, SPEC 2006, SPEC 2017, GAPBS and Microbenchmarks. We also conducted over 4000 experiments across all benchmark suites with a large configuration space (e.g. 195 different configurations for PARSEC).
Using a systematic approach allows us to conduct more well-defined experiments with a large parameter space. In one of our earlier blog posts, we are able to figure out what versions of Linux work with gem5-19 with a large configuration space (5 Linux kernel versions x 4 CPU models x 4 CPU counts x 2 Memory systems x 2 Boot types). Such an experiment is possible without gem5art. However, with a small overhead of writing a gem5art launch script for creating and documenting artifacts, we don’t have to worry about backing up the artifacts and remembering the location and the structure the result folders. Only a simple query is required to get the results from the database.
gem5art also allows us to conduct cross experimental studies. For example, we conduct the same experiment as above with gem5-20. Since we documented what we did for gem5-19 experiment, the gem5-20 is more of a systematic process such that we only follow the document and we don’t need to remember how the inputs of gem5-19 were created. Only minimal changes are required (changing gem5-19 artifact to gem5-20 artifact). Other than that, we don’t need to worry about using wrong inputs (e.g. using gem5-19 binary, checking out wrong branch on gem5.googlesource.com, using a different disk image), and we know for certain that, the only difference between the two experiments is the gem5 binary.
gem5 experiments, especially gem5 full system experiments, are challenging. The challenge does not only come from the steep learning curve of setting up the experiments, but also Linux kernel and gem5 are being actively developed that makes input parameter space even larger over time. Being able to conduct experiments in such a fast changing environment requires a huge amount of effort. We believe that gem5art provides a framework and resource for easing the challenge and letting more users to contribute to the gem5 project.