# HOWTO Systems Research

## Systems Research

• Systems research: solving problems in computer systems
• Performance, energy consumption, cost, security, etc.
• Worst research: try to solve non-existing problems
• Make the performance of a keyboard device driver 10x faster
• Does the performance of the keyboard device driver really matter?
• Finding real, important problems are critical

## How to find real, important problem

• What problem area is important
• What problems are already solved and what are not solved yet
• What are commonly used techniques in this field
• What are commonly agreed ways to evaluate a proposed methodology
• However, a paper is the result of at least one-year effort
• You are already two years behind than the authors

## How to find real, important problem as early as possible

• Attend top-tier conference and chat with researchers
• What are you working on these days?
• Good to get a sense of research direction
• If you are lucky, you can find a new, interesting problem
• Build a prototype of an existing system
• Best way to have deepest understanding of an existing system
• It takes time so it is risky if there is no new, interesting problem
• Benchmarking
• Economic way to find new, important problems

## How to use benchmarking to discover new problems

• Run existing benchmark in new setting
• E.g., run mosbench on 200 core machine -> Can we find any new bottleneck?
• E.g., run mosbench on docker, virtual machine, or unikernel -> Does new isolation mechanism introduce new overhead?
• E.g., run FxMark on an extremely-fast NVM or a slow SMR drive -> How does performance characteristics of storage device affect performance and scalability of storage stack?

## How to use benchmarking to discover new problems

• Compare existing approaches with the same benchmark
• Can have deeper understanding of existing approaches beyond what described in the paper
• Can draw design principles of a new approach

## What are important in benchmarking

• This is not an one time job
• Automation, automation, automation!
• Without automation, this is not a reproducible science.
• We human do make mistakes.
• Automation is one of way to avoid mistakes (or make the same mistakes consistently)
• If we are copying log files here and there, this is the sign that we should automate

## What are important in benchmarking

• Automate as much as possible
• Run all necessary combinations of experiments
• Generate graphs
• (Email generated graphs automatically)
• It does affect our productivity a lot!
• Run a benchmark script before having meeting (or going to bed, leaving office, etc)
• After meeting, check graph and devise what to run next or what to profile
• bash, awk, and python are our friends

## Examples of good benchmarks

• Mosbench (or our forked vbench)
• https://pdos.csail.mit.edu/archive/mosbench/
• https://github.com/sslab-gatech/vbench
• FxMark
• https://github.com/sslab-gatech/fxmark

## Tips for more meaningful benchmarking

• Make your results easy to compare
• ## transactions/msec instead of ## transactions
• Abort ratio instead of ## aborts
• Double check benchmark parameters
• number of initial elements in a hash table
• update ratio
• random distribution: uniform vs. zipf

## Tips for more meaningful benchmarking

• Use scalable memory allocator, such as jemalloc or tcmalloc
• Use git, tmux, and gnuplot, seriously