A New Tool to Accurately Simulate Complex Systems

Researchers at MIT developed a tool that eliminates a source of bias in simulations. Researchers can easily and unknowingly select an algorithm that performs worse on actual data than the simulation predicted that it should.

The new method eliminates this source of bias in trace-driven simulation. It can help researchers design better algorithms for such applications as improving video quality on the internet and increasing the performance of data processing systems.

The machine-learning algorithm draws on the principles of causality to learn how the system’s behavior affected the data traces and replayed a correct, unbiased version of the trace during the simulation. The simulation method correctly predicted which newly designed algorithm would be best for video streaming, which led to less rebuffering and higher visual quality. Existing simulators that do not account for bias would have pointed researchers to a worse-performing algorithm.

In video streaming, an adaptive bitrate algorithm decides the video quality, or bitrate, to transfer to a device based on real-time data on the user’s bandwidth. Researchers can collect real data from users during a video stream for a trace-driven simulation. They used to simulate what would have happened to network performance had the platform used a different adaptive bitrate algorithm in the same conditions.

In video streaming, network performance is affected by the bitrate adaptation algorithm’s choices and intrinsic elements, like network capacity. However, researchers often cannot directly observe intrinsic properties.

The new tool, called CausalSim, is used so that the algorithm can learn the underlying characteristics of a system using only the trace data. It takes trace data collected through a randomized control trial. It estimates the underlying functions that produced those data and tells the researchers how a new algorithm would change the outcome under the same underlying conditions that a user experienced.

Using a typical trace-driven simulator, bias might lead a researcher to select a worse-performing algorithm, even though the simulation indicates it should be better. CausalSim helps researchers determine the best algorithm that was tested.

During the experiment, CausalSim consistently improved simulation accuracy, resulting in algorithms that made half as many errors as those designed using baseline methods.