SIGMOD 2010 Repeatability & Workability Evaluation
                               for paper #396
                 Positional Update Handling in Column Stores
                                     by
Sándor Héman, Marcin Zukowski, Niels Nes, Lefteris Sidirourgos, Peter Boncz


Hardware & Software environment
===============================

       | Paper                   | Review                          
-------+-------------------------+---------------------------------
 class | desktop                 | desktop                         
 CPU   | Intel Core2Quad Q6600   | Intel Core2Quad Q9550           
 cores | 4                       | 4                               
 GHz   | 2.4                     | 2.83                            
 RAM   | 8 GB                    | 8 GB                            
 OS    | Fedora 12               | Fedora 12                       
       | (Linux 2.6.31)          | (Linux 2.6.31)                  


Submission
==========

The authors provided 
 - pre-compiled binaries for 64-bit (x86_64) Fedora 12 Linux
   (source code could not be provided due to IP);
 - data generators for the micro benchmark datasets;
 - TPC data generators for the TPC-H datasets;
 - scripts to re-run all experiments, generate the graphs and re-build the
 - paper (.pdf) using the generated graphs --- the latter only for the micro
   benchmark experiments; the scripts to re-run the TPC-H experiments do
   only produce the result graphs, but not include them into the paper;


Repeatability Evaluation
========================

Process
-------

The given instructions are clear and simple to follow, and the scripts work
flawlessly. In the original submission, there was a small bug in on of the
scripts such that one graph (Figure 16) was not properly re-created and
included in the paper (although the respective experiments did run without
errors). On request, the authors fixed this bug instantly.

Detailed Results
----------------

* Micro benchmarks (Figures 16, 17, 18)

  For all experiments, the repeated results confirm the original ones with
  differences not exceeding what is to be expected due to (minor) hardware
  differences.

  Only deletes in Figure 16 appear to be fast that reported in the paper. In
  a discussion, the authors explained that this is due to a slight
  optimization in their code base after the experiments for their paper had
  been run.

* TCP-H experiments (Figure 19)

  Also for the TCP-H experiments the normalized performance results show the
  same behaviour as in the original paper.

Summary
-------

All experiments could be repeated and the repeatability results confirm the
original results.


Workability Evaluation
======================

Workability was only evaluated to the extend that the experiments were
successfully repeated on different hardware platforms (laptop with Core2Duo,
4 GB RAM & single disk; desktop with Core2Quad, 8 GB RAM & 2-disk RAID-0,
server with dual Core2Quad, 64 GB RAM & multi-disk RAID-5) and the the TPC-H
experiments were successfully run with scale factor 1 & 10.