[an error occurred while processing this directive]
<p>
Large Scale Data Engineering is a MSc course by <a href="http://www.cs.vu.nl/en/research/artificial-intelligence/kr/People">VU University professor</a> in the <a href="http://homepages.cwi.nl/~boncz/InauguralLecturePeterBoncz.pdf">Large-Scale Analytical Data Management</a> special chair <a href="http://homepages.cwi.nl/~boncz">Peter Boncz</a>, and <a href="http://hannes.muehleisen.org">Hannes Mühleisen</a> from the <a hre="https://www.cwi.nl/research-groups/database-architectures">Database Architectures</a> research group of CWI, developed specifically for the <a href="http://amsterdamdatascience.nl">Amsterdam Data Science</a> initiative.
</p>
<div class="post_title">Goals &amp; Scope</div>
<p>
The goal of the course is to gain insight into and experience with algorithms and infrastructures for managing big data.
</p><p>
This course confronts the students with some data management tasks, where the challenge is that the mere size 
of this data causes naive solutions, and/or solutions that work only on a single machine, to stop being practical.
</p><p>
Solving such tasks requires the computer scientist to have insight in the main factors that underlie algorithm 
performance (access pattern, hardware latency/bandwidth), as well as possess certain skills and experience in 
managing large-scale computing infrastructure. 
</p>
<div class="post_title">Results</div>
<p>
The course has now completed succesfully. Here is a small (biased ;-) sample of the <b>results</b> of the second assignment, where students had to analyze datasets using cluster technology:
<ul>
<li><a href="http://wikistats.ins.cwi.nl/lsde-data/group1/">World Map of Faces</a>, which used MapReduce to crawl a 100 million picture archive from Flickr, run face-detection on the images, and aggregate these into map overlays at various zoom levels. <i>Created by <b>Till Dohmen</b> and <b>Florian Gomelo</b></i>.</li>
<li><a href="results/group2/nightclubs_visualize_html/">Secret New York City Nightlife</a>, which analyzed a 27GB New York taxitrips dataset for nightly patterns. The result is a New York map showing those locations which are much more often taxi drive destinations at night than during the day, and which do not map to known nightclubs. <i>Created by <b>David Mueller</b> and <b>Alessandro Zonta</b></i>.</li>
</ul>
</p>
<div class="post_title">Course Structure</div>
<p>
There are two lectures per week, on <b>Monday 11:00-12:45</b> and <b>Thursday 09:15-10:45</b> in room <b>P647</b>.
More information about the lectures is in the <a href="schedule.shtml">schedule</a>.
This course may take a significant amount of time and effort, and requires significant practical work.
The practicals are done outside lecture hours, at the discretion of the students.
</p>
<p>
In the <a href="practical1.shtml">first assignment</a> the students can work either on
their own laptops via a prepared VM, or in the cloud using
an <a href="http://aws.amazon.com/ec2/instance-types/t2/">Amazon EC2 Micro Instance</a>.
</p>
<p>
The <a href="practical2.shtml">second assignment</a>, using a Hadoop Cluster, will be done on the 
<a href="https://surfsara.nl/project/hadoop">SurfSARA Hadoop cluster</a> (90 machines, 720 cores, 1.2PB storage). 
</p>

<div class="post_title">Tasks</div>
<p>
The students must work in groups consisting of two students; the <a href="https://bb.vu.nl/webapps/portal/frameset.jsp?tab_tab_group_id=_2_1&url=%2Fwebapps%2Fblackboard%2Fexecute%2Flauncher%3Ftype%3DCourse%26id%3D_102162_1%26url%3D" target="_parent">VU BlackBoard</a> is used for registering student groups and for reporting grades. 
<p>
</p>
For this course, each group must deliver the following:
<ul>
<li>a program which runs your assignment1 data loader (reorg) and the benchmark queries (with correct results) within the deadline on our test hardware (a VM running on a dedicated machine, single-core CPU, 1GB RAM). There is an online competition between the student groups for maximum speed. The evaluation set in this competition consists of 50 queries, and we take the median query time as the score. You are asked to use git as your code repository, since the competition software automatically checks out <a href="http://gitref.org/basic/#commit">git commits</a>, benchmarks your software and adjusts the <a href="http://wikistats.ins.cwi.nl/leaderboard">leaderboard</a>.  </li>
<li>a presentation (of ~20 minutes) during one of the lectures summarizing two research papers on the topic of Assignment2. This topic must be chosen from the list of pre-defined topics, and no two groups can choose the same topics. The students can choose the topic first-come-first-served, based on the final ranking in the Assignment1 speed competition. A draft of the presentation should be emailed to <a href="mailto:lsde2015@outlook.com">lsde2015@outlook.com</a> 24 hours before presentation for early feedback and tips.</li>
<li>a written report (PDF) for Assignment2, which contains a summary of the presentation (as a "related work" section) and a description of the Hadoop project and its results. Plus all code created for that project in a tar file. This should be submitted via blackboard.</li>
</ul> 
<p>
The final grade (1-10) is weighted (30%,20%,40%) between these respective tasks with the final 10% based on your individual observed participation during the lectures.
</p>

<div class="post_title">Getting Help</div>
<p>
<img src="images/skype_logo.png" style="float:right;width:42px;height:42px"/>
You can email questions to <a href="mailto:lsde2015@outlook.com">lsde2015@outlook.com</a> or via chat or audio
or video connection to that account via <a href="http://www.skype.com">skype</a>.
</p>

<div class="post_title">Acknowledgements</div>
<p>
The lecture slides for LSDE2015 are based on those used in the 
<a href="http://www.inf.ed.ac.uk/teaching/courses/exc/">Extreme Computing</a> course, and were graciously 
provided by <a href="http://homepages.inf.ed.ac.uk/sviglas/">dr. Stratis Viglas</a>, of 
<a href="http://www.ed.ac.uk/schools-departments/informatics/">University of Edinburgh</a>.
</p>

[an error occurred while processing this directive]