Project 2

StackExchange is a multi-site platform for asking and answering questions. Users earn reputation when their posts (questions and answers) are upvoted (details here).


Create a single repository called cinf401-project-2, and indicate in a README the members of the group. Submit your Java source code, your R code, and your R Markdown. Do not include your data files.

All users, posts, etc. have been data acquired from this torrent and uploaded in HDFS under /data/stackexchange (web view). Have a look at /bigdata/data/stackexchange/readme.txt (on delenn, not HDFS) to understand the fields in the various XML files.

Using Hadoop and MapReduce, plus minimal R processing, complete the three tasks below.

Task 1

What is the age distribution of users across all sites as a whole? Be sure only unique users are counted (identified by AccountId field) and nobody is double-counted. Show your findings with an appropriate plot. Ensure your plot has easy-to-understand axis labels and all labels are readable.

Task 2

What are the top 10 tags (by frequency of posts) for each stackexchange subsite? Show as a list or table for each subsite, in alphabetical order by subsite name.

Task 3

How strong is the correlation between user reputation and number of posts (questions or answers) by that user? Look at all stackexchange sites as a whole. Sum the different reputation values and post counts for the same user (identified by AccountId) across all the sites in which they are a member. (Be aware that the posts files do not include the user’s AccountId, only the site-specific user id.) Report your findings with a scatterplot and correlation value.


Considering the dataset lives on delenn in regular files, it may be faster to process the files with traditional unix tools or scripting languages (assuming you arranged a parallel processing strategy). The purpose of the project is to practice with MapReduce, but IRL we should always choose “the best tool for the job.”

Refer to the MapReduce cookbook for guidance on how to run an MR job over multiple files.

Store your MR output in HDFS under a file path like /users/jeckroth/gp2/... (but obviously not jeckroth).

CINF 401 material by Joshua Eckroth is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Source code for this website available at GitHub.