This notebook gives detailed information to help better understand our paper. In this notebook, we will demonstrate how to construct the subreddits' social networks created by more than 2.7 billion comments. Additionally, we will demonstrate how to calculate various statistics related to the subreddits. This code is licensed under a BSD license. See license file.
Before we begin, make sure you have installed all the required Python packages. (The instructions below use pip. You can use easy_install, too.) Also, consider using virtualenv for a cleaner installation experience instead of sudo. We also recommend running the code via IPython Notebook.
First, we need to download the compressed Reddit dataset files from pushshift.io website. This dataset was created by Jason Michael Baumgartner. Additional details about this dataset can be found at this Link. Downloading this hundreds-of-GB dataset can take a considerable amount of time. To save time, you can download only one month’s, or several months’, worth of data. After we download the dataset, we notice that the dataset is organized in directories, where each directory contains the posts of a specific year. These directories contain posts that were published from December 2005 to the most recent month. For this tutorial, we utilized over 2.71 billion comments that were posted from December 2005 through October 2016. Let's create a single SFrame that contains all these posts. To achieve this, we first will convert each monthly zipped file into an SFrame object using the following code:
import os
import logging
import bz2
from datetime import datetime
import graphlab as gl
import graphlab.aggregate as agg
import fnmatch
gl.canvas.set_target('ipynb')
gl.set_runtime_config('GRAPHLAB_CACHE_FILE_LOCATIONS', '/data/tmp')
gl.set_runtime_config('GRAPHLAB_DEFAULT_NUM_GRAPH_LAMBDA_WORKERS', 128)
gl.set_runtime_config('GRAPHLAB_DEFAULT_NUM_PYLAMBDA_WORKERS', 128)
basedir = "/data/reddit/raw" # Replace this with the directory which you downloaded the file into
sframes_dir = "/data/reddit/sframes/" # Replace this with the directory you want to save the SFrame to
tmp_dir = "/data/tmp" # Replace this with the directory you want to save the SFrame to
def get_month_from_path(path):
m = os.path.basename(path)
m = m.split(".")[0]
return int(m.split("-")[-1])
def get_year_from_path(path):
y = os.path.basename(path)
y = y.split(".")[0]
return int(y.lower().replace("rc_","").split("-")[0])
def json2sframe(path):
"""
Creates an SFrame object from the file in the input path
:param path: path to a file that contains a list of JSON objects, each JSON is saved in a separate line.
The file can also be compressed in bz2 format.
:return: SFrame object created from the file in the input path. The SFrame also contains information regarding
each post date & time
:rtype: gl.SFrame
"""
if not path.endswith(".bz2"):
sf = gl.SFrame.read_json(path, orient="lines")
else:
dpath = decompress_bz2(path)
sf = gl.SFrame.read_json(dpath, orient="lines")
#remove the decompressed file
os.remove(dpath)
#add datetime information
sf['month'] = get_month_from_path(path)
sf['year'] = get_year_from_path(path)
sf['datetime']= sf["created_utc"].apply(lambda utc: datetime.fromtimestamp(float(utc)))
return sf
def decompress_bz2(inpath, outpath=None):
"""
Decompress bz2 to the outpath, if the outpath is not provided then decompress the file to the inpath directory
:param inpath: decompress bz2 file to the outpath
:param outpath: output path for the decompress file
:return: the output file path
"""
if outpath is None:
outpath = tmp_dir + os.path.sep + os.path.basename(inpath) + ".decompressed"
out_file = file(outpath, 'wb')
logging.info("Decompressing file %s to %s" % (inpath,outpath))
in_file = bz2.BZ2File(inpath, 'rb')
for data in iter(lambda : in_file.read(100 * 1024), b''):
out_file.write(data)
out_file.close()
in_file.close()
return outpath
def match_files_in_dir(basedir, ext):
"""
Find all files in the basedir with 'ext' as filename extension
:param basedir: input basedir
:param ext: filename extension
:return: list of file paths with the input extension
"""
matches = []
for root, dirnames, filenames in os.walk(basedir):
for filename in fnmatch.filter(filenames, ext):
matches.append(os.path.join(root, filename))
return matches
#Creating all SFrames
for p in match_files_in_dir(basedir, "*.bz2"):
logging.info("Analyzing of %s " % p)
outp = sframes_dir + os.path.sep + os.path.basename(p).replace(".bz2", ".sframe")
if os.path.isdir(outp): #if file already exists skip it
logging.info("Skipping the analysis of %s file" % p)
continue
sf = json2sframe(p)
sf.save(outp)
Now let’s join all the SFrame objects into a single object. Please notice that different posts contain different metadata information about each post. Therefore, we will create a single SFrame which contains all the various metadata information.
join_sframe_path = sframes_dir + os.path.sep + "join_all.sframe" # Where to save the join large SFrame object
def get_all_cols_names(sframes_dir):
"""
Return the column names of all SFrames in the input path
:param sframes_dir: directory path which contains SFrames
:return: list of all the column names in all the sframes in the input directory
:rtype: set()
"""
sframes_paths = [sframes_dir + os.path.sep + s for s in os.listdir(sframes_dir)]
column_names = set()
for p in sframes_paths:
if not p.endswith(".sframe"):
continue
print p
sf = gl.load_sframe(p)
column_names |= set(sf.column_names())
return column_names
def get_sframe_columns_type_dict(sf):
"""
Returns a dict with the sframe column names as keys and column types as values
:param sf: input SFrame
:return: dict with the sframe column names as keys and column types as values
:rtype dict[str,str]
"""
n = sf.column_names()
t = sf.column_types()
return {n[i]: t[i]for i in range(len(n))}
def update_sframe_columns_types(sf, col_types):
"""
Updates the input sframe column types according to the input types dict.
:param sf: input SFrame object
:param col_types: dict in which the keys are the column names and the values are the columns types
:return: SFrame object with column types update to the col_types dict. If a column doesn't exist in the SFrame object
then a new column is added with None values
:rtype: gl.SFrame
"""
sf_cols_dict = get_sframe_columns_type_dict(sf)
for k,v in col_types.iteritems():
if k not in sf_cols_dict:
sf[k] = None
sf[k] = sf[k].astype(v)
elif v != sf_cols_dict[k]:
sf[k] = sf[k].astype(v)
return sf
def join_all_sframes(sframes_dir, col_types):
"""
Joins all SFrames in the input directory where the column types are according to col_types dict
:param sframes_dir:
:param col_types: dict with column names and their corresponding types
:return: merged SFrame of all the SFrames in the input directory
"rtype: gl.SFrame
"""
sframes_paths = [sframes_dir + os.path.sep + s for s in os.listdir(sframes_dir) if s.endswith(".sframe")]
sframes_paths.sort()
sf = gl.load_sframe(sframes_paths[0])
sf = update_sframe_columns_types(sf, col_types)
for p in sframes_paths[1:]:
if not p.endswith(".sframe"):
continue
print "Joining %s" % p
sf2 = update_sframe_columns_types(gl.load_sframe(p), col_types)
sf2.__materialize__()
sf = sf.append(sf2)
sf.__materialize__()
return sf
# use the inferred column type according to last month posts' SFrame. Set all other columns to be
# as type str
#col_names = get_all_cols_names(sframes_dir)
sf = gl.load_sframe(sframes_dir + '/RC_2015-05.sframe')
d = get_sframe_columns_type_dict(sf)
for c in col_names:
if c not in d:
print "Found new column %s" %c
d[c] = str
#Create Single SFrame
sf = join_all_sframes(sframes_dir, d)
sf.save(join_sframe_path)
At the end of this process, we obtained an SFrame with 2,718,784,464 rows, which is about 444 GB in size. Let's use the show function to get a better understanding of the data.
gl.canvas.set_target('ipynb')
sf = gl.load_sframe(join_sframe_path)
#sf.show() # running this can take considerable amount of time
Let's clean it by removing columns that aren't useful for creating the subreddit's social network. Namely, we remove the following columns: "archived," "downs," "retrieved_on," "banned_by," "likes," "user_reports," "saved," "report_reasons," "approved_by," "body_html," "created," "mod_reports," and "num_reports.”
sf = sf.remove_columns(["archived", "downs", "retrieved_on", "banned_by", "likes","user_reports", "saved",
"report_reasons", "approved_by", "body_html", "created", "mod_reports", "num_reports"])
sf.__materialize__()
Let's also delete users' posts that are from users that are probably bots and from those who have posted too many messages.
#First let's find how many posts the most active users posted
posts_count_sf = sf.groupby('author', gl.aggregate.COUNT())
posts_count_sf.sort('Count',ascending=False).print_rows(50)
To clean the dataset, we removed redditors who posted over 100,000 times and seemed to post automatic content. Additionally, we used the praw Python package to parse the comments posted in the BotWatchman subreddit in order to identify bots and remove them. We use the following code to assemble a bots list:
import praw
def get_bots_set():
#Please insert your Reddit application's authentication information.
#See more details at: https://github.com/reddit/reddit/wiki/OAuth2-Quick-Start-Example#first-steps
secret ='<insert your secert string>'
client_id = '<insert your client-id string>'
user_agent = '<insert your user-agent string>'
reddit = praw.Reddit(client_id=client_id,client_secret=secret, user_agent=user_agent)
sr = reddit.subreddit('BotWatchman')
bots_set = set()
for p in sr.search("overview for", limit=2000):
bots_set.add(p.title.split(" ")[2])
return bots_set
def get_remove_profiles_list(sf):
# We get the bots set twice to reduce the variance of the results
bots_set = get_bots_set()
bots_set |= get_bots_set()
posts_count_sf = sf.groupby('author', gl.aggregate.COUNT())
delete_users = set(posts_count_sf[posts_count_sf['Count'] > 100000]['author'])
delete_users |= bots_set
return delete_users
delete_users = get_remove_profiles_list(sf)
print "Delete Users List (%s users):\n %s" % (len(delete_users), delete_users)
Next, we used the following code to filter out comments posted by bot redditors that appeared in the bots list.
sf = sf[sf['author'].apply(lambda a: a not in delete_authors)]
len(sf)
sf.save("/data/reddit_data_no_txt_without_bots_and_deleted_authors.sframe")
We are left with about 2.39 billion comments.
We want to better understand the structure and evolution of subreddits. Let's calculate some interesting statistics on these subreddit communities. We will start by calculating the number of unique subreddits, and then we’ll create histograms of the number of posts on each subreddit.
#For running this section please make sure you created 'reddit_data_no_txt_without_bots_and_deleted_authors.sframe' as explained
# in the previous section
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
import graphlab as gl
import graphlab.aggregate as agg
sns.set(color_codes=True)
sns.set_style("darkgrid")
print "The dataset contains %s unique subreddits." % len(sf['subreddit'].unique())
g = sf.groupby('subreddit', {'posts_num': agg.COUNT()})
sns.distplot(g['posts_num'],bins=20, axlabel="posts")
We have 371,833 subreddits in the dataset. From the above histogram, we can see that the overwhelming majority of subreddits have very few posts. Let's look at the histogram of subreddits with at least a million posts.
g_mil = g[g['posts_num'] >= 1000000]
print "%s subreddits with at least a million posts" % len(g_mil)
sns.distplot(g_mil['posts_num'], axlabel="posts")
We discover that only 357 subreddits, less 0.1% of all the subreddits, contain more than a million posts. Let's calculate how many posts these subreddits contain in total.
print "The most popular subreddits contain %s posts" % g_mil['posts_num'].sum()
The most popular subreddits contain over 1.62 billion posts. In other words, less than 0.1% of the subreddits contain 68.2% of the posts. This result reminds me of the fact that over 57% of the world's population lives in the ten most populous countries. Let's map the users' activity in each subbreddit. Namely, we will find how many distinct user names there are in each subreddit, and what subreddits have the most unique users.
g = sf.groupby('subreddit', {'distinct_authors_number':agg.COUNT_DISTINCT('author')})
g = g.sort('distinct_authors_number', ascending=False)
g.print_rows(100)
By calculating the elapsed time between users' first post and last post, we can also estimate how much time users have been active in each subreddit.
Important: running the following code block may take considerable time
sf['created_utc'] = sf['created_utc'].astype(int)
subreddit_users = sf.groupby(['subreddit', 'author'], {'start_date':agg.MIN('created_utc'), 'end_date': agg.MAX('created_utc'), 'posts_num':agg.COUNT()} )
subreddit_users['activity_time'] = subreddit_users.apply(lambda d: d['end_date'] - d['start_date'])
subreddit_users
Let's calculate the average time users have been active in each subreddit. To understand the activity time distribution across the subreddits, let's plot a histogram of average activity time.
g = subreddit_users.groupby('subreddit', {'avg_active_time_in_seconds': agg.AVG('activity_time')})
g['avg_active_time_in_days'] = g['avg_active_time_in_seconds'].apply(lambda sec: sec/(60.0*60.0*24))
sns.distplot(g['avg_active_time_in_days'], axlabel="days", bins=20 )
g['avg_active_time_in_days'].sketch_summary()
We can see from the above results that most of the subreddits' users are active for a very limited time, with an average of less than a month and a median of less than a day.
In our study, we focused on analyzing the various patterns in which users join each subreddit (also referred to as the Join-Rate-Curve). In this section, we present the code that was used to create these curves. To create the join-rate-curve of each subreddit, we first created a TimeSeries object with the subreddit information. Throughout this section, we will use the Science subreddit as an example.
from datetime import datetime, timedelta
science_sf = sf[sf['subreddit'] =='science']
science_sf['datetime'] = science_sf['created_utc'].apply(lambda utc:datetime.utcfromtimestamp(int(utc)))
subreddit_ts = gl.TimeSeries(science_sf, index='datetime')
subreddit_ts
We will use the following function to create the subreddit user arrival curve from the TimeSeries object.
import math
def get_subreddit_users_arrival_curve(subreddit_ts, weeks_number=4):
"""
Calculates the percent of authors that joined after X weeks from all authors that joined the subreddit
between the date of the first comment and the date of the last comment
:param subreddit_ts: TimeSeries with the subreddit posts information
:param weeks_number: the number of weeks to set the time-interval between each two calculations
:return: dict in which all the keys are the number of weeks since the first comment was posted and the
corresponding percentage of authors that joined the subreddit up until this week
:rtype: dict
"""
dt = subreddit_ts.min_time
end_dt = subreddit_ts.max_time
authors = set()
d = {0: 0}
td = timedelta(days=7 * weeks_number)
count = 1
total_authors_num = float(len(subreddit_ts['author'].unique()))
while dt + td <= end_dt:
ts = subreddit_ts.slice(dt, dt + td)
authors |= set(ts['author'])
print "Calculating the user arrival curve between %s and %s" % (dt, dt + td)
dt += td
d[count * weeks_number] = len(authors) / total_authors_num
count += 1
ts = subreddit_ts.slice(dt, subreddit_ts.max_time)
authors |= set(ts['author'])
subreddit_age = subreddit_ts.max_time - subreddit_ts.min_time
d[math.ceil( subreddit_age.days/ 7.0)] = len(
authors) / total_authors_num # round the number of weeks up mainly for drawing the graph
return d
d = get_subreddit_users_arrival_curve(subreddit_ts)
keys = d.keys()
keys.sort()
values = [d[k] for k in keys]
plt.plot(keys, values)
plt.xlabel('Weeks')
plt.ylabel('Percentage')
In our study, we constructed the subreddit social networks by creating links between users that replied to other users’ posts. In this section, we will present the code which was used to create the subreddits' underlying social networks. As an example, we will use the Datasets subreddit's social network.
def get_subreddit_vertices_timeseries(subreddit_sf):
"""
Creates a vertices Timeseries object
:return: TimeSeries with the join time of each user to each subreddit
:rtype: gl.TimeSeries
"""
sf = subreddit_sf.groupby("author", {'mindate': agg.MIN("created_utc"),
'maxdate': agg.MAX("created_utc")})
sf['mindate'] = sf['mindate'].apply(lambda timestamp: datetime.fromtimestamp(timestamp))
sf['maxdate'] = sf['maxdate'].apply(lambda timestamp: datetime.fromtimestamp(timestamp))
sf.rename({"author": "v_id"})
return gl.TimeSeries(sf, index='mindate')
def get_subreddit_interactions_timeseries(subreddit_sf):
"""
Creates subreddits interactions TimeSeries. Interaction exists between two subreddit users if user A posted a comment
and user B replied to A's comment
:return: TimeSeries with the subreddit interactions
:rtype: gl.TimeSeries
"""
subreddit_sf['parent_name'] = subreddit_sf["parent_id"]
subreddit_sf['parent_kind'] = subreddit_sf['parent_id'].apply(lambda i: i.split("_")[0] if "_" in i and i.startswith("t") else None)
subreddit_sf['parent_id'] = subreddit_sf['parent_id'].apply(lambda i: i.split("_")[1] if "_" in i and i.startswith("t") else None)
sf = subreddit_sf[subreddit_sf['parent_kind'] == 't1'] # only reply to comments counts
sf = sf['author', "created_utc", "id", "parent_id", "link_id"]
sf = sf.join(subreddit_sf, on={"parent_id": "id"})
sf['datetime'] = sf['created_utc'].apply(lambda timestamp: datetime.fromtimestamp(timestamp))
sf.rename({'author': 'src_id', 'author.1': 'dst_id'})
sf = sf['src_id', 'dst_id', 'datetime']
return gl.TimeSeries(sf, index='datetime')
def create_sgraph(v_ts, i_ts):
"""
Creates an SGraph object from the vertices and interaction TimeSeries objects
:param v_ts: vertices TimeSeries
:param i_ts: interactions TimeSeries
:return: SGraph with the input data
"""
edges = i_ts.to_sframe().groupby(['src_id', "dst_id"], operations={'weight': agg.COUNT(),
'mindate': agg.MIN('datetime'),
'maxdate': agg.MAX('datetime')})
g = gl.SGraph(vertices=v_ts.to_sframe(), edges=edges, vid_field="v_id", src_field="src_id", dst_field="dst_id")
return g
datasets_sf = sf[sf['subreddit'] =='datasets']
datasets_sf.__materialize__()
datasets_sf["created_utc"] = datasets_sf["created_utc"].astype(int)
vertices = get_subreddit_vertices_timeseries(datasets_sf)
interactions = get_subreddit_interactions_timeseries(datasets_sf)
vertices.print_rows(10)
interactions.print_rows(10)
g = create_sgraph(vertices,interactions)
We created an SGraph object from the SFrame. Let's visualize the constructed social network.
g.summary()
import os
import logging
import bz2
from datetime import datetime
import graphlab as gl
import graphlab.aggregate as agg
import fnmatch
#gl.canvas.set_target('ipynb')
gl.set_runtime_config('GRAPHLAB_CACHE_FILE_LOCATIONS', '/data/tmp')
gl.set_runtime_config('GRAPHLAB_DEFAULT_NUM_GRAPH_LAMBDA_WORKERS', 128)
gl.set_runtime_config('GRAPHLAB_DEFAULT_NUM_PYLAMBDA_WORKERS', 128)
basedir = "/data/reddit/raw" # Replace this with the directory which you downloaded the file into
sframes_dir = "/data/reddit/sframes/" # Replace this with the directory you want to save the SFrame to
tmp_dir = "/data/tmp" # Replace this with the directory you want to save the SFrame to
join_sframe_path = sframes_dir + os.path.sep + "join_all.sframe" # Where to save the join large SFrame object
sf = gl.load_sframe("/data/reddit_data_no_txt_without_bots_and_deleted_authors.sframe")
subreddit_users = gl.load_sframe("/data/subreddits_users.sframe")
We can use GraphLab's analytics toolkit to calculate various topological properties, such as the degree distribution and the graph's number of triangles. We hope to elaborate on this in a future tutorial. For now, let's use Networkx package to draw this subreddit's social network.
import networkx as nx
def sgraph2nxgraph(sgraph):
""" Converts a directed sgraph object into networkx object
:param sgraph: GraphLab SGraph object
:return: Networkx Directed Graph object
"""
nx_g = nx.DiGraph()
vertices = list(sgraph.get_vertices()['__id'])
edges = [(e['__src_id'], e['__dst_id']) for e in sgraph.get_edges()]
nx_g.add_nodes_from(vertices)
nx_g.add_edges_from(edges)
return nx_g
nx_g = sgraph2nxgraph(g)
Let's draw a subgraph of the subreddit
import networkx as nx
import random
def draw_graph(g,layout_func=nx.spring_layout):
pos = layout_func(g)
d = nx.degree(g)
n_sizes = [v * 25 +5 for v in d.values()]
nx.draw(g, nodelist=d.keys(), node_size=n_sizes, node_color='blue')
# Selecting only sample of vertices with degree greater than 0
d = nx.degree(nx_g)
v_list = [v for v in d.keys() if d[v] >0]
h = nx_g.subgraph(v_list[:500])
draw_graph(h)
The social network dataset created as a result of our study opens the door for new and exciting research opportunities. This dataset can help not only to better understand the social structure of the Reddit community in particular, but also to understand how social networks and online communities evolve over time. Moreover, this corpus can be used as a ground-truth dataset for many studies in the field of social networks. Some examples of what can be done with this corpus are:
We would love to hear other ideas on what possible work can be done with our provided datasets.
Further reading material: