Skip to contents

Scrape Reddit subreddits into a SQLite database

Source: R/05_scraper.R
scrape_reddit.Rd

Authenticates with the Reddit API and collects posts and their full comment trees from specified subreddits. Unlike keyword-based scrapers, this collects ALL content and lets the progressive sequential coder downstream decide which entries match the research question (no keyword pre-filter).

Usage

scrape_reddit(
  config = NULL,
  db_path = NULL,
  subreddits = NULL,
  posts_per_subreddit = NULL,
  include_comments = NULL,
  sort_by = NULL,
  time_filter = NULL
)

Arguments

config

ThematicConfig object or list with a $scraping section (a bare list is also accepted and treated as the scraping section).

db_path

Path to SQLite database (created if missing)

subreddits

Character vector of subreddit names (without "r/")

posts_per_subreddit

Max NEW posts to add per subreddit (default 500). Already-stored posts do not count against this budget, so re-running reaches genuinely new content rather than re-counting duplicates.

include_comments

Logical; also scrape full comment trees (default TRUE)

sort_by

Sort method: "new", "hot", "top", "rising" (default "new")

time_filter

Time filter for "top" sort: "hour", "day", "week", "month", "year", "all" (default "all")

Value

List with counts: posts_added, comments_added, posts_skipped, comment_fetch_failures (posts whose comments could not be fetched this run; they are retried next run), and truncated_subreddits (a character vector of subreddits whose collection stopped early because of an API error, so the caller can tell a genuinely empty result apart from an incomplete one).

Details

Comments are collected deeply: nested reply threads are walked recursively, "load more comments" placeholders are expanded via the Reddit API, and "continue this thread" branches deeper than the API's per-response depth are followed by re-fetching the tree rooted at the parent comment (bounded by a generous recursion backstop that real threads do not reach).

The access token is refreshed automatically (proactively before expiry and on a 401), so long multi-subreddit runs do not truncate when the initial token ages out.

Comment fetching is recoverable: if a post's comments cannot be fetched on one run, the post is kept and its comments are backfilled on a later run (tracked per post), and the count of failed fetches is returned.