Preprocess text data for analysis — preprocess

Cleans text, removes artifacts, filters by length, and removes duplicates. Supports platform-specific cleaning rules and custom regex patterns.

Usage

preprocess_text(data, config = list())

Arguments

data: Standardized tibble (must have std_text column)
config: Preprocessing config section from YAML

Value

Filtered tibble with cleaned text

Details

Default transforms (with source_type = "reddit", the default when unset): URLs are removed; u/ and at-mentions are deleted; r/<name> references are replaced by the literal redaction token [subreddit] (a privacy marker; bracketed editorial insertions are standard qualitative-research convention); markdown bold / italic / strikethrough markup is stripped and quoted (>) lines are dropped; [removed] / [deleted] placeholders are removed; HTML entities and unicode escapes are decoded; text is NFC-normalized and whitespace is collapsed.

All downstream coding, T0.1 quote verification, report excerpts, and QDPX export operate on this cleaned std_text – so "verbatim" in the T0.1 guarantee means verbatim with respect to the cleaned analytic text. The raw platform text is preserved unchanged in original_text.