LibGuides: Finding Data & Statistics: Text Mining

A note about TCU-licensed databases

Most of our licensed databases usually

limit the number of citations or articles that can be downloaded at once
prohibit systematic downloading (downloading of substantial collections)
prohibit automated downloading (using of scripts)
prohibit datamining directly on the vendor's servers
prohibit the redistribution of content (including cleaned data)

Please contact your librarian with questions about any specific resource or database.

Freely available corpora

the @unitedstates project
@unitedstates is a shared commons of data and tools for the United States. Made by the public, used by the public.

Featuring work from people with the Sunlight Foundation, GovTrack.us, the New York Times, the Electronic Frontier Foundation, and the Internet
American Presidency Project
One of the most comprehensive collection of web resources on the American presidency, including documents, public papers, executive orders, addresses, press conferences, debates, election data, approval ratings, much more. Data topics include Relations with Congress / Popularity / Public Appearances / Growth of the Executive Branch / Presidential Selection / State of the Union and Inaugural Address Charts / Presidential Disability
arXiv Bulk Data Access
Awesome Public Datasets
An awesome list of high-quality open datasets (HQOD) in public domains (on-going).
Chronicling America
OCR bulk downloads from the Library of Congress of America's historic newspaper pages from 1836-1922
CMU Movie Summary Corpus
This page provides links to a dataset of movie plot summaries and associated metadata. This data was collected by David Bamman, Brendan O'Connor, and Noah Smith at the Language Technologies Institute and Machine Learning Department at Carnegie Mellon University.
Common Crawl
Open repository of web crawl data that can be accessed and analyzed by anyone.
Congressional and Federal Government Web Harvests
Since 2006, the National Archives and Records Administration (NARA) has harvested Congressional web sites at the end of each Congress. They also did a wider harvest of federal websites for the 2004 Presidential transition.
Consumer Complaint Database
From the Consumer Financial Protection Bureau
Corpus of Contemporary American English
the largest freely-available corpus of English, and the only large and balanced corpus of American English.
David D. Lewis - Test Collections
Reuters-21578 (and Reuters-22173): The most widely used text categorization test collection.

RCV1 (Reuters Corpus Volume 1): A large, high quality, recently released collection of news stories. Likely to become the new standard benchmark in text categorization research.

TREC-AP : A text categorization task based on the Associated Press articles used in the NIST TREC evaluations.
GloWbE: Corpus of Global Web-Based English
Download 440 million words of full-text data for COCA, or 1.8 billion words for GloWbE. With this data, you will have the corpora on your computer, rather than having to use the web interface. The data comes in three formats: tables for relational databases, word/lemma/PoS (vertical format), or text (linear format).
GovTrack’s bulk raw data (Congress)
HathiTrust
Non-Google digitized collection: Approximately 550,000 public domain volumes as of March 2015, primarily, though not exclusively, English language materials published prior to 1923. Google-digitized volumes (requires institutional signature): Approximately 4.8 million public domain volumes as of March 2015, representing a wide variety of languages, subjects, and dates. See the visualizations of HathiTrust public domain volumes.
Internet Archive
How to download in bulk using wget
JSTOR for Research
Data for Research is a free service for researchers wishing to analyze content on JSTOR through a variety of lenses and perspectives. If you require more than 1,000 documents or a type of data not available through the interactive portion of the site, please contact us at: support@ithaka.org
Open American National Corpus
OpenLibrary
Open Library is an open, editable library catalog, building towards a web page for every book ever published.
OpenSubtitles.org
Movie subtitles
Patents & Trademarks (Google bulk downloads)
Project Gutenberg
Project Gutenberg offers over 50,000 free ebooks. Note from Terms of Use: This website is intended for human users only. Any perceived use of automated tools to access this website will result in a temporary or permanent block of your IP address.
Public.Resource.Org
Bulk downloadable content harvested from government web sites and other sources
PubMed Central Open Access subset
Robots Reading Vogue
Project from Yale based on Vogue Archive (ProQuest)
SNAP: Stanford Network Analysis Project
Stanford Network Analysis Platform (SNAP) is a general purpose network analysis and graph mining library.
Text Creation Partnership
The Text Creation Partnership creates standardized, accurate XML/SGML encoded electronic text editions of early print books. We transcribe and mark up the text from the millions of page images in ProQuest's Early English Books Online, Gale Cengage's Eighteenth Century Collections Online, and Readex's Evans Early American Imprints.
University of Oxford Text Archive
The University of Oxford Text Archive develops, collects, catalogues and preserves electronic literary and linguistic resources for use in Higher Education, in research, teaching and learning. The OTA also gives advice on the creation and use of these resources, and is involved in the development of standards and infrastructure for electronic language resources.
Wikipedia - Wikimedia Downloads
WordHoard
tagged literary texts
Yahoo Webscope Datasets
A reference library of interesting and scientifically useful datasets for non-commercial use by academics and other scientists.

Tutorials

Neil Caren - Learning Python for Social Scientists
Includes examples from all kinds of data sources
Ted Underwood - Seven ways humanists are using computers to understand text.