English Language & Usage Stack Exchange is a question and answer site for linguists, etymologists, and serious English language enthusiasts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I must admit that I'm unsure whether or not this is the right forum for this question. It may belong to statistics and AI also. If there exists a more suitable forum, then tell me.

The thing is, I want to analyse a lot of English text for an AI project (Confabulation theory). Is there an online collection of freely available English texts? Books, news would be preferred — scientific texts will properly not do, due to large amounts of math etc.


locked by simchona Aug 20 '12 at 19:53

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by simchona Aug 20 '12 at 19:52

Questions on English Language & Usage Stack Exchange are expected to relate to English language and usage within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Lorem Ipsum placeholder text? (Used quite often for design mockups.) Not English though, but it could work. – JFW May 10 '11 at 15:45
@JFW It won't :/ I need proper English text. – Lasse Espeholt May 10 '11 at 15:50
AI has been closed down, but I'll ask around if this would be on-topic on CrossValidated. Update: it would be indeed welcome there, especially if you don't want to limit the discussion to simply obtaining the data. However, if a) our community deems the question as it is on-topic for our site and b) you are actually satisfied with the quality of answers you're getting here, then of course there's no real need for migration. – RegDwigнt May 10 '11 at 15:56
up vote 8 down vote accepted

Project Gutenberg contains more than 33000 classic books for free, downloadable in different formats.

Also, look at the affiliates sites, where you can find even more books.

EDIT: you may try to contact them to see if there is a way for you to download the books "painlessly" (e.g. with an automated script). I would suggest you ask permission before trying to download all those books automatically. Also, consider making a donation if you end up using their data.

EDIT2: here are the instructions for accessing the site with a robot.

Thanks :) This link is suitable for robots: gutenberg.org/robot/harvest – Lasse Espeholt May 10 '11 at 15:57
Good, I'll edit my answer – nico May 10 '11 at 16:03
@mmeyers: thanks for the edit... I always misspell his name :P – nico May 10 '11 at 17:18
Project Gutenberg have created several CDs and DVDs available via torrent/ftp/http/etc for easy download: The CD and DVD Project. – Hugo Jun 13 '12 at 10:31

Project Gutenberg is a poor choice if you are looking for contemporary language, as most of the PG texts are from the early 20th century and earlier. For large samples of more recent text, you want one of the many available text corpora, such as the following:

Some corpora require a fee for access, and others are free to use. Consult each Web site for specifics.


If n-grams are fine (i.e., you don't need complete sentences and such), you could also try the Google n-grams dataset.


The Gutenberg project has a specific page for getting offline dumps efficiently - Information About Robot Access to our Pages


And for modern texts, there are a lot of blogs which use various Creative Commons licenses, most commonly CC BY-SA.


You can use Wikipedia and the Internet.


Not the answer you're looking for? Browse other questions tagged or ask your own question.