Take the 2-minute tour ×
English Language & Usage Stack Exchange is a question and answer site for linguists, etymologists, and serious English language enthusiasts. It's 100% free, no registration required.

I must admit that I'm unsure whether or not this is the right forum for this question. It may belong to statistics and AI also. If there exists a more suitable forum, then tell me.

The thing is, I want to analyse a lot of English text for an AI project (Confabulation theory). Is there an online collection of freely available English texts? Books, news would be preferred — scientific texts will properly not do, due to large amounts of math etc.

share
    
Lorem Ipsum placeholder text? (Used quite often for design mockups.) Not English though, but it could work. –  JFW May 10 '11 at 15:45
    
@JFW It won't :/ I need proper English text. –  Lasse Espeholt May 10 '11 at 15:50
3  
AI has been closed down, but I'll ask around if this would be on-topic on CrossValidated. Update: it would be indeed welcome there, especially if you don't want to limit the discussion to simply obtaining the data. However, if a) our community deems the question as it is on-topic for our site and b) you are actually satisfied with the quality of answers you're getting here, then of course there's no real need for migration. –  RegDwigнt May 10 '11 at 15:56
comments disabled on deleted / locked posts

locked by simchona Aug 20 '12 at 19:53

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off topic by simchona Aug 20 '12 at 19:52

Questions on English Language & Usage Stack Exchange are expected to relate to English language and usage within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

6 Answers

up vote 8 down vote accepted

Project Gutenberg contains more than 33000 classic books for free, downloadable in different formats.

Also, look at the affiliates sites, where you can find even more books.

EDIT: you may try to contact them to see if there is a way for you to download the books "painlessly" (e.g. with an automated script). I would suggest you ask permission before trying to download all those books automatically. Also, consider making a donation if you end up using their data.

EDIT2: here are the instructions for accessing the site with a robot.

share
2  
Thanks :) This link is suitable for robots: gutenberg.org/robot/harvest –  Lasse Espeholt May 10 '11 at 15:57
    
Good, I'll edit my answer –  nico May 10 '11 at 16:03
    
@mmeyers: thanks for the edit... I always misspell his name :P –  nico May 10 '11 at 17:18
    
Project Gutenberg have created several CDs and DVDs available via torrent/ftp/http/etc for easy download: The CD and DVD Project. –  Hugo Jun 13 '12 at 10:31
add comment

Project Gutenberg is a poor choice if you are looking for contemporary language, as most of the PG texts are from the early 20th century and earlier. For large samples of more recent text, you want one of the many available text corpora, such as the following:

Some corpora require a fee for access, and others are free to use. Consult each Web site for specifics.

share
add comment

If n-grams are fine (i.e., you don't need complete sentences and such), you could also try the Google n-grams dataset.

share
add comment

The Gutenberg project has a specific page for getting offline dumps efficiently - Information About Robot Access to our Pages

share
add comment

And for modern texts, there are a lot of blogs which use various Creative Commons licenses, most commonly CC BY-SA.

share
add comment

You can use Wikipedia and the Internet.

share
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.