12
votes

I must admit that I'm unsure whether or not this is the right forum for this question. It may belong to statistics and AI also. If there exists a more suitable forum, then tell me.

The thing is, I want to analyse a lot of English text for an AI project (Confabulation theory). Is there an online collection of freely available English texts? Books, news would be preferred — scientific texts will properly not do, due to large amounts of math etc.

3
  • Lorem Ipsum placeholder text? (Used quite often for design mockups.) Not English though, but it could work.
    – JFW
    Commented May 10, 2011 at 15:45
  • @JFW It won't :/ I need proper English text.
    – Lasse Espeholt
    Commented May 10, 2011 at 15:50
  • 3
    AI has been closed down, but I'll ask around if this would be on-topic on CrossValidated. Update: it would be indeed welcome there, especially if you don't want to limit the discussion to simply obtaining the data. However, if a) our community deems the question as it is on-topic for our site and b) you are actually satisfied with the quality of answers you're getting here, then of course there's no real need for migration.
    – RegDwigнt
    Commented May 10, 2011 at 15:56

6 Answers 6

Reset to default
8
votes

Project Gutenberg contains more than 33000 classic books for free, downloadable in different formats.

Also, look at the affiliates sites, where you can find even more books.

EDIT: you may try to contact them to see if there is a way for you to download the books "painlessly" (e.g. with an automated script). I would suggest you ask permission before trying to download all those books automatically. Also, consider making a donation if you end up using their data.

EDIT2: here are the instructions for accessing the site with a robot.

4
  • 2
    Thanks :) This link is suitable for robots: gutenberg.org/robot/harvest
    – Lasse Espeholt
    Commented May 10, 2011 at 15:57
  • Good, I'll edit my answer
    – nico
    Commented May 10, 2011 at 16:03
  • @mmeyers: thanks for the edit... I always misspell his name :P
    – nico
    Commented May 10, 2011 at 17:18
  • Project Gutenberg have created several CDs and DVDs available via torrent/ftp/http/etc for easy download: The CD and DVD Project.
    – Hugo
    Commented Jun 13, 2012 at 10:31
6
votes

Project Gutenberg is a poor choice if you are looking for contemporary language, as most of the PG texts are from the early 20th century and earlier. For large samples of more recent text, you want one of the many available text corpora, such as the following:

Some corpora require a fee for access, and others are free to use. Consult each Web site for specifics.

2
votes

If n-grams are fine (i.e., you don't need complete sentences and such), you could also try the Google n-grams dataset.

1
vote

The Gutenberg project has a specific page for getting offline dumps efficiently - Information About Robot Access to our Pages

0
votes

And for modern texts, there are a lot of blogs which use various Creative Commons licenses, most commonly CC BY-SA.

0
votes

You can use Wikipedia and the Internet.

Not the answer you're looking for? Browse other questions tagged .