Another in my series of Python scripting blog posts. This time I’m sharing a script that can rip through RSS feeds and devour their content and stuff it into a database in a way that scales up to 1000s of feeds. To accomplish this the script is multi-threaded.
The big problem with scaling up a web script like this is that there is a huge amount of latency when requesting something over the internet. Due to the bandwidth as well as remote processing time it can take as long as a couple of seconds to get anything back. Requesting one feed after the other in series will waste a lot of time, and that makes this type of script a prime candidate for some threading.
I borrowed parts of this script from this post: Threaded data collection with Python, including examples
What could you do with all this content? Just off the top of my head I can think of many interesting things to do:
- Create histograms of the publish times of posts to find out the most/least popular days and times are for publishing
- Plot trends of certain words or phrases over time
- create your own aggregation website
- get the trending topics by doing counting the occurrence of words by day
- Try writing some natural language processing algorithms
This script is coded at 20 threads, but that really needs to be fine tuned for the best performance. Depending on your bandwidth and the sites you want to grab you may want to tweak the THREAD_LIMIT value.
import sqlite3 import threading import time import Queue from time import strftime import feedparser # available at http://feedparser.org THREAD_LIMIT = 20 jobs = Queue.Queue(0) rss_to_process = Queue.Queue(THREAD_LIMIT) DATABASE = "rss.sqlite" conn = sqlite3.connect(DATABASE) conn.row_factory = sqlite3.Row c = conn.cursor() #insert initial values into feed database c.execute('CREATE TABLE IF NOT EXISTS RSSFeeds (id INTEGER PRIMARY KEY AUTOINCREMENT, url VARCHAR(1000));') c.execute('CREATE TABLE IF NOT EXISTS RSSEntries (entry_id INTEGER PRIMARY KEY AUTOINCREMENT, id, url, title, content, date);') c.execute("INSERT INTO RSSFeeds(url) VALUES('http://www.halotis.com/feed/');") feeds = c.execute('SELECT id, url FROM RSSFeeds').fetchall() def store_feed_items(id, items): """ Takes a feed_id and a list of items and stored them in the DB """ for entry in items: c.execute('SELECT entry_id from RSSEntries WHERE url=?', (entry.link,)) if len(c.fetchall()) == 0: c.execute('INSERT INTO RSSEntries (id, url, title, content, date) VALUES (?,?,?,?,?)', (id, entry.link, entry.title, entry.summary, strftime("%Y-%m-%d %H:%M:%S",entry.updated_parsed))) def thread(): while True: try: id, feed_url = jobs.get(False) # False = Don't wait except Queue.Empty: return entries = feedparser.parse(feed_url).entries rss_to_process.put((id, entries), True) # This will block if full for info in feeds: # Queue them up jobs.put([info['id'], info['url']]) for n in xrange(THREAD_LIMIT): t = threading.Thread(target=thread) t.start() while threading.activeCount() > 1 or not rss_to_process.empty(): # That condition means we want to do this loop if there are threads # running OR there's stuff to process try: id, entries = rss_to_process.get(False, 1) # Wait for up to a second except Queue.Empty: continue store_feed_items(id, entries) conn.commit()