There are a number of services out there such as Google Cash Detective that will go run some searches on Google and then save the advertisements so you can track who is advertising for what keywords over time. It’s actually a very accurate technique for finding out what ads are profitable.
After tracking a keyword for several weeks it’s possible to see what ads have been running consistently over time. The nature of Pay Per Click is that only profitable advertisements will continue to run long term. So if you can identify what ads, for what keywords are profitable then it should be possible to duplicate them and get some of that profitable traffic for yourself.
The following script is a Python program that perhaps breaks the Google terms of service. So consider it as a guide for how this kind of HTML parsing could be done. It spoofs the User-agent to appear as though it is a real browser, and then does a search through all the keywords stored in an sqlite database and stores the ads displayed for that keyword in the database.
The script makes use of the awesome Beautiful Soup library. Beautiful Soup makes parsing HTML content really easy. But because of the nature of scraping the web it is very fragile since it makes several assumptions about the structure of the Google results page and if they change their site then the script could break.
#!/usr/bin/env python import sys import urllib2 import re import sqlite3 import datetime from BeautifulSoup import BeautifulSoup # available at: http://www.crummy.com/software/BeautifulSoup/ conn = sqlite3.connect("espionage.sqlite") conn.row_factory = sqlite3.Row def get_google_search_results(keywordPhrase): """make the GET request to Google.com for the keyword phrase and return the HTML text """ url='http://www.google.com/search?hl=en&q=' + '+'.join(keywordPhrase.split()) req = urllib2.Request(url) req.add_header('User-agent', 'Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13') page = urllib2.urlopen(req) HTML = page.read() return HTML def scrape_ads(text, phraseID): """Scrape the text as HTML, find and parse out all the ads and store them in a database """ soup = BeautifulSoup(text) #get the ads on the right hand side of the page ads = soup.find(id='rhsline').findAll('li') position = 0 for ad in ads: position += 1 #display url parts = ad.find('cite').findAll(text=True) site = ''.join([word.strip() for word in parts]).strip() ad.find('cite').replaceWith("") #the header line parts = ad.find('a').findAll(text=True) title = ' '.join([word.strip() for word in parts]).strip() #the destination URL href = ad.find('a')['href'] start = href.find('&q=') if start != -1 : dest = href[start+3:] else : dest = None print 'error', href ad.find('a').replaceWith("") #body of ad brs = ad.findAll('br') for br in brs: br.replaceWith("%BR%") parts = ad.findAll(text=True) body = ' '.join([word.strip() for word in parts]).strip() line1 = body.split('%BR%').strip() line2 = body.split('%BR%').strip() #see if the ad is in the database c = conn.cursor() c.execute('SELECT adID FROM AdTable WHERE destination=? and title=? and line1=? and line2=? and site=? and phraseID=?', (dest, title, line1, line2, site, phraseID)) result = c.fetchall() if len(result) == 0: #NEW AD - insert into the table c.execute('INSERT INTO AdTable (`destination`, `title`, `line1`, `line2`, `site`, `phraseID`) VALUES (?,?,?,?,?,?)', (dest, title, line1, line2, site, phraseID)) conn.commit() c.execute('SELECT adID FROM AdTable WHERE destination=? and title=? and line1=? and line2=? and site=? and phraseID=?', (dest, title, line1, line2, site, phraseID)) result = c.fetchall() elif len(result) > 1: continue adID = result['adID'] c.execute('INSERT INTO ShowTime (`adID`,`date`,`time`, `position`) VALUES (?,?,?,?)', (adID, datetime.datetime.now(), datetime.datetime.now(), position)) def do_all_keywords(): c = conn.cursor() c.execute('SELECT * FROM KeywordList') result = c.fetchall() for row in result: html = get_google_search_results(row['keywordPhrase']) scrape_ads(html, row['phraseID']) if __name__ == '__main__' : do_all_keywords()