Simple Ruby Screen Scraper using Mechanize, Hpricot and XPath

In my previous post, simple-ruby-screen-scraper-in-just-5.html, I explained how to do Screen Scraping in just 5 lines without using XPath. In this post, I will achieve the same task using XPath, before going through this, I would recommend you to go through previous post to get an idea about the complete scenario.

Here also I will be scraping a link "Quality Assurance" given on the left panel under "My Blogs" section on my website "http://www.kumarritesh.com/" with XPath. Here is the code:

require 'rubygems'
require 'mechanize'
require 'hpricot'

agent = Mechanize.new
page = agent.get('http://www.kumarritesh.com')
@response = page.content
doc = Hpricot(@response)

(doc/"/html/body/div/div/div[2]/div[2]/div[3]/div/ul/li/a")[0].innerHTML

All the code is same as mentioned in the previous post, only last line changes.

In my previous post, I parsed the page by searching the div class, li id and anchor link.

Here, I will make use of XPath. 

To get XPath, inspect the "Quality Assurance" link in Firebug. You will get the code:

<a href="/index.php/quality-assurance">Quality Assurance</a>

Right click and select "Copy XPath" option. You will get following code:

/html/body/div/div/div[2]/div[2]/div[3]/div/ul/li/a

 

Pass the code as displayed in the last line, it will return all the links, so to get the first link we have taken the zeroth element and then its HTML format. Similarly, we can have the XPath of any text, label, links and can scrap it very easily.

Simple Ruby Screen Scraper in just 5 lines without using XPath

Simple Ruby Screen Scraper in just 5 lines, true, its very -very basic scrapper that I will  explain in this article. You can call it Screen Scraping, Web Screen Scraping, Data Scraping, Website Scraping, Web Page Scraping, Web Crawler .... ....... my main aim is to explain how to extract a particular link or label from a web page.

For scraping, I am using Ruby version - 1.8.7 which should be installed on your system. Along with this, Mechanize (1.0.0) and Hpricot  (0.8.4) gem. I will be scraping a link "Quality Assurance" given on the left panel under "My Blogs" section on my website "http://www.kumarritesh.com/". So lets start.

1. Open command prompt (I am working on Windows 7). Then open Interactive Ruby Shell (irb) prompt.

C:\Users\ritesh>irb
irb(main):001:0>

2. Open site "http://www.kumarritesh.com/" and inspect the div in which link "Quality Assurance" is present. Web page Code of that block looks like the code given below:

<div id="module_77" class="tabcontent tabopen" tabindex="-1" role="tabpanel" aria-hidden="false" aria-expanded="true" aria-labelledby="link_77">
  <ul class="menu">
    <li id="item-466"><a href="/index.php/quality-assurance">Quality Assurance</a></li>
    <li id="item-467"><a href="/index.php/search-engine-optimization">Search Engine Optimization</a></li>
    <li id="item-468"><a href="/index.php/ruby-on-rails">Ruby on Rails</a></li>
    <li id="item-470"><a href="/index.php/blogs-links">Blog Links</a></li>
  </ul>
</div>

 3. Write the codes as follows:

require 'rubygems'
require 'mechanize'
require 'hpricot'

agent = Mechanize.new
page = agent.get('http://www.kumarritesh.com')
@response = page.content
doc = Hpricot(@response)
doc.search("//div[@class='tabcontent tabopen']").search("//li[@id ='item-466']").search("a").innerHTML

Last line will give you the output as "Quality Assurance". Very much confused with the code, let me take you through the code step by step.

First three lines load hpricot, rubygems and mechanize

require 'rubygems'
require 'mechanize'
require 'hpricot'

Instantiate a new mechanize object:

agent = Mechanize.new  

Now we'll use the agent we've created to fetch website page "http://www.kumarritesh.com/" and store the page content in an object, then passing that object to Hpricot. Hpricot loads the contents into a document object.

page = agent.get('http://www.kumarritesh.com')
@response = page.content
doc = Hpricot(@response)

 Now, last line will give you the expected output by parsing the doc

doc.search("//div[@class='tabcontent tabopen']").search("//li[@id ='item-466']").search("a").innerHTML

 What it does is it searches for the div class 'tabcontent tabopen' and then inside that div class, searches for li with id 'item-466' and then finally searches for anchor link.

'innerHTML' will remove all HTML elements and thus you get the result as "Quality Assurance"

In the coming articles, I will be taking you deep into the ocean of scraping where you will be diving like a big whale here and there finding some useful & interesting concepts, so stay tuned