Simple Ruby Scraping Script and storing the result in database

In my previous post Simple Ruby Screen Scraper using Mechanize, Hpricot and XPath , I explained how to do screen scraping by writing the code on irb (Interactive Ruby Shell) console. Instead of writing the script line by line on console we can create a Ruby script that can be executed which will make our task much easier. So, in this post you will find how to write a Ruby script, how to execute a Ruby script, also how to scrap and store the result in table.

Here I will be scraping all the links given on the left panel under "My Blogs" section on my website "http://www.kumarritesh.com/" with XPath and will store the result in table in db. To store the result in table, let us create the table first in MySql. I named the db as 'scraping_db' and table as 'scraps'. Here is the code to create the db and table

CREATE DATABASE /*!32312 IF NOT EXISTS*/`scraping_db` /*!40100 DEFAULT CHARACTER SET latin1 */;
USE `scraping_db`;

/*Table structure for table `scraps` */

DROP TABLE IF EXISTS `scraps`;

CREATE TABLE `scraps` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `links` varchar(255) DEFAULT NULL,
  `created_at` datetime DEFAULT NULL,
  `updated_at` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;

Now, I created the script file named 'screen-scraper.rb' and wrote the following code:

require 'rubygems'
require 'open-uri'
require 'hpricot'
require 'active_record'

ActiveRecord::Base.establish_connection(
    :adapter  => 'mysql',
    :host     => 'localhost',
    :username => 'root',
    :password => '',
    :database => 'scraping_db')

class Scrap < ActiveRecord::Base
  @url      = "http://www.kumarritesh.com"
  @response = ''
  open(@url, "User-Agent" => "Ruby/#{RUBY_VERSION}"  ) 
{ |f|
    @response = f.read
  }

  doc   = Hpricot(@response)
  links = (doc/"/html/body/div/div/div[2]/div[2]/div[3]/div/ul/li/a").innerHTML
  puts "#{links} "

  links.each do |link|
    scraplink = Scrap.new(:links => link)
    scraplink.save
    puts "#{link} "
  end
end

Browse to the folder where this script is saved. Execute the above code by command "ruby screen-scraper.rb"
It will print "Quality AssuranceSearch Engine OptimizationRuby on RailsBlog Links" twice on the console and will create one record with data in the links column as "Quality AssuranceSearch Engine OptimizationRuby on RailsBlog Links".

So, now you know how to write a Ruby script, how to execute a Ruby script, also how to scrap and store the result in table.

Simple Ruby Screen Scraper using Mechanize, Hpricot and XPath

In my previous post, simple-ruby-screen-scraper-in-just-5.html, I explained how to do Screen Scraping in just 5 lines without using XPath. In this post, I will achieve the same task using XPath, before going through this, I would recommend you to go through previous post to get an idea about the complete scenario.

Here also I will be scraping a link "Quality Assurance" given on the left panel under "My Blogs" section on my website "http://www.kumarritesh.com/" with XPath. Here is the code:

require 'rubygems'
require 'mechanize'
require 'hpricot'

agent = Mechanize.new
page = agent.get('http://www.kumarritesh.com')
@response = page.content
doc = Hpricot(@response)

(doc/"/html/body/div/div/div[2]/div[2]/div[3]/div/ul/li/a")[0].innerHTML

All the code is same as mentioned in the previous post, only last line changes.

In my previous post, I parsed the page by searching the div class, li id and anchor link.

Here, I will make use of XPath. 

To get XPath, inspect the "Quality Assurance" link in Firebug. You will get the code:

<a href="/index.php/quality-assurance">Quality Assurance</a>

Right click and select "Copy XPath" option. You will get following code:

/html/body/div/div/div[2]/div[2]/div[3]/div/ul/li/a

 

Pass the code as displayed in the last line, it will return all the links, so to get the first link we have taken the zeroth element and then its HTML format. Similarly, we can have the XPath of any text, label, links and can scrap it very easily.

Simple Ruby Screen Scraper in just 5 lines without using XPath

Simple Ruby Screen Scraper in just 5 lines, true, its very -very basic scrapper that I will  explain in this article. You can call it Screen Scraping, Web Screen Scraping, Data Scraping, Website Scraping, Web Page Scraping, Web Crawler .... ....... my main aim is to explain how to extract a particular link or label from a web page.

For scraping, I am using Ruby version - 1.8.7 which should be installed on your system. Along with this, Mechanize (1.0.0) and Hpricot  (0.8.4) gem. I will be scraping a link "Quality Assurance" given on the left panel under "My Blogs" section on my website "http://www.kumarritesh.com/". So lets start.

1. Open command prompt (I am working on Windows 7). Then open Interactive Ruby Shell (irb) prompt.

C:\Users\ritesh>irb
irb(main):001:0>

2. Open site "http://www.kumarritesh.com/" and inspect the div in which link "Quality Assurance" is present. Web page Code of that block looks like the code given below:

<div id="module_77" class="tabcontent tabopen" tabindex="-1" role="tabpanel" aria-hidden="false" aria-expanded="true" aria-labelledby="link_77">
  <ul class="menu">
    <li id="item-466"><a href="/index.php/quality-assurance">Quality Assurance</a></li>
    <li id="item-467"><a href="/index.php/search-engine-optimization">Search Engine Optimization</a></li>
    <li id="item-468"><a href="/index.php/ruby-on-rails">Ruby on Rails</a></li>
    <li id="item-470"><a href="/index.php/blogs-links">Blog Links</a></li>
  </ul>
</div>

 3. Write the codes as follows:

require 'rubygems'
require 'mechanize'
require 'hpricot'

agent = Mechanize.new
page = agent.get('http://www.kumarritesh.com')
@response = page.content
doc = Hpricot(@response)
doc.search("//div[@class='tabcontent tabopen']").search("//li[@id ='item-466']").search("a").innerHTML

Last line will give you the output as "Quality Assurance". Very much confused with the code, let me take you through the code step by step.

First three lines load hpricot, rubygems and mechanize

require 'rubygems'
require 'mechanize'
require 'hpricot'

Instantiate a new mechanize object:

agent = Mechanize.new  

Now we'll use the agent we've created to fetch website page "http://www.kumarritesh.com/" and store the page content in an object, then passing that object to Hpricot. Hpricot loads the contents into a document object.

page = agent.get('http://www.kumarritesh.com')
@response = page.content
doc = Hpricot(@response)

 Now, last line will give you the expected output by parsing the doc

doc.search("//div[@class='tabcontent tabopen']").search("//li[@id ='item-466']").search("a").innerHTML

 What it does is it searches for the div class 'tabcontent tabopen' and then inside that div class, searches for li with id 'item-466' and then finally searches for anchor link.

'innerHTML' will remove all HTML elements and thus you get the result as "Quality Assurance"

In the coming articles, I will be taking you deep into the ocean of scraping where you will be diving like a big whale here and there finding some useful & interesting concepts, so stay tuned