Robby on Rails: Get to Know a Gem: Hpricotthoughts.sort_by{|t| t[:topic]}.collect tag:www.robbyonrails.com,2005:TypoTypo2007-02-13T08:48:47-05:00Robby Russellurn:uuid:5fe64d73-34ca-4619-90ad-4657c497533f2007-02-13T08:48:00-05:002007-02-13T08:48:47-05:00Get to Know a Gem: Hpricot<p>In this new series, <em>Get to Know a Gem</em>, we’re going to take a look at <a href="http://code.whytheluckystiff.net/hpricot/">hpricot</a>.</p>
<h2>What is Hpricot?</h2>
<p>WhyTheLuckyStiff <a href="http://redhanded.hobix.com/inspect/hpricot01.html">released Hpricot in July of 2006</a> in an effort to bring fast <span class="caps">HTML</span> parsing to the masses. It’s currently unknown what prompted it, but my guess would be that Why is secretly scraping all the pages on the internet that archive the future. To speed it up, Why has written the Hpricot scanner in C, to be much faster than the other options available in Ruby.</p>
<h2>Installation</h2>
<p>This process… is as always with most gems, very simple.</p>
<pre><code>
$ sudo gem install hpricot
Password:
Need to update 23 gems from http://gems.rubyforge.org
.......................
complete
Select which gem to install for your platform (powerpc-darwin8.7.0)
1. hpricot 0.5 (ruby)
2. hpricot 0.5 (mswin32)
3. hpricot 0.4 (mswin32)
4. hpricot 0.4 (ruby)
5. Cancel installation
> 1
</code></pre>
<p>Great, let’s now play with it!</p>
<h2>Usage</h2>
<p>In this first example, we’re going to use Hpricot to parse a web page through the Open-URI library. For this, we’ll need to require a few libs.</p>
<pre><code>
require 'rubygems'
require 'hpricot'
require 'open-uri'
</code></pre>
Now that we have the libraries loaded, we can create a new Hpricot object and in this example, we’ll load the <a href="http://www.planetargon.com/about.html"><span class="caps">PLANET ARGON</span> About page</a>.
<pre><code>
# Open the PLANET ARGON about page
page = Hpricot( open( 'http://www.planetargon.com/about.html' ) )
</code></pre>
<p>Great, let’s have some parsing fun. Let’s parse for the first instance of a <code>div</code> with a class name of <code>team</code>. Hpricot will return array of elements that meet your search request.</p>
<pre><code>
page.search( "//div[@class='team']" ).size
=> 7
</code></pre>
<p>Great, this is a good sign that I need to add several people to the website. :-)</p>
<p>If we want to peak at the first instance of this class, we can do:</p>
<pre><code>
page.search( "//div[@class='team']" ).first
=> {elem <div class="team"> "\n" {elem <div class="team_name"> {elem <strong> "Robby Russell" </strong>} ", Founder &#38; Executive Director" </div>} ....SNIP
</code></pre>
<p>You’ll notice that there is a <strong> element within the results, which we can search deeper into this tree.</p>
<pre><code>
page.search( "//div[@class='team']" ).first.search( "//strong" )
=> #<Hpricot::Elements[{elem <strong> "Robby Russell" </strong>}]>
</code></pre>
<p>Hpricot provides a method named <code>inner_html</code>, which will return the contents within the element.</p>
<pre><code>
page.search( "//div[@class='team']" ).first.search( "//strong" ).inner_html
=> "Robby Russell"
</code></pre>
<p>Let’s now iterate through each of the elements and output all of the team member names.</p>
<pre><code>
# search for each team member div and iterate through them
page.search( "//div[@class='team']" ).each do |team|
puts team.search( "//strong").inner_html
end
Robby Russell
Allison Beckwith
Brian Ford
Nicole Fritz
Alain Bloch
Audrey Eschright
Gary Blessington
</code></pre>
<p>So, there you have it. A quick and basic introduction into using Hpricot for parsing <span class="caps">HTML</span> content. You can use Hpricot for a wide variety of structured data, such as <span class="caps">XML</span> and <span class="caps">CSS</span>. For more examples, please visit the <a href="http://code.whytheluckystiff.net/hpricot/wiki/HpricotBasics">HpricotBasics</a> page.</p>
<h2>Final Thoughts</h2>
<p>I’m going to guess that Why built this for <a href="http://hoodwink.d">hoodwink.d</a>, which I’ve been a regular user of for a <a href="http://www.robbyonrails.com/articles/2005/08/23/boys-from-the-hoodwink-d">long time</a>. I haven’t spent much time playing with the <a href="http://www.w3.org/TR/xpath">XPath syntax</a> and playing around with Hpricot has given me a much better understanding of it.</p>
<p>As mentioned at the beginning of this post, I am going to make <em>Getting to Know a Gem</em> a regular feature on my blog. If you know of a lesser known Gem that needs some attention, please <a href="mailto:suggestions@robbyonrails.com">send a suggestion to me</a>.</p>
<p>Until next time…</p><p>In this new series, <em>Get to Know a Gem</em>, we’re going to take a look at <a href="http://code.whytheluckystiff.net/hpricot/">hpricot</a>.</p>
<h2>What is Hpricot?</h2>
<p>WhyTheLuckyStiff <a href="http://redhanded.hobix.com/inspect/hpricot01.html">released Hpricot in July of 2006</a> in an effort to bring fast <span class="caps">HTML</span> parsing to the masses. It’s currently unknown what prompted it, but my guess would be that Why is secretly scraping all the pages on the internet that archive the future. To speed it up, Why has written the Hpricot scanner in C, to be much faster than the other options available in Ruby.</p>
<h2>Installation</h2>
<p>This process… is as always with most gems, very simple.</p>
<pre><code>
$ sudo gem install hpricot
Password:
Need to update 23 gems from http://gems.rubyforge.org
.......................
complete
Select which gem to install for your platform (powerpc-darwin8.7.0)
1. hpricot 0.5 (ruby)
2. hpricot 0.5 (mswin32)
3. hpricot 0.4 (mswin32)
4. hpricot 0.4 (ruby)
5. Cancel installation
> 1
</code></pre>
<p>Great, let’s now play with it!</p>
<h2>Usage</h2>
<p>In this first example, we’re going to use Hpricot to parse a web page through the Open-URI library. For this, we’ll need to require a few libs.</p>
<pre><code>
require 'rubygems'
require 'hpricot'
require 'open-uri'
</code></pre>
Now that we have the libraries loaded, we can create a new Hpricot object and in this example, we’ll load the <a href="http://www.planetargon.com/about.html"><span class="caps">PLANET ARGON</span> About page</a>.
<pre><code>
# Open the PLANET ARGON about page
page = Hpricot( open( 'http://www.planetargon.com/about.html' ) )
</code></pre>
<p>Great, let’s have some parsing fun. Let’s parse for the first instance of a <code>div</code> with a class name of <code>team</code>. Hpricot will return array of elements that meet your search request.</p>
<pre><code>
page.search( "//div[@class='team']" ).size
=> 7
</code></pre>
<p>Great, this is a good sign that I need to add several people to the website. :-)</p>
<p>If we want to peak at the first instance of this class, we can do:</p>
<pre><code>
page.search( "//div[@class='team']" ).first
=> {elem <div class="team"> "\n" {elem <div class="team_name"> {elem <strong> "Robby Russell" </strong>} ", Founder &#38; Executive Director" </div>} ....SNIP
</code></pre>
<p>You’ll notice that there is a <strong> element within the results, which we can search deeper into this tree.</p>
<pre><code>
page.search( "//div[@class='team']" ).first.search( "//strong" )
=> #<Hpricot::Elements[{elem <strong> "Robby Russell" </strong>}]>
</code></pre>
<p>Hpricot provides a method named <code>inner_html</code>, which will return the contents within the element.</p>
<pre><code>
page.search( "//div[@class='team']" ).first.search( "//strong" ).inner_html
=> "Robby Russell"
</code></pre>
<p>Let’s now iterate through each of the elements and output all of the team member names.</p>
<pre><code>
# search for each team member div and iterate through them
page.search( "//div[@class='team']" ).each do |team|
puts team.search( "//strong").inner_html
end
Robby Russell
Allison Beckwith
Brian Ford
Nicole Fritz
Alain Bloch
Audrey Eschright
Gary Blessington
</code></pre>
<p>So, there you have it. A quick and basic introduction into using Hpricot for parsing <span class="caps">HTML</span> content. You can use Hpricot for a wide variety of structured data, such as <span class="caps">XML</span> and <span class="caps">CSS</span>. For more examples, please visit the <a href="http://code.whytheluckystiff.net/hpricot/wiki/HpricotBasics">HpricotBasics</a> page.</p>
<h2>Final Thoughts</h2>
<p>I’m going to guess that Why built this for <a href="http://hoodwink.d">hoodwink.d</a>, which I’ve been a regular user of for a <a href="http://www.robbyonrails.com/articles/2005/08/23/boys-from-the-hoodwink-d">long time</a>. I haven’t spent much time playing with the <a href="http://www.w3.org/TR/xpath">XPath syntax</a> and playing around with Hpricot has given me a much better understanding of it.</p>
<p>As mentioned at the beginning of this post, I am going to make <em>Getting to Know a Gem</em> a regular feature on my blog. If you know of a lesser known Gem that needs some attention, please <a href="mailto:suggestions@robbyonrails.com">send a suggestion to me</a>.</p>
<p>Until next time…</p>