IPWAY Blog

Using web scraping has become crucial for companies and developers to collect data from the internet effectively. Ruby, recognized for its simplicity and readability is widely used for web scraping projects. This detailed guide will explore the effective methods and strategies for web scraping using Ruby encompassing everything from setup to extracting information, from dynamic web pages.

Web scraping refers to the method of gathering data from websites enabling you to automate the extraction of amounts of information swiftly and effectively. With its syntax and robust libraries, Ruby proves to be a superb choice for web scraping tasks. This piece will walk you through the process of web scraping with Ruby covering everything, from initial setup to advanced strategies.

Is Ruby Good for Web Scraping?

Ruby is a programming language that is interpreted, open source and dynamically typed. It supports object oriented and procedural development. Ruby prioritizes simplicity with its syntax that is both easy to write and read naturally. This focus on efficiency has led to Ruby being used for various applications, including web scraping.

The abundance of third party libraries in Ruby, known as “gems ” makes it especially suitable for web scraping. These gems cover a range of tasks making it simple to download web pages analyze HTML content and extract data.

To sum up conducting web scraping with Ruby is not possible but also uncomplicated thanks, to the numerous libraries available.

Installing Ruby

Before you begin scraping data make sure to set up Ruby. Here are the step by step instructions, for installing Ruby on both Windows and macOS:

Installing Ruby on Windows

Download RubyInstaller:

• Visit the RubyInstaller website.
• Be sure to get the suggested edition of Ruby. The installation package comes with the Ruby programming language, RubyGems and a built in development environment.

Run the Installer:

• Please click on the file you just downloaded to begin the installation process.
• Make sure to follow the instructions displayed on the screen. Remember to tick the box that mentions “Include Ruby executables in your PATH” while installing. This is a step, for using Ruby through the command line.

Verify Installation:

• To open Command Prompt press the Windows key and R together type “cmd,”. Then press Enter.
• Make sure to type ‘ruby v’. Hit Enter. If you see the installed version of Ruby displayed it means that Ruby has been installed correctly on your system.

Update RubyGems:

• RubyGems serves as the go to package manager for Ruby. While it typically comes pre installed you have the option to update it by executing the specified command, in Command Prompt:

gem update --system

Installing Ruby on macOS

Using Homebrew:

• Homebrew is a tool, for macOS that helps you easily install software. If you don’t already have Homebrew, open Terminal and enter the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

• Once Homebrew is installed, install Ruby by running:

brew install ruby

Verify Installation:

• Open Terminal and type:

ruby -v

To confirm that Ruby has been installed on your macOS system execute this command to display the installed Ruby version.

Update RubyGems:

• Similar to Windows RubyGems serves as the package manager, for Ruby. Is already included in the setup. To perform an update simply execute:

gem update --system

Alternative Method for macOS

Using Ruby Version Manager (RVM):

• RVM is an used approach, for setting up Ruby enabling you to handle various Ruby versions. To install RVM execute the provided command in your Terminal:

\curl -sSL https://get.rvm.io | bash -s stable

• After installation, load RVM into your shell session:

source ~/.rvm/scripts/rvm

• Install the latest version of Ruby using RVM:

rvm install ruby

Verify Installation:

• Check the installed Ruby version:

ruby -v

Update RubyGems:

• As with other methods, update RubyGems if necessary:

gem update --system

Best Gems in Web Scraping Using Ruby

Ruby’s environment comprises a variety of gems that enhance the efficiency and ease of web scraping. These gems offer functionalities like handling HTTP requests, parsing HTML content, cookie management and more. Below are some recommended Ruby gems for web scraping that you might find beneficial:

Nokogiri

Nokogiri stands out as the go to gem in the Ruby community for parsing through HTML and XML. With its capability to navigate documents using CSS selectors and XPath it proves to be a tool for fetching information, from web pages.

Installation:

Install Nokogiri by running:

gem install nokogiri

Usage:

Here’s a basic example of how to use Nokogiri:

require 'nokogiri'
require 'httparty'

url = 'https://example.com'
response = HTTParty.get(url)
parsed_page = Nokogiri::HTML(response.body)

titles = parsed_page.css('h1').map(&:text)
puts titles

This script retrieves the HTML content, from the given URL analyzes it and retrieves the text contained within all
tags.

HTTParty

HTTParty is a tool that streamlines the process of sending HTTP requests. It’s user friendly. Works seamlessly with other popular Ruby gems such, as Nokogiri.

Installation:

Install HTTParty by running:

gem install httparty

Usage:

Here is a simple illustration of how to send an HTTP GET request.

require 'httparty'

url = 'https://example.com'
response = HTTParty.get(url)

if response.success?
  puts response.body
else
  puts "Failed to retrieve the webpage"
end

Mechanize

Mechanize is a tool designed to automate website interactions, managing cookies, sessions and form submissions. It’s handy, for extracting data from pages that need login credentials or other types of engagement.

Installation:

Install Mechanize by running:

gem install mechanize

Usage:

Here is a simple illustration of how to utilize Mechanize for extracting data from a website:

require 'mechanize'

agent = Mechanize.new
page = agent.get('https://example.com')
puts page.title

page.links.each do |link|
  puts link.text
end

Watir

Watir, also known as Web Application Testing in Ruby, is a tool for automating web browsers. It comes in handy when extracting content that relies on the execution of JavaScript.

Installation:

Install Watir and a web driver like Selenium by running:

gem install watir
gem install selenium-webdriver

Usage:

Here’s a basic example of using Watir to scrape a dynamic website:

require 'watir'

browser = Watir::Browser.new :chrome
browser.goto 'https://example.com'

puts browser.title
browser.close

Kimurai

Kimurai represents a web scraping framework designed for Ruby utilizing Nokogiri, Watir and Capybara as its foundation. With its user interface for overseeing various spiders it serves as a robust solution, for intricate scraping assignments.

Installation:

To set up Kimurai simply include it in your Gemfile. Then run bundle install.

gem 'kimurai'

Usage:

Here’s a basic example of a Kimurai spider:

require 'kimurai'

class ExampleSpider < Kimurai::Base
  @name = 'example_spider'
  @start_urls = ['https://example.com']
  @engine = :mechanize

  def parse(response, url:, data: {})
    response.css('h1').each do |heading|
      puts heading.text
    end
  end
end

ExampleSpider.crawl!

Scraping static pages

Static web pages contain content that is coded directly into the HTML source making them simpler to extract compared to pages. Now lets dive into the process of scraping web pages with Ruby, which includes sending HTTP requests analyzing HTML code and managing the extracted data.

Making an HTTP Request

When you want to scrape information from a webpage the initial thing to do is send an HTTP request to the specific URL. To accomplish this task in Ruby we rely on the HTTParty gem, which’s a handy tool, for handling HTTP requests.

Install HTTParty: To set up HTTParty open your terminal. Execute the command provided below.

gem install httparty

Make a Request: Lets make a Ruby script file, like “scrape_static.rb “. Add this code to send an HTTP GET request;

require 'httparty'

url = 'https://example.com'
response = HTTParty.get(url)

if response.success?
  puts response.body
else
  puts "Failed to retrieve the webpage"
end

This program retrieves the HTML content, from the given URL. Displays it on the screen.

Parsing HTML with Nokogiri

After obtaining the HTML content the next step is to analyze it to retrieve the information you’re looking for. Nokogiri stands out as a tool, for parsing through HTML and XML documents using Ruby programming language.

Install Nokogiri:

Install Nokogiri by running:

Install Nokogiri by running:

Parse HTML:

Add the following code to your script to parse the HTML using Nokogiri:

require 'nokogiri'
require 'httparty'

url = 'https://example.com'
response = HTTParty.get(url)

if response.success?
  parsed_page = Nokogiri::HTML(response.body)
  puts parsed_page.title
else
  puts "Failed to retrieve the webpage"
end

This piece of code retrieves the HTML content, from the webpage. Uses Nokogiri to analyze it. It then. Displays the title of the page.

Extracting Data

You have the ability to utilize Nokogiris CSS selectors to find and extract elements from the HTML document.

Locate Elements:

For instance to gather all the headings (designated as <h1> elements) from the webpage:

require 'nokogiri'
require 'httparty'

url = 'https://example.com'
response = HTTParty.get(url)

if response.success?
  parsed_page = Nokogiri::HTML(response.body)
  headings = parsed_page.css('h1')
  headings.each do |heading|
    puts heading.text
  end
else
  puts "Failed to retrieve the webpage"
end

Writing Scraped Data to a CSV File

Once you’ve gathered the information you may consider saving it in a CSV document. Ruby’s built-in CSV library makes this easy:

require 'csv'

CSV.open("data.csv", "w") do |csv|
  csv << ["Title", "URL"]
  csv << ["Example Title", "https://example.com"]
end

This program generates a CSV file. Saves the information into it.

Scraping Dynamic Pages

Creating web pages with JavaScript involves more complex methods for data extraction. Here’s a guide, on navigating them with Watir.

Required Installation

First, you need to install Watir and a web driver like Selenium:

gem install watir
gem install selenium-webdriver

Loading a Dynamic Website

Here’s how to load a dynamic website using Watir:

require 'watir'

browser = Watir::Browser.new :chrome
browser.goto 'https://example.com'

puts browser.title
browser.close

This code will launch the website using the Chrome browser and display the title.

Locating HTML Elements via CSS Selectors

You can locate and interact with HTML elements using CSS selectors:

require 'watir'

browser = Watir::Browser.new :chrome
browser.goto 'https://example.com'

element = browser.element(css: 'h1')
puts element.text

browser.close

This code snippet retrieves the content of the
tag present, on the webpage.

Handling Pagination

Numerous websites utilize pagination to present sets of data. Here is a guide, on managing it:

require 'watir'

browser = Watir::Browser.new :chrome
browser.goto 'https://example.com'

loop do
  puts browser.text
  next_button = browser.button(text: 'Next')
  break unless next_button.exists?

  next_button.click
  sleep 2 # wait for the page to load
end

browser.close

The program will go through the pages by clicking the “Next” button until it’s no longer available.

Creating a CSV File

Finally, save the scraped data to a CSV file:

require 'csv'
require 'watir'

browser = Watir::Browser.new :chrome
browser.goto 'https://example.com'

CSV.open("dynamic_data.csv", "w") do |csv|
  csv << ["Content"]

  loop do
    content = browser.text
    csv << [content]

    next_button = browser.button(text: 'Next')
    break unless next_button.exists?

    next_button.click
    sleep 2 # wait for the page to load
  end
end

browser.close

Conclusion

Using Ruby for web scraping is a method to collect information from websites. The simplicity of Ruby and the presence of gems such as Nokogiri, HTTParty, Mechanize and Watir make it a fantastic option for newcomers and seasoned developers alike. Whether you’re extracting data, from dynamic pages Ruby equips you with the necessary tools to complete the task effectively. Dive into the realm of web scraping using Ruby and harness the capabilities of automated data retrieval.

Take your data scraping to the next level with IPWAY’s datacenter proxies!