Python Web Scraping

Learning Objectives

Web scraping is a useful technique for collecting information from websites when an appropriate API or other structured data source is not available.

What Is Web Scraping?

Web scraping is the process of programmatically retrieving information from web pages and extracting useful data from their HTML structure.

Instead of manually opening hundreds of pages and copying information, a Python program can automate repetitive collection tasks.

Website
   ↓
HTTP Request
   ↓
HTML
   ↓
HTML Parser
   ↓
Extract Data
   ↓
Clean Data
   ↓
Save / Analyze Data

Ethical & Legal Considerations

Technical ability does not automatically mean that every website should be scraped.

Before collecting data, check the website's terms, applicable laws, access restrictions and any published scraping guidance.

When an official API is available and suitable for your purpose, it is often preferable to use the API.

Understanding HTML

Web pages are primarily structured using HTML (HyperText Markup Language).

<html>

    <body>

        <h1>Python Products</h1>

        <p class="price">$500</p>

        <a href="/products">
            View Products
        </a>

    </body>

</html>

Scraping involves locating the HTML elements that contain the information you need.

Downloading a Web Page

The Requests library introduced in the previous lesson can retrieve the HTML of a web page.

import requests

url = "https://example.com"

response = requests.get(
    url,
    timeout=10
)

response.raise_for_status()

html = response.text

print(html)

At this stage, you have the raw HTML. The next step is parsing it.

Beautiful Soup

Beautiful Soup is a Python library designed to parse HTML and XML documents.

It provides convenient methods for navigating a document and finding the elements you need.

from bs4 import BeautifulSoup

html = """
<html>
    <body>
        <h1>Hello Python</h1>
    </body>
</html>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

print(soup.h1.text)

Hello Python

Installing Beautiful Soup

Install Beautiful Soup using pip.

python -m pip install beautifulsoup4

Requests and Beautiful Soup are often used together:

python -m pip install requests beautifulsoup4

Parsing HTML

Once you have HTML, create a Beautiful Soup object.

from bs4 import BeautifulSoup

html = """
<div>
    <h1>Laptop</h1>
    <p>$1200</p>
</div>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

print(soup.get_text())

Laptop
$1200

Beautiful Soup creates a structure that makes it easier to navigate the HTML document.

Finding HTML Tags

You can access common elements directly.

html = """
<h1>Python Course</h1>
<p>Learn Python step by step.</p>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

print(soup.h1)
print(soup.p)

You can also search for specific elements.

heading = soup.find("h1")

print(heading.text)

Python Course

find() and find_all()

find() returns the first matching element.

heading = soup.find("h1")

print(heading.text)

find_all() returns all matching elements.

paragraphs = soup.find_all("p")

for paragraph in paragraphs:
    print(paragraph.text)

This distinction is extremely important when scraping lists of products, articles or other repeated elements.

CSS Selectors

Beautiful Soup supports CSS selector-based searches through select() and select_one().

html = """
<div class="product">
    <h2>Laptop</h2>
    <p class="price">$1200</p>
</div>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

product = soup.select_one(".product")

price = product.select_one(".price")

print(price.text)

$1200

Common selectors include:

.product
.price
#main
div.product
div .price
a[href]

Extracting Text

Once an element is found, you can extract its text.

title = soup.find("h1")

print(title.text)

For more controlled extraction:

text = title.get_text(
    strip=True
)

print(text)

strip=True removes unnecessary whitespace around the extracted text.

Extracting Attributes

HTML elements often contain useful attributes.

html = """
<a href="/products/1">
    Laptop
</a>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

link = soup.find("a")

print(link["href"])

/products/1

You can also use:

print(link.get("href"))

get() is useful when an attribute might not exist.

Extracting Links

Links are represented by the HTML <a> element.

links = soup.find_all("a")

for link in links:

    url = link.get("href")

    text = link.get_text(
        strip=True
    )

    print(text, url)

A real website may contain links without an href, so using get() can make the code safer.

Scraping Tables

HTML tables can contain useful structured information.

html = """
<table>

    <tr>
        <th>Product</th>
        <th>Price</th>
    </tr>

    <tr>
        <td>Laptop</td>
        <td>1200</td>
    </tr>

</table>
"""

soup = BeautifulSoup(
    html,
    "html.parser"
)

rows = soup.select("table tr")

for row in rows:

    cells = row.find_all(
        ["th", "td"]
    )

    values = [
        cell.get_text(
            strip=True
        )
        for cell in cells
    ]

    print(values)

['Product', 'Price']
['Laptop', '1200']

Scraping Images

Image URLs are commonly stored in the src attribute.

images = soup.find_all("img")

for image in images:

    source = image.get("src")

    print(source)

You can also extract alternative text:

alt_text = image.get("alt")

print(alt_text)

Cleaning Scraped Data

Raw scraped information often contains whitespace, currency symbols, duplicated values or missing fields.

price_text = "$1,299.99"

clean_price = (
    price_text
    .replace("$", "")
    .replace(",", "")
)

price = float(clean_price)

print(price)

1299.99

Data cleaning is an important part of scraping because collecting data is only the first step.

Web Page
   ↓
Raw Data
   ↓
Clean Data
   ↓
Structured Data
   ↓
Analysis

Handling Scraping Errors

Web scraping combines network operations with HTML parsing, so multiple types of failures are possible.

import requests
from bs4 import BeautifulSoup

url = "https://example.com"

try:

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    title = soup.find("h1")

    if title:
        print(title.get_text(strip=True))
    else:
        print("Title not found.")

except requests.exceptions.Timeout:

    print("Request timed out.")

except requests.exceptions.RequestException as error:

    print("Request failed:", error)

Dynamic Websites

Not every website sends all visible content directly in the initial HTML response.

Some pages use JavaScript to retrieve or generate content after the page loads.

Browser
   ↓
Initial HTML
   ↓
JavaScript executes
   ↓
API request
   ↓
Additional data
   ↓
Final page

If the information you need is not present in the HTML returned by Requests, Beautiful Soup alone may not be enough.

Pagination

Product catalogs, article archives and search results are often divided into multiple pages.

/products?page=1
/products?page=2
/products?page=3

A simple scraper can follow multiple pages.

import requests
from bs4 import BeautifulSoup

for page in range(1, 4):

    url = (
        "https://example.com/products"
        f"?page={page}"
    )

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    products = soup.select(
        ".product"
    )

    for product in products:

        print(
            product.get_text(
                " ",
                strip=True
            )
        )

Real websites may use next-page links, cursor-based pagination or other mechanisms.

Practical Web Scraping Example

Let's combine Requests and Beautiful Soup into a small product scraper.

import requests
from bs4 import BeautifulSoup

url = "https://example.com/products"

try:

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    products = soup.select(
        ".product"
    )

    for product in products:

        name_element = product.select_one(
            ".product-name"
        )

        price_element = product.select_one(
            ".price"
        )

        if not name_element or not price_element:
            continue

        name = name_element.get_text(
            strip=True
        )

        price = price_element.get_text(
            strip=True
        )

        print(
            f"{name}: {price}"
        )

except requests.exceptions.RequestException as error:

    print("Scraping request failed:", error)

This example demonstrates a practical pattern:

Request
   ↓
Check Response
   ↓
Parse HTML
   ↓
Find Products
   ↓
Extract Fields
   ↓
Clean Data
   ↓
Process / Save

Web Scraping Best Practices

1. Prefer APIs When Available

An official API is usually more structured and stable than scraping HTML.

2. Use Timeouts

Network requests should not wait indefinitely.

3. Control Request Frequency

Avoid sending a large number of requests in a very short period.

4. Cache Results When Appropriate

If the same information does not need to be retrieved repeatedly, save previous results.

5. Validate Extracted Data

Websites can change their structure or return incomplete information.

6. Keep Scrapers Maintainable

Use functions and clear variable names rather than writing one giant script.

7. Respect Website Rules

Responsible scraping is part of professional data collection.

Common Mistakes

Practice Exercises

Mini Challenge — Ecommerce Product Scraper

Build a small ecommerce data collector that retrieves publicly accessible product information from a permitted practice website.

Starter Structure

import requests
from bs4 import BeautifulSoup


def scrape_products(url):

    response = requests.get(
        url,
        timeout=10
    )

    response.raise_for_status()

    soup = BeautifulSoup(
        response.text,
        "html.parser"
    )

    products = []

    for card in soup.select(
        ".product"
    ):

        name_element = card.select_one(
            ".product-name"
        )

        price_element = card.select_one(
            ".price"
        )

        if not name_element or not price_element:
            continue

        name = name_element.get_text(
            strip=True
        )

        price_text = price_element.get_text(
            strip=True
        )

        products.append({
            "name": name,
            "price": price_text
        })

    return products

Advanced Extension

After collecting the products:

• convert prices to numbers,
• find the cheapest product,
• find the most expensive product,
• calculate the average price,
• save the records into SQLite.

Website
   ↓
Requests
   ↓
Beautiful Soup
   ↓
Product Data
   ↓
Data Cleaning
   ↓
Analysis
   ↓
SQLite

Knowledge Check

Lesson Summary

Web scraping allows Python programs to collect information from web pages and turn unstructured HTML into useful data.

You now know how to collect and structure information from web pages. In the next lesson, we move into one of Python's most important data-science libraries: NumPy.