How to scrape a list of items from a webpage with Python

If the page is plain server-rendered HTML — no login, no JavaScript building the list after load — Python's standard library is enough to pull repeated items (product cards, listings, search results) into a CSV. No pip install needed:

import csv, sys
from html.parser import HTMLParser

class ItemParser(HTMLParser):
    def __init__(self, item_tag, item_class):
        super().__init__()
        self.item_tag, self.item_class = item_tag, item_class
        self.items, self._in_item, self._text = [], False, ""

    def handle_starttag(self, tag, attrs):
        classes = dict(attrs).get("class", "").split()
        if tag == self.item_tag and self.item_class in classes:
            self._in_item = True
            self._text = ""

    def handle_data(self, data):
        if self._in_item:
            self._text += data

    def handle_endtag(self, tag):
        if self._in_item and tag == self.item_tag:
            self.items.append(self._text.strip())
            self._in_item = False

if __name__ == "__main__":
    html = open(sys.argv[1], encoding="utf-8").read()
    p = ItemParser(item_tag="div", item_class="product-card")
    p.feed(html)
    with open(sys.argv[2], "w", newline="", encoding="utf-8") as f:
        w = csv.writer(f)
        for item in p.items:
            w.writerow([item])

Save the page's HTML, run python3 scrape.py page.html out.csv, and adjust item_tag/item_class to match the tag and CSS class actually wrapping each item on the page you're targeting — view source or your browser's inspector will show you both.

Where a script like this stops being enough

This one grabs a single block of text per item. Real jobs usually need several fields per item at once — name, price, link — pulled from different child tags, written straight to CSV or JSON, on a page that might paginate. And some pages don't hand you the list in the HTML at all: the content is built by JavaScript after the page loads, or it sits behind a login. That needs a headless browser or session handling, and at that point it's not a five-line script anymore, it's a small program worth paying someone to write and hand you.

The finished version, free

The snippet above works on one known shape of page. listscrape.py builds a real DOM instead of matching patterns, takes selectors and attribute fields, guesses the repeating block with --auto, and reads robots.txt the way that actually works — which is not the obvious way.

listscrape.py on GitHub — one file, no dependencies, public domain, tested on Python 3.9 to 3.13. Keep it, change it, ship it in something you sell.

Or skip writing it yourself

Send the target page and the fields you need, and get back a script that pulls all of them — name, price, link, whatever the page has — into CSV or JSON, ready to re-run whenever you need a fresh export. (A page that needs JavaScript rendering or a login is a custom job, scoped separately — see the details page.)

79€ — one-time, fixed, 48h

Buy — Stripe checkout details

← See the other two packages (CSV cleanup, API/webhook bridges)