Two ways urllib.robotparser gets robots.txt wrong

One refuses sites that welcome you. The other changes its answer depending on what order the rules happen to be written in. Both are measured below, on four Python versions.

We hit this building a small scraper. The robots.txt said, in full:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

And the scraper refused to read the site. Politely, deliberately, and completely wrongly.

The symptom

This is the whole reproduction. No scraper required:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://hookforge.dev/robots.txt")
rp.read()

print(rp.can_fetch("my-crawler", "https://hookforge.dev/"))
# False

print(rp.disallow_all, rp.allow_all, len(rp.entries))
# True False 0

read() did not raise. It did not warn. It came back having decided that every URL on the site is off limits, and entries is empty because it never parsed a single line of the file.

Meanwhile the file is right there, and it allows everything:

$ curl -s -o /dev/null -w '%{http_code}\n' \
    -A 'my-crawler/1.0' https://hookforge.dev/robots.txt
200

The cause, from the source

It is eleven lines of CPython, and both halves of the problem are visible in them:

def read(self):
    """Reads the robots.txt URL and feeds it to the parser."""
    try:
        f = urllib.request.urlopen(self.url)
    except urllib.error.HTTPError as err:
        if err.code in (401, 403):
            self.disallow_all = True
        elif err.code >= 400 and err.code < 500:
            self.allow_all = True
        err.close()
    else:
        raw = f.read()
        self.parse(raw.decode("utf-8").splitlines())

First half: urllib.request.urlopen(self.url), with no headers. So robots.txt is requested as Python-urllib/3.13 — not as your crawler, and not as anything a bot-filtering rule was written with in mind.

Second half: a 403 sets disallow_all. That part is correct. RFC 9309 §2.3.1.3 says an "unavailable for legal reasons" or access-denied status on robots.txt means the crawler must assume complete disallow. The parser is doing exactly what the spec asks.

The bug is in the join between them. The 403 was about Python-urllib, not about your crawler — but by the time read() sees a status code, the user agent that earned it is gone, and a refusal aimed at one identity has been applied to another.

How often does it actually happen?

We first wrote "about half the web" and then went and measured, which is a better order to do those two things in. Fetching each site's robots.txt twice — once as urllib's default, once with a normal crawler UA:

Two in six, in a deliberately small and unscientific sample. Not half the web — but not rare either, and the sites it hits are the ones behind the CDNs that answer unknown user agents with a challenge. If your crawler mysteriously refuses one site in three and works fine on the rest, this is a cause worth eliminating early.

Stack Overflow is the interesting row. It answers everyone with 418, and 418 is a 4xx that is not 401 or 403 — so it lands in the allow_all = True branch. Their bot defence makes robotparser conclude that there are no rules at all, which is the opposite failure and arguably the worse one.

The fix

Fetch the file yourself, with the user agent you are actually going to crawl as, and hand the text to the parser. RobotFileParser.parse() takes a list of lines, so nothing else changes:

import urllib.error, urllib.parse, urllib.request, urllib.robotparser

UA = "my-crawler/1.0 (+https://example.com/bot)"

def robots_for(base):
    """A RobotFileParser for `base`, fetched with OUR user agent."""
    rp = urllib.robotparser.RobotFileParser()
    req = urllib.request.Request(base + "/robots.txt",
                                 headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            body = r.read().decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        # Now these statuses mean what the RFC says they mean, because they
        # were earned by the agent that is going to do the crawling.
        rp.disallow_all = e.code in (401, 403)
        rp.allow_all = not rp.disallow_all
        return rp
    except Exception:
        # Unreachable robots.txt is "no rules", not "no crawling" -- refusing
        # here makes the crawler useless against every site without one.
        rp.allow_all = True
        return rp
    rp.parse(body.splitlines())
    return rp

def allowed(url):
    parts = urllib.parse.urlparse(url)
    return robots_for(f"{parts.scheme}://{parts.netloc}").can_fetch(UA, url)

That fixes the fetching. It does not fix the matching — the rp.parse() on the last line is the same parser with the same ordering bug, which is the next section. If you need both, keep this function for getting the text and evaluate the rules yourself.

Three details worth keeping:

• Decode with errors="replace". The stdlib version does a bare .decode("utf-8"), so one stray byte in a robots.txt raises UnicodeDecodeError out of read().
• Keep the 401/403 rule. Do not "fix" this by ignoring those codes. Now that the request carries your real user agent, a 403 is a genuine refusal and should be obeyed.
• A network error is not consent, and not refusal either. Treating an unreachable robots.txt as "no rules" is what well-behaved crawlers do; treating it as "stay out" means one flaky DNS lookup stops a job for a site that never objected.

The second one: order changes the answer

That first bug is loud once you find it — the crawler refuses everything and you go looking. This one is quiet, and it is still there after you fix the first.

Two robots.txt files. Same two rules, opposite order:

>>> import urllib.robotparser as rp
>>>
>>> a = rp.RobotFileParser()
>>> a.parse(["User-agent: *", "Allow: /public/", "Disallow: /"])
>>> a.can_fetch("my-crawler", "https://example.com/public/x")
True
>>>
>>> b = rp.RobotFileParser()
>>> b.parse(["User-agent: *", "Disallow: /", "Allow: /public/"])
>>> b.can_fetch("my-crawler", "https://example.com/public/x")
False

Same site, same permission, opposite answers. RFC 9309 §2.2.2 is explicit that order carries no meaning: the most specific rule wins — the longest matching path — and Allow wins a tie. urllib.robotparser takes the first match instead.

And the second shape is the common one. Block everything, then open these paths is how a great many real robots.txt files are written — which means a first-match parser reads a deliberate, carefully-scoped crawling policy as a total ban. The sites it refuses are precisely the ones that went to the trouble of saying yes.

It is being fixed, and most Pythons do not have the fix

We only found this because a test asserted the bug was still there, and CI went red. Measured on every version we could reach:

So the fix landed in a recent 3.13 patch. If you are on anything older — which is most deployed Python — you still have it, and you cannot tell from the major version.

The fix

Evaluate the rules yourself. It is about thirty lines and it removes the version question entirely:

import re

def rule_matches(path, rule):
    """Does this robots.txt path rule apply? Handles * and $."""
    if rule == "":
        return False                      # empty Disallow means "allow all"
    pattern = "".join(
        ".*" if ch == "*" else ("$" if ch == "$" else re.escape(ch))
        for ch in rule)
    if not rule.endswith("$"):
        pattern += ".*"
    return re.match(pattern, path) is not None

def robots_allows(rules, path):
    """RFC 9309: longest matching rule wins, Allow wins a tie.

    `rules` is [(path_rule, is_allow)] for the group that applies to you.
    """
    best_len, best_allow = -1, True
    for rule, allow in rules:
        if not rule_matches(path, rule):
            continue
        if len(rule) > best_len or (len(rule) == best_len and allow):
            best_len, best_allow = len(rule), allow
    return best_allow

>>> rules = [("/", False), ("/public/", True)]     # order does not matter
>>> robots_allows(rules, "/public/x"), robots_allows(rules, "/other")
(True, False)
>>> robots_allows(list(reversed(rules)), "/public/x")
True

Three details worth keeping: an empty Disallow: is the documented way to say allow everything and must not match anything; $ only anchors when it is the last character; and a group for a named user agent beats the * group entirely rather than adding to it.

The larger point

The first one is nobody's mistake. robotparser follows the RFC on status codes. The CDN is right to challenge an unrecognised agent. urlopen has always sent its own UA. Every piece behaves correctly and the result is a crawler that refuses a site which explicitly allowed it — silently, with no exception and no log line, which is why it survives code review.

The second one is a plain deviation from the spec, and it has the same shape: no error, no warning, just a False that looks exactly like a site saying no. Neither shows up in a test unless you go looking for it, because both fail in the direction that seems responsible.

The bugs that last are rarely inside a function. They are in the gap between two functions that each did exactly what they promised.

← HookForge: fixed-price automation scripts, delivered in 48h