Repository navigation
mcp-server-fetch: IndexError (list index out of range) on pages that simplify to an empty tree when beautifulsoup4 4.13.x is installed #5158
Description
Summary
mcp-server-fetch 2026.10.10 crashes with "IndexError: list index out of range" on any page whose content simplifies to an empty DOM tree when beautifulsoup4 4.13.x is installed (which an unpinned pip install pulls today). The tool call fails with a bare error; with beautifulsoup4 <= 4.12 the same page returns the intended "Page failed to be simplified from HTML".
Environment
- Windows 11 (build 26300), Python 3.13.14
- mcp-server-fetch 2026.10.10, readabilipy 0.3.0, beautifulsoup4 4.13.4
- started via python -m mcp_server_fetch
Reproduction
- pip install "beautifulsoup4==4.13.4"
- Call the fetch tool on a URL that serves a JavaScript-only SPA shell - for example https://z.ai/blog/glm-5.3-flash, which serves a 604-byte document whose body is only plus a module script (content is rendered client-side).
Traceback
File "...mcp_server_fetch\server.py", line 155, in extract_content_from_html
ret = readabilipy.simple_json.simple_json_from_html_string(html, use_readability=True)
File "...readabilipy\simple_json.py", line 81, in simple_json_from_html_string
"content": str(simple_tree_from_html_string(html))
File "...readabilipy\simple_tree.py", line 70, in simple_tree_from_html_string
root.append(soup)
File "...bs4\element.py", line 2062, in append
return self.insert(len(self.contents), tag)[0]
IndexError: list index out of range
Analysis
beautifulsoup4 4.13 changed Tag.insert() to return the list of elements actually inserted, and Tag.append() now indexes that list (self.insert(len(self.contents), tag)[0]). Appending an empty BeautifulSoup object inserts nothing, so indexing raises IndexError. readabilipy 0.3.0 declares beautifulsoup4>=4.7.1 with no upper bound.
The comment in src/fetch/server.py ("Without Node, readabilipy's pure-Python extractor never returns empty content: an empty page comes back as '
'...") documents the pre-4.13 behaviour that this crash replaces. This is effectively the crash-mode sibling of #4989 (closed).Upstream issue filed on ReadabiliPy: alan-turing-institute/ReadabiliPy#123
Note that the Readability.js (Node) path, which would bypass the pure-Python extractor here, is unreachable on Windows for a separate reason (readabilipy cannot resolve npm.cmd): alan-turing-institute/ReadabiliPy#124
Suggested fix
Pin beautifulsoup4<4.13 in mcp-server-fetch dependencies (workaround verified below), or handle the empty-tree case upstream in readabilipy.
Workaround
pip install "beautifulsoup4==4.12.3" - verified: the reproduction URL then returns "Page failed to be simplified from HTML" instead of crashing, and content-rich pages extract unchanged.
Activity
cliffhall commentedon Oct 19, 2026
Triage: Priority Medium (total 6)
buglabelStill present on v2 (checked by reading the current code).
Board: Servers V2 (#43), Status
Incoming, awaiting maintainer review (no milestone yet). Part of the backlog triage in #4875.fix(fetch): report simplification failure instead of crashing on empt…
charan-rathore commentedon Oct 19, 2026
Reproduced with beautifulsoup4 4.13.4 and readabilipy forced into pure-Python mode (no Node on PATH, matching the Windows report): every input that simplifies to an empty tree - an SPA shell, an empty body, a bare
<div>, an empty string - crashes withIndexError: list index out of rangefromTag.append(bs4 4.13 hasTag.insertreturn the inserted elements, and appending an empty tree inserts none).I have a fix ready on a branch (https://github.com/charan-rathore/servers/tree/patch-1):
extract_content_from_htmlcatches theIndexErrorand returns the same<error>Page failed to be simplified from HTML</error>the existing empty-content paths return, so the tool call degrades the same way it does under bs4 <= 4.12. The regression test pins the pure-Python path with the repo'spython_readabilityfixture, fails without the fix, and passes with it; the fullsrc/fetchsuite passes 146/146, andruff format/ruff checkare clean (only the 3 pre-existing upstream findings).I could not open the PR because this repository currently limits pull requests to collaborators. Happy to open it as soon as that lifts, or a maintainer can pull the branch directly.
Prepared with AI assistance (Instinct assisted).