read_file(URL): PDF path fetches the same URL twice and loses first-request controls #786

Summary

On current main, read_file for a PDF URL performs two independent HTTP fetches of the same URL.

The first request happens in readFileFromUrl():

const response = await fetch(url, {
    signal: controller.signal
});

const contentType = response.headers.get('content-type') || 'text/plain';
const isPdf = isPdfFile(contentType) || url.toLowerCase().endsWith('.pdf');

if (isPdf) {
    const pdfResult = await parsePdfToMarkdown(url);
    ...
}

Then parsePdfToMarkdown(url) reaches loadPdfToBuffer(), which performs a second request:

async function loadPdfToBuffer(source: string): Promise<Buffer | ArrayBuffer> {
    if (isUrl(source)) {
        const response = await fetch(source);
        return await response.arrayBuffer();
    }
    ...
}

I checked the current default branch before filing; this behavior is still present.

Why this matters

The second fetch is a separate network transaction and does not inherit the first request's controls.

In particular:

• the first request's AbortController / URL-fetch timeout does not govern the second fetch;
• the first response status check is not applied to the second response;
• any request-specific headers, dispatcher/proxy, redirect policy, or other per-request networking behavior added to readFileFromUrl() would be lost on the actual PDF download;
• signed / one-shot / stateful URLs may produce different content or fail on the second request;
• the URL consumes bandwidth twice;
• the body of the first successful PDF response is never used.

This is independent of any particular proxy configuration.

Suggested direction

The smallest clean fix seems to be to consume the already-fetched PDF response body once and pass that buffer into the PDF parsing layer, instead of passing the URL back into a helper that downloads it again.

For example, the PDF parser could expose a buffer-oriented entry point, while keeping the existing URL/file convenience API for callers that need it.

That would preserve the request semantics established by readFileFromUrl() and make the network operation single-shot.

Relevant current-main paths

• src/tools/filesystem.ts — readFileFromUrl()
• src/tools/pdf/markdown.ts — loadPdfToBuffer() / parsePdfToMarkdown()