[mnriem]

Description

Harden initial URL validation in build_request() and open_url() by applying the existing HTTPS/loopback transport policy before resolving credentials or constructing requests. HTTPS remains supported, along with HTTP for unambiguous loopback hosts.

Workflow and step catalog fetchers now bypass stale-cache fallback when initial URL, redirect, or final URL validation is rejected, including during forced refresh. These failures retain their URL-policy classification so online bundle catalog fetching and reference validation report definitive errors rather than falling back to packaged snapshots or leaving references unchecked. This intentional hardening prevents expired metadata from masking a rejected URL and makes the outcome independent of which validator rejects it first. Ordinary fetch failures retain their existing availability and stale-cache fallbacks.

Validate raw Location and legacy URI headers before urllib processes redirects. Malformed absolute targets are rejected instead of being normalized onto the current host or surfacing as unclassified parsing failures. This keeps malformed-target handling consistent with other URL-policy rejections while preserving valid relative and scheme-relative redirects and existing authentication-header handling.

Add coverage for accepted and rejected URLs across authentication providers, expired-cache behavior with and without forced refresh, and actual urllib redirect-response processing for every supported redirect status. Document the transport policy and the rationale for the catalog fallback changes in design/catalogs.md.

Testing

Set up this checkout's .venv and editable install using the local development guide. Run commands from the checkout root.

• Tested locally with .venv/bin/specify --help (Windows PowerShell: .\.venv\Scripts\specify.exe --help)
• Ran relevant existing tests with .venv/bin/python -m pytest tests -q (Windows PowerShell: .\.venv\Scripts\python.exe -m pytest tests -q); use affected test paths for focused checks and LC_ALL=en_US.UTF-8 on macOS
• Tested with a sample project (if applicable)

Commands and results:

• .venv/bin/specify --help — passed.
• .venv/bin/python -m pytest tests/specify_cli/authentication -q --tb=short — 389 passed.
• .venv/bin/python -m pytest tests/test_download_security.py tests/test_extensions.py tests/specify_cli/bundles tests/specify_cli/extensions tests/specify_cli/presets tests/specify_cli/integrations/test_catalog.py tests/specify_cli/integrations/test_command_list.py tests/specify_cli/workflows -q --tb=short — 4,013 passed, 29 skipped.
• LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests -q — full suite completed: 15,042 passed, 134 skipped, 1 failed. The failure was tests/specify_cli/test_version_execution.py::TestWindowsUvToolEnvironment::test_non_windows_uv_tool_dry_run_still_previews_installer: terminal wrapping split the long interpreter path in the dry-run output assertion. The version implementation and this test are unchanged by this PR.
• COLUMNS=240 LC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests/specify_cli/test_version_execution.py::TestWindowsUvToolEnvironment::test_non_windows_uv_tool_dry_run_still_previews_installer -q --tb=short — 1 passed. Widening the terminal resolves the isolated output-wrapping assertion without source changes; this isolated rerun is not a full-suite pass.

Sample-project testing is not applicable to this HTTP-helper change.

AI Disclosure

• I did not use AI assistance for this contribution
• I did use AI assistance (fill in the disclosure below)

AI disclosure: GitHub Copilot using GPT-6.1 Sol generated the implementation, tests, documentation, commit message, and PR description and ran the automated checks. Work was performed in autonomous agent mode at the contributor's direction; reasoning effort used the runtime configuration and was not exposed.

[Copilot]

🟡 Changes recommended

Initial URL-policy rejections are misclassified as transient outages, causing bundle reference validation to warn and leave references unchecked rather than definitively reject them.

1 open issue

• Classify initial URL-policy rejections as definitive failures · New

What changed in this PR

Applies the existing HTTPS/loopback transport policy to initial requests in Spec Kit’s authenticated HTTP helpers, before credentials are resolved or requests are constructed.

Changes:

• Reject unsafe or malformed initial URLs in build_request() and open_url().
• Add acceptance and rejection tests across authentication providers and unauthenticated requests.
• Document the initial-request policy.

Validation was limited to source inspection and a focused standard-library check; project tests and before/after regression runs were not performed.

File	Description
tests/​specify_cli/​authentication/​test_authentication.py	Tests accepted URLs and early rejection before credential resolution.
src/​specify_cli/​authentication/​http.py	Validates initial URLs in both shared HTTP helpers.
docs/​reference/​authentication.md	Documents HTTPS requirements and the loopback HTTP exception.

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

The “catalog validation” wording overstates the implemented behavior because ordinary catalog lookups can still skip failed sources or use stale caches.

2 open issues

• Classify initial URL-policy rejections as definitive failures
• Clarify online bundle catalog and reference validation scope · New

_{Review effort: Max (auto)}

[Copilot]

🔵 Needs a closer look

At least one reviewer session failed, leaving the review evidence incomplete.

1 open issue

• Classify initial URL-policy rejections as definitive failures

1 resolved since last review

• Clarify online bundle catalog and reference validation scope

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

Workflow and step catalog fetchers suppress URL-policy errors and return stale metadata even with force_refresh=True, allowing bundle validation to accept cached data from rejected URLs.

1 open issue

• Reject URL policy failures instead of returning stale cache · New

1 resolved since last review

• Classify initial URL-policy rejections as definitive failures

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

Expired-cache fallback differs between redirect validators in both catalog workflows, making rejected-redirect behavior depend on which validator runs first.

2 open issues

• Clarify fallback behavior for rejected redirects · New
• Reject URL policy failures instead of returning stale cache

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

The change alters stale-cache fallback behavior for redirect and final-URL rejections without explicit justification, while mocked redirect tests miss malformed targets that can still return expired cache.

2 open issues

• Malformed redirect targets still fall back to expired cache · New
• Reject URL policy failures instead of returning stale cache

1 resolved since last review

• Clarify fallback behavior for rejected redirects

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

The PR description omits the stale-cache fallback bypass for URL-policy errors and stricter raw redirect-header validation, leaving both behavior changes and their hardening rationale unexplained.

3 open issues

• Malformed redirect targets still fall back to expired cache
• Reject URL policy failures instead of returning stale cache
• Document catalog fallback and redirect validation behavior changes · New

_{Review effort: Max (auto)}

[Copilot]

🔵 Needs a closer look

Security-sensitive shared HTTP/redirect changes and the catalog stale-cache policy change require maintainer sign-off, and the full test suite has not been run.

2 open issues

• Malformed redirect targets still fall back to expired cache
• Reject URL policy failures instead of returning stale cache

1 resolved since last review

• Document catalog fallback and redirect validation behavior changes

_{Review effort: Max (auto)}

[Copilot]

🟡 Changes recommended

An unresolved formatting nit leaves urllib’s <urlopen error …> wrapper in user-facing URL-policy rejection errors in both catalog implementations instead of displaying exc.reason.

1 open issue

• Show the URL-policy rejection reason without urllib’s wrapper · New

2 resolved since last review

• Malformed redirect targets still fall back to expired cache
• Reject URL policy failures instead of returning stale cache

_{Review effort: Max (auto)}

[Copilot]

🔵 Needs a closer look

Maintainer sign-off is needed for the stale-cache fallback policy change requiring separate approval and the security-sensitive redirect changes in the shared authenticated HTTP client.

No open issues 🎉

1 resolved since last review

• Show the URL-policy rejection reason without urllib’s wrapper

_{Review effort: Max (auto)}