fix(capture): end special-scheme URL authorities at a backslash (#2767)
* fix(capture): end special-scheme URL authorities at a backslash The native mitmproxy adapter removed transport userinfo with `^(...//)[^/?#]*@`, so the authority ran to the last "@" before "/", "?" or "#". For `https://example.test\part@tail` it treated `example.test\part` as userinfo, reported `https://tail` as the Location, and withdrew the original bytes as a transport credential. WHATWG URL parsing, used by the shared TypeScript owner (recordedTransportUrl.ts) and by HAR captures, ends a special-scheme or scheme-relative authority at a backslash, so that "@" is ordinary path text. Locate the authority with the same boundary rule before removing userinfo. Bytes are handled as Latin-1 text so unrelated bytes stay identical. Userinfo before the backslash is still removed and redacted. Add source unit tests for the adapter projection to test:python. Fixes #2671 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(capture): skip special-scheme leading slashes before userinfo Ending the authority at a backslash exposed a credential: for `https://\user:pass@host/path`, WHATWG parsing skips every leading "/" or "\" after a special scheme (and in scheme-relative references), so `user:pass` is userinfo, but the new boundary stopped at the first backslash and found no "@". The same applied to `https:\\user:pass@host` and `https:user:pass@host`. Start the authority after that whole slash run for non-file special schemes and scheme-relative references. File URLs keep their "//" boundary because they carry no userinfo. Load the adapter in tests with fresh mitmproxy stand-ins scoped to its import instead of leaving them in sys.modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(capture): parse URL controls and mixed network authorities * fix(capture): distinguish native paths and URL base authorities * fix(capture): preserve file-relative path evidence after separators * fix(comparison): recheck historical calls-only changes (#2223) * fix(comparison): preserve incomplete function call evidence Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty. Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls. * fix(comparison): recheck historical calls-only changes --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> * fix(android): make DEX descriptor pattern portable * fix(ci): isolate Linux Hopper process fixture inventory * fix(capture): remove empty URL userinfo markers * test(capture): cover empty URL userinfo markers * test(ci): isolate Hopper process inventory fixtures * fix(ci): declare process inventory callback * test(ci): isolate alternate launcher process fixture * fix(capture): keep URLs that WHATWG rejects instead of redacting userinfo File URLs cannot carry userinfo (an "@" in a file host fails the parse), and a special authority whose host or port WHATWG rejects fails the whole URL. Neither has credentials, so the bridge now keeps their source text and byte sidecar. Checked against Node's URL over 30,000 spellings: no under-redaction, over-redaction 20,540 -> 5,865 (remaining: empty userinfo, IPv4 range, IDNA, non-special and baseless references). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(capture): strip trailing URL whitespace and validate IP hosts and ports The previous host check counted trailing C0 controls and spaces, which WHATWG strips, so `https://user:pw@host ` kept its password. Trailing C0/space is now removed from the parse view (raw offsets are unchanged). Special hosts now also follow the WHATWG IPv4 parser (ends-in-a-number, hex/octal parts, range) and IPv6 literal syntax (zone IDs rejected), and ports are bounded lexically before int(), which raised ValueError past 4300 digits on Python 3.11+ and failed the whole capture. Node URL differential over 126,000 spellings: under-redaction 8,084 -> 0, ValueError 2,550 -> 0; remaining special-scheme over-redaction is IDNA and the pinned empty-userinfo case. Cost: +0.6 us per URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(capture): redact userinfo before hosts that need Unicode mapping A non-ASCII special host may map under UTS #46 to an accepted host (fullwidth digits, ideographic full stops, ignored code points), so treat it as accepted instead of running the ASCII IPv4 check on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: morluto <76467478+morluto@users.noreply.github.com> Co-authored-by: Lucien <164857476+dakjdakd@users.noreply.github.com>
This commit is contained in:
2 files changed
+180
-7
No files matched your search
@@ -6,11 +6,13 @@ The containing mitmdump process and its configuration directory are REA-owned.
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import ipaddress
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import resource
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote_to_bytes
|
||||
|
||||
from mitmproxy import ctx, version
|
||||
from mitmproxy.io import tnetstring
|
||||
@@ -46,6 +48,8 @@ SCHEME_PREFIX = r"^[\x00-\x20]*([a-z][a-z0-9+.-]*):"
|
||||
NETWORK_PREFIX = r"^[\x00-\x20]*[/\\]{2}"
|
||||
URL_IGNORABLE_CONTROLS = "\t\n\r"
|
||||
URL_CONTROL_TRANSLATION = {ord(char): None for char in URL_IGNORABLE_CONTROLS}
|
||||
C0_CONTROL_OR_SPACE = "".join(chr(code) for code in range(0x21))
|
||||
FORBIDDEN_DOMAIN_CODE_POINTS = set(C0_CONTROL_OR_SPACE) | set("#%/:<>?@[\\]^|\x7f")
|
||||
|
||||
|
||||
def authority_start(text, base_scheme=None):
|
||||
@@ -54,11 +58,14 @@ def authority_start(text, base_scheme=None):
|
||||
Follows WHATWG URL parsing as src/domain/recordedTransportUrl.ts does:
|
||||
special schemes (and scheme-relative references) skip every leading "/"
|
||||
or "\\" before the authority, so `https://\\user:pass@host` still has
|
||||
userinfo. File URLs carry no userinfo and keep their "//" boundary.
|
||||
userinfo. File URLs never carry userinfo: an "@" in a file host makes
|
||||
WHATWG reject the whole URL, so there are no credentials to remove.
|
||||
"""
|
||||
scheme = re.match(SCHEME_PREFIX, text, flags=re.I)
|
||||
name = scheme.group(1).lower() if scheme is not None else None
|
||||
if name is not None and name in SPECIAL_SCHEMES and name != "file":
|
||||
if name == "file" or (name is None and base_scheme == "file"):
|
||||
return None
|
||||
if name is not None and name in SPECIAL_SCHEMES:
|
||||
# A same-scheme Location without two slash-equivalent separators is
|
||||
# relative to the recorded request, so its @ belongs to the path.
|
||||
following = text[scheme.end():scheme.end() + 2]
|
||||
@@ -69,7 +76,7 @@ def authority_start(text, base_scheme=None):
|
||||
if scheme is not None:
|
||||
if text[scheme.end():scheme.end() + 2] != "//":
|
||||
return None
|
||||
return scheme.end() + 2, name in SPECIAL_SCHEMES
|
||||
return scheme.end() + 2, False
|
||||
# A non-special base treats backslashes as authority data and only
|
||||
# recognizes two forward slashes as a network reference.
|
||||
if base_scheme is not None and base_scheme not in SPECIAL_SCHEMES:
|
||||
@@ -79,13 +86,87 @@ def authority_start(text, base_scheme=None):
|
||||
if prefix is None:
|
||||
return None
|
||||
start = prefix.end()
|
||||
if base_scheme == "file":
|
||||
return start, True
|
||||
while start < len(text) and text[start] in "/\\":
|
||||
start += 1
|
||||
return start, True
|
||||
|
||||
|
||||
IPV4_DIGITS = {8: "01234567", 10: "0123456789", 16: "0123456789abcdefABCDEF"}
|
||||
|
||||
|
||||
def ipv4_number(part):
|
||||
"""WHATWG IPv4 number parser: decimal, 0x hex or 0-prefixed octal, else None."""
|
||||
if part == "":
|
||||
return None
|
||||
radix, digits = 10, part
|
||||
if part[:2] in ("0x", "0X"):
|
||||
radix, digits = 16, part[2:]
|
||||
elif len(part) > 1 and part[0] == "0":
|
||||
radix, digits = 8, part[1:]
|
||||
if digits == "":
|
||||
return 0
|
||||
if any(char not in IPV4_DIGITS[radix] for char in digits):
|
||||
return None
|
||||
# Bound decimal text before int(): Python limits decimal conversions to
|
||||
# 4300 digits, and anything past 10 significant digits exceeds IPv4.
|
||||
significant = digits.lstrip("0")
|
||||
if radix == 10 and len(significant) > 10:
|
||||
return 2**32
|
||||
return int(digits, radix)
|
||||
|
||||
|
||||
def domain_parses(domain):
|
||||
"""Apply WHATWG's ends-in-a-number check and IPv4 parser to a domain."""
|
||||
parts = domain.split(".")
|
||||
if parts[-1] == "" and len(parts) > 1:
|
||||
parts.pop()
|
||||
last = parts[-1]
|
||||
if not ((last != "" and last.isascii() and last.isdigit()) or ipv4_number(last) is not None):
|
||||
return True
|
||||
numbers = [ipv4_number(part) for part in parts]
|
||||
if len(parts) > 4 or any(number is None for number in numbers):
|
||||
return False
|
||||
return all(number <= 255 for number in numbers[:-1]) and numbers[-1] < 256 ** (5 - len(numbers))
|
||||
|
||||
|
||||
def special_host_parses(host_port):
|
||||
"""Whether WHATWG could accept a special authority's host and port.
|
||||
|
||||
A rejected host fails the whole URL, so it has no credentials to remove.
|
||||
This mirrors the forbidden-code-point, empty-host, IPv4, IPv6 and port
|
||||
rules for ASCII hosts. UTS #46 mapping is not modeled: a non-ASCII host
|
||||
may map to an accepted one (fullwidth digits, ideographic full stops,
|
||||
ignored code points), so it is treated as accepted and still redacts.
|
||||
"""
|
||||
if host_port.startswith("["):
|
||||
host_end = host_port.find("]") + 1
|
||||
literal = host_port[1 : host_end - 1]
|
||||
if host_end == 0 or "%" in literal:
|
||||
return False
|
||||
try:
|
||||
ipaddress.IPv6Address(literal)
|
||||
except ValueError:
|
||||
return False
|
||||
else:
|
||||
colon = host_port.find(":")
|
||||
host_end = len(host_port) if colon < 0 else colon
|
||||
host = unquote_to_bytes(host_port[:host_end]).decode("utf-8", errors="replace")
|
||||
# Mapping never removes ASCII, so these rejections hold either way.
|
||||
if host == "" or any(char in FORBIDDEN_DOMAIN_CODE_POINTS for char in host):
|
||||
return False
|
||||
if host.isascii() and not domain_parses(host.lower()):
|
||||
return False
|
||||
port = host_port[host_end:]
|
||||
if port == "":
|
||||
return True
|
||||
digits = port[1:]
|
||||
if port[0] != ":" or not (digits == "" or (digits.isascii() and digits.isdigit())):
|
||||
return False
|
||||
# Bound the text first: int() refuses decimal strings over 4300 digits.
|
||||
significant = digits.lstrip("0")
|
||||
return len(significant) <= 5 and int(significant or "0") <= 65535
|
||||
|
||||
|
||||
def without_userinfo(value, base_scheme=None):
|
||||
"""Remove an authority's userinfo using the WHATWG authority boundary.
|
||||
|
||||
@@ -96,7 +177,8 @@ def without_userinfo(value, base_scheme=None):
|
||||
text = value.decode("latin-1") if binary else value
|
||||
# WHATWG removes these controls before interpreting scheme/separators.
|
||||
# Keep that parse view separate from the exact source spelling and bytes.
|
||||
parsed_text = text.translate(URL_CONTROL_TRANSLATION)
|
||||
# It also drops trailing C0 controls and spaces; offsets are unchanged.
|
||||
parsed_text = text.translate(URL_CONTROL_TRANSLATION).rstrip(C0_CONTROL_OR_SPACE)
|
||||
located = authority_start(parsed_text, base_scheme)
|
||||
if located is None:
|
||||
return value
|
||||
@@ -106,7 +188,7 @@ def without_userinfo(value, base_scheme=None):
|
||||
while end < len(parsed_text) and parsed_text[end] not in terminators:
|
||||
end += 1
|
||||
at = parsed_text.rfind("@", start, end)
|
||||
if at < start:
|
||||
if at < start or (special and not special_host_parses(parsed_text[at + 1 : end])):
|
||||
return value
|
||||
parsed_offset = 0
|
||||
raw_start = 0
|
||||
|
||||
@@ -165,6 +165,97 @@ class TransportUserinfoTests(unittest.TestCase):
|
||||
def test_file_urls_keep_path_text(self):
|
||||
self.assertEqual(capture.without_userinfo("file:///a@b"), "file:///a@b")
|
||||
|
||||
def test_file_urls_reject_userinfo_and_keep_their_source(self):
|
||||
# WHATWG rejects an "@" in a file host, so nothing parses as credentials.
|
||||
self.assertEqual(
|
||||
capture.without_userinfo("file://user:pass@host/x"), "file://user:pass@host/x"
|
||||
)
|
||||
for value in [b"\\\\user:pass@host/x", b"/\\user:pass@host/x", b"//user:pass@host/x"]:
|
||||
with self.subTest(value=value):
|
||||
text, sidecar, redactions = location(value, b"file")
|
||||
self.assertEqual(text, value.decode("ascii"))
|
||||
self.assertEqual(sidecar["state"], "retained")
|
||||
self.assertEqual(redactions, [])
|
||||
|
||||
def test_rejected_special_hosts_keep_their_source(self):
|
||||
# A rejected host or port fails the whole URL, so it has no userinfo.
|
||||
for value in [
|
||||
"https:user@%",
|
||||
"https://user@",
|
||||
"https://user:pass@h^st/x",
|
||||
"https://user@ho st/x",
|
||||
"https://user@host:99999/x",
|
||||
"https://user@host:12a",
|
||||
"https://user@[::1",
|
||||
]:
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(capture.without_userinfo(value), value)
|
||||
text, sidecar, redactions = location(b"https://user:pass@h^st/x", b"https")
|
||||
self.assertEqual(text, "https://user:pass@h^st/x")
|
||||
self.assertEqual(sidecar["state"], "retained")
|
||||
self.assertEqual(redactions, [])
|
||||
|
||||
def test_accepted_special_hosts_still_remove_userinfo(self):
|
||||
for value, expected in [
|
||||
("https:user@%41", "https:%41"),
|
||||
("https://user@[::1]:8443/x", "https://[::1]:8443/x"),
|
||||
("https://user@host:/x", "https://host:/x"),
|
||||
("wss://user:pass@host:65535", "wss://host:65535"),
|
||||
("https://user@0x7f.1/", "https://0x7f.1/"),
|
||||
("https://user@1.2.3.4./", "https://1.2.3.4./"),
|
||||
("https://user@4294967295/", "https://4294967295/"),
|
||||
("https://user@1.2.3.08a/", "https://1.2.3.08a/"),
|
||||
("https://user@[::ffff:1.2.3.4]/", "https://[::ffff:1.2.3.4]/"),
|
||||
]:
|
||||
with self.subTest(value=value):
|
||||
self.assertEqual(capture.without_userinfo(value), expected)
|
||||
|
||||
def test_unicode_hosts_that_may_map_to_accepted_hosts_remove_userinfo(self):
|
||||
# UTS #46 maps these to 1.2.3.4; mapping is not modeled, so redact.
|
||||
for host in ["%EF%BC%91.2.3.4", "1。2.3.4", "%C2%AD1.2.3.4", "１.2.3.4"]:
|
||||
with self.subTest(host=host):
|
||||
self.assertEqual(
|
||||
capture.without_userinfo(f"https://user:pw@{host}/"), f"https://{host}/"
|
||||
)
|
||||
text, sidecar, redactions = location(b"https://user:pw@%EF%BC%91.2.3.4/", b"https")
|
||||
self.assertEqual(text, "https://%EF%BC%91.2.3.4/")
|
||||
self.assertEqual(sidecar["state"], "redacted")
|
||||
self.assertEqual(len(redactions), 1)
|
||||
# ASCII rejections still hold for non-ASCII hosts.
|
||||
self.assertEqual(capture.without_userinfo("https://user@１^/"), "https://user@１^/")
|
||||
|
||||
def test_trailing_controls_and_spaces_do_not_hide_userinfo(self):
|
||||
# WHATWG strips trailing C0 controls and spaces before parsing.
|
||||
for suffix in [b" ", b"\x01", b"\x00 \x1f"]:
|
||||
with self.subTest(suffix=suffix):
|
||||
value = b"https://user:pw@example.test" + suffix
|
||||
text, sidecar, redactions = location(value, b"https")
|
||||
self.assertEqual(text, "https://example.test" + suffix.decode("latin-1"))
|
||||
self.assertEqual(sidecar["state"], "redacted")
|
||||
self.assertEqual(len(redactions), 1)
|
||||
|
||||
def test_rejected_ip_literals_keep_their_source(self):
|
||||
for value in [
|
||||
"https://user:pass@256.256.256.256/path",
|
||||
"https://user@1.2.3.4.5/",
|
||||
"https://user@1.2.3.09/",
|
||||
"https://user@4294967296/",
|
||||
"https://user@a.0x/",
|
||||
"https://user@1..2/",
|
||||
"https://user@1." + "9" * 5000 + "/",
|
||||
"https://user@[1::2::3]/",
|
||||
"https://user@[::1%25eth0]/",
|
||||
]:
|
||||
with self.subTest(value=value[:40]):
|
||||
self.assertEqual(capture.without_userinfo(value), value)
|
||||
|
||||
def test_overlong_ports_are_compared_without_numeric_overflow(self):
|
||||
# Python 3.11+ refuses int() on decimal strings over 4300 digits.
|
||||
accepted = "https://user@host:" + "0" * 5000 + "80/x"
|
||||
self.assertEqual(capture.without_userinfo(accepted), "https://host:" + "0" * 5000 + "80/x")
|
||||
rejected = "https://user@host:" + "9" * 5000 + "/x"
|
||||
self.assertEqual(capture.without_userinfo(rejected), rejected)
|
||||
|
||||
def test_non_special_schemes_keep_backslashes_in_the_authority(self):
|
||||
self.assertEqual(capture.without_userinfo("foo://a\\b@host/x"), "foo://host/x")
|
||||
self.assertEqual(capture.without_userinfo(b"//a\\b@host"), b"//a\\b@host")
|
||||
|
||||
Reference in new issue
Block a user