Normalizing unencoded utf8

[markov2]

For instance on [https://www.meadjohnson.com.cn/](some Chinese page) I find horrible html like

<ul class="search-hot"><li><a data-id="dhl_yqhl" href="/tag/孕期护理">孕期护理</a></li>
<li><a data-id="dhl_yqzz" href="/tag/孕期症状">孕期症状</a></li>
<li><a data-id="dhl_yqyy" href="/tag/孕期营养">孕期营养</a></li>
<li><a data-id="dhl_dnfy" href="/tag/大脑发育">大脑发育</a></li>
<li><a data-id="dhl_cj" href="/tag/产检">产检</a></li>
<li><a data-id="dhl_fstj" href="/tag/辅食添加">辅食添加</a></li>
</ul>

Module URI normalizes them into hex encodings. Should they? (open question)

[sysread]

Should is loaded. URI::Fast does not because it's illegal. URI::Fast::IRI is fine with it because it allows unicode, but it won't then encode them for you.

You could probably hack together something using URI::Fast::encode, but I'm hesitant to make the parser that flexible, since the spec is strict about only accepting ASCII.

[markov2]

I agree with your remark about the standard. However, these links are relative urls found in HTML. We also remove blanks around them, and translate backslash in forward slash. Do we translate enclosed blanks into a +?

I think that my opinion is that normalize() should fix it. Maybe the html cleanup, before that. Maybe ignore it ;-)

[sysread]

URI::Fast is not currently translating blanks into + (or, more correctly, %20), but I'm leaning toward not starting down the slope of trying to fix broken URLs. There are so many ways URLs can be broken and certainly there are conflicting resolutions.

As an example, should a unicode hostname be translated into percent-encoding or punycode? Either could be argued to be the de facto Right Way.

[markov2]

I did some counting on a small set of HTML pages (mainly from links found in sitemaps), which may interest you:

# A count on a random dataset of 41_912 HTML files, resulted in this
#  9_525_806  100.0%  links
#     29_140    0.3%  blank links
#      2_746    0.0%  empty query (trailing '?')
#    165_393    1.7%  empty fragment (trailing '#')
#     56_843    0.6%  contains blank(s)
#    151_208    1.6%  contains '+'
#    114_258    1.2%  contains utf8 (1_347_928 characters)
#      1_326    0.0%  ipv4 hosts
#      6_753    0.1%  use of default port number
# Schemes
#  7_613_385   80.0%  https:
#  1_695_666   17.8%  http:
#     60_295    0.6%  data:
#        550    0.0%  about:
#    112_975    1.2%  javascript:
#          4    0.0%  ftp:
#     18_771    0.2%  mailto:
#     24_160    0.3%  other schemes (not always, but mostly broken

I have benchmarked various options to tackle the utf8's myself. They all start with the same:

my $b;      # we cannot utf8::encode constants, $1 is a constant
$url =~ s!([^\x20-\xf0])!$b = $1; utf8::encode($b); PROCESS_B!ge

where PROCESS_B converts the separate bytes of a single character. Various attempts:

$b                                             # do nothing, as base timing
unpack('H*', $b) =~ s/(..)/%\U$1/gr
join '', map sprintf("%%%02X", ord), split //, $b!ge
join '%', '', map pack('H2', $_), split //, $b!ge
join '', map $to_hex{$_}, split //, $b!ge      # %to_hex is table 0 -> '%00', etc

Running on 10 million "links" ($url = '2021年07月12日' x 5), these all ran between 15.7 and 16.1 seconds (computers are fast). Even the first "no-op" ! Getting the utf8 character from the string is the only time-consuming component here.

When is see the name html_url, I expect you feed it with something you found in an HTML page, to have it translated into a standard compliant URL. When browsers support the quirks, than html_url should accept it. So: the input is a mess, but the output is clean (or undef).

I will need all weirdness removed, so will create a wrapper around the current html_url() as long as need. You have already put so much effort in this!

[markov2]

As an example, should a unicode hostname be translated into percent-encoding or punycode? Either could be argued to be the de facto Right Way.

There is no unicode in hostnames. Hostnames are restricted to [a-z0-9] and weird rules with dashes. The may be displayed as containing characters outside this range applyingTLD specific rules, via IDN (Net::IDN::Encode). When percent-encoding exists in a URL, it is only superfluous encoding of the few permitted characters: should always be removed (in normalization)

Question is: do browsers accept utf8 in hostnames found in HTML? That would make encoding to IDN needed, either in my html_url-wrapper or your html_url. Probably they do. I do not have that answer (yet).

[sysread]

I expect the popular browsers would make a best guess at the right IDN encoding for a uri specified in utf8. But that would take great care and a lot of security review; since they could result in more than one valid uri, it would be an easy way to hijack links. I do not believe I have the expertise nor time to add those kinds of heuristics to URI::Fast :)

It's worth noting, however, that Net::IDN::Punycode is implemented in XS, so it should be reasonably performant.

[markov2]

Why do you think the browsers have to guess? When the hostname contains %-encoding, then translate it into utf8 as you do for a path. If the result really contains any character which is not in /[a-z0-9.]/, you need to call the IDN library which knows the right IDN based on TLD. I do not see any guessing. (Only for URLs which were found in html)

I think Net::IDN2
(thanks you again for your work)

[sysread]

I'm not seeing a Net::IDN2 on cpan. Do you mean https://metacpan.org/pod/Net::IDN::Encode?

[markov2]

Net::LibIDN2 https://metacpan.org/pod/Net::LibIDN2