D issues are now tracked on GitHub. This Bugzilla instance remains as a read-only archive.
Issue 15440 - std.uni outputs \u0069\u0307 as the lower case of \u0130
Summary: std.uni outputs \u0069\u0307 as the lower case of \u0130
Status: RESOLVED INVALID
Alias: None
Product: D
Classification: Unclassified
Component: phobos (show other issues)
Version: D2
Hardware: All All
: P1 normal
Assignee: No Owner
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-12-13 21:55 UTC by Jack Stouffer
Modified: 2016-01-12 04:27 UTC (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description Jack Stouffer 2015-12-13 21:55:24 UTC
\u0130 is a Turkish character and its lower case version is a normal lower case i, but all of the functions in std.uni which handle making numbers lower cased (e.g. toLower, asCapitalized) output \u0069\u0307 instead of just \u0069.
Comment 1 ag0aep6g 2015-12-14 00:02:15 UTC
Here are three Unicode documents and what they say about the lowercase of U+0130. (search for "LATIN CAPITAL LETTER I WITH DOT ABOVE"):

1) <http://www.unicode.org/charts/PDF/U0100.pdf> says: "lowercase is 0069 i".

2) <http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt> gives U+0069 as the lowercase, too, if I read it right.

3) <http://www.unicode.org/Public/UCD/latest/ucdxml/ucd.nounihan.grouped.zip> gives 'slc="0069" lc="0069 0307"'. I assume "slc" means "simple lowercase", and "lc" means "lowercase".

So it seems that the "simple lowercase" is 'i', but the proper(?) lowercase is "\u0069\u0307".

That makes sense when it's supposed to be reversible without assuming a Turkish context. Uppercasing "\u0069\u0307" you get "\u0049\u0307" ('I' + combining dot) which is equivalent to "\u0130".

Seems to me that std.uni is playing by the book, and that there's a point in what the book says. But I don't know enough about Unicode to speak with certainty.
Comment 2 ag0aep6g 2016-01-09 21:34:28 UTC
Changing the resolution to INVALID. As far as I know, the changelog lists all FIXED issues, and this shouldn't be in the changelog, because we didn't actually change anything.
Comment 3 Ali Cehreli 2016-01-11 20:10:55 UTC
It looks like I am outdated on this issue because I had never heard of the 0069 0307 sequence before H. S. Teoh brought the following change to my attention:

  https://github.com/D-Programming-Language/phobos/pull/3848

I've learned since then that the two-character sequence should be the default but TR locale should still use just 0069. According to the following quote, Java 7 behaves differently depending on locale:

  http://grepalex.com/2013/02/14/java-7-and-the-dotted--and-dotless-i/

<quote>
CODE       LOWER   TITLE   UPPER  LANGUAGE
0130;  0069 0307;   0130;   0130;
0130;  0069;        0130;   0130;       tr;
0130;  0069;        0130;   0130;       az;

Entries with a language take precedence over those without, so in my JVM where the default locale is English, the first row of the mapping is used, which lines-up with the codepoints that we saw outputted in our Java 7 example. Therefore to make Java do the right thing here for Turkish, we need to explicitly specify the Turkish locale (“tr” is the ISO 639 alpha-2 language code for Turkish) to the toLowerCase method
</quote>

Should std.uni be locale-aware?
Comment 4 Jack Stouffer 2016-01-12 04:27:27 UTC
(In reply to Ali Cehreli from comment #3)
> Should std.uni be locale-aware?

Yes, though in what way it would achieve this is an interesting question.

I think you should make a seperate bug report for this.