https://github.com/D-Programming-Language/phobos/pull/1017

@bear: Please see the comments here: https://github.com/D-Programming-Language/phobos/pull/1017

The feature can be implemented but to!() was rejected, so we need to come up with some alternative function names and put them somewhere other than std.conv. 

Personally I don't see how people will be expected to find an obscure function name like 'codePointIdx'. This isn't related unicode representation at all, there should be no confusion with Unicode when it comes to representing 0-9, it's always the same regardless of encoding.

(In reply to comment #2)
> @bear: Please see the comments here:
> https://github.com/D-Programming-Language/phobos/pull/1017
> 
> The feature can be implemented but to!() was rejected, so we need to come up
> with some alternative function names and put them somewhere other than
> std.conv. 
> 
> Personally I don't see how people will be expected to find an obscure function
> name like 'codePointIdx'. This isn't related unicode representation at all,
> there should be no confusion with Unicode when it comes to representing 0-9,
> it's always the same regardless of encoding.

Well, that's why we have std.ascii, no? For all char operations when we don't care about unicode.

In all fairness, unicode defines "is numeric" (which we already have) and "numeric value" (which we *should* have).

C# and java both implement the methods "getNumericValue". Java even implements one taking chars, and another taking int (dchar)
http://msdn.microsoft.com/en-us/library/system.char.getnumericvalue.aspx
http://docs.oracle.com/javase/1.4.2/docs/api/java/lang/Character.html

I'd say we should just add:
std.ascii.getNumericValue
std.uni.getNumericValue
(or plain numericValue)

I already wrote the ascii version (easy as pie), and support for the [Nd] group, using a binary search, followed by an offset from the lower bound.

[Nl] and [Po] require a straight up mapping of codepoint to value, but I'm still writing the parser that extract the data for the raw UCD (http://www.unicode.org/Public/6.2.0/ucdxml/).

The file is too large for std.xml to handle, so it's back to C++ for me :/

The only questions I have is:
Return value: int or double?
Input is not numeric: -1 or exception?

(In reply to comment #3)
> Well, that's why we have std.ascii, no? For all char operations when we don't
> care about unicode.
> 
> In all fairness, unicode defines "is numeric" (which we already have) and
> "numeric value" (which we *should* have).

Damn Unicode, why does it need to have 10 different ways to represent something? :)

> The only questions I have is:
> Return value: int or double?

int, because int is implicitly convertible to double, not vice-versa. At least for the ascii part, if Unicode has code points that represent floating-point values.. then I really don't understand what Unicode is about anymore.

> Input is not numeric: -1 or exception?

Hmm.. although exceptions are preferred I think for performance reasons we might consider using -1.

>Java even implements
> one taking chars, and another taking int (dchar)

That's because Java folks used to have only 16bit chars. Now true codepoints are going in form of 'int'.

> http://msdn.microsoft.com/en-us/library/system.char.getnumericvalue.aspx
> http://docs.oracle.com/javase/1.4.2/docs/api/java/lang/Character.html
> 
> I'd say we should just add:
> std.ascii.getNumericValue
> std.uni.getNumericValue
> (or plain numericValue)
> 

Agreed and the name should be numericValue.

> I already wrote the ascii version (easy as pie), and support for the [Nd]
> group, using a binary search, followed by an offset from the lower bound.
> 
> [Nl] and [Po] require a straight up mapping of codepoint to value, but I'm
> still writing the parser that extract the data for the raw UCD
> (http://www.unicode.org/Public/6.2.0/ucdxml/).
> 

I'm wrapping up a revamp of std.uni that makes it piece of cake to create character sets. And maps are converted to multi-staged tables that are faster the binary search on a large set. I'd suggest to wait a bit on it (so as to not duplicate work) and introduce only std.ascii version as the most useful.

The ongoing polishing, fixing and testing against ICU is going on here:
https://github.com/blackwhale/gsoc-bench-2012

> The file is too large for std.xml to handle, so it's back to C++ for me :/
> 
http://www.unicode.org/Public/UNIDATA/UnicodeData.txt

Same thing but no useless XML trash. Description of fields is somewhere in the middle of this document 
http://www.unicode.org/reports/tr44/

> The only questions I have is:
> Return value: int or double?

Should be rational to acurately represent things like "1/5" character ;)
I do suspect some simple custom type could do (2 shorts packed in one struct etc.).

> Input is not numeric: -1 or exception?

-1 is fine I think as this rather low level (per character) and it's not at all convenient to throw (and then catch).

Ok I think there are two enhancements here, one for the simple ascii int->char, char->int, and the other more complicated Unicode implementation which monarch/dmitry know more about.

I think we should split up the Unicode enhancement into a new bugzilla entry since the ASCII one can be implemented right now so this issue can be closed soon.

It would be nice to have a separate issue filed for tracking Unicode support progress. It can maybe include things like issue 9173 too.

(In reply to comment #7)
> It would be nice to have a separate issue filed for tracking Unicode support
> progress. It can maybe include things like issue 9173 too.

Reporters could add "Unicode" into the Keywords box for these types of issues so we can filter them out.

> Ok I think there are two enhancements here, one for the simple ascii int->char,
> char->int, and the other more complicated Unicode implementation which
> monarch/dmitry know more about.
> 
> I think we should split up the Unicode enhancement into a new bugzilla entry
> since the ASCII one can be implemented right now so this issue can be closed
> soon.

I'm a bit too busy to do the actual pull, but I wrote code, doc and test for
this already.

//----
/++
    If $(D c) is an ASCII digit, returns the
    corresponding numeric value. Returns -1 otherwise.
  +/
int numericValue(dchar c) @safe pure nothrow
{
    return ('0' <= c && c <= '9') ? (c - '0') : -1;
}
unittest
{
    int counter = 0;
    foreach (char c; 0 .. 80)
    {
        if (isDigit(c))
            assert(numericValue(c) == counter++);
        else
            assert(numericValue(c) == -1);
    }
}
//----

Not much, but there is never any reason to do the same work twice...

(In reply to comment #5)
> 
> I'm wrapping up a revamp of std.uni that makes it piece of cake to create
> character sets. And maps are converted to multi-staged tables that are faster
> the binary search on a large set. I'd suggest to wait a bit on it (so as to not
> duplicate work) and introduce only std.ascii version as the most useful.
> 
> The ongoing polishing, fixing and testing against ICU is going on here:
> https://github.com/blackwhale/gsoc-bench-2012

OK: The thing I was having trouble though is that existing binary search returns a bool, whereas I need the actual entry, so I can do "value - entry[0]", eg:

//----
    static immutable dchar[2][] table1 = [
    [ 0x0030,  0x0039], //
    [ 0x0660,  0x0669], //ARABIC-INDIC
    [ 0x06F0,  0x06F9], //EXTENDED ARABIC-INDIC

...
//---
That's because all the entries in [Nd] are consecutive numerals starting at 0. I can also cram a select couple of entries from [Nl] and [Po] that also use this scheme.

So if I have the unicode 0x0665 (The ARABIC-INDIC numeral '6'), I'd want to find [ 0x0660,  0x0669], and then "return 0x0665 - 0x0660".

Well, I don't need the entire pair, but at least the lhs of the pair.

If you could keep that in mind during your re-write. Or not. Just throwing it out there.

For all other entries in [Nl] and [Po], I'd have:
    static immutable dchar[2][] table1 = [
    [ 0x261D,  100], //ROMAN NUMERAL ONE HUNDRED

So that's just basic dictionary. But I don't think you can statically allocate an AA. So yeah, just throwing that your direction too.

> > The file is too large for std.xml to handle, so it's back to C++ for me :/
> > 
> http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
> 
> Same thing but no useless XML trash. Description of fields is somewhere in the
> middle of this document 
> http://www.unicode.org/reports/tr44/

Nice, TY.

> > The only questions I have is:
> > Return value: int or double?
> 
> Should be rational to acurately represent things like "1/5" character ;)
> I do suspect some simple custom type could do (2 shorts packed in one struct
> etc.).
> 
> > Input is not numeric: -1 or exception?
> 
> -1 is fine I think as this rather low level (per character) and it's not at all
> convenient to throw (and then catch).

The only issue I have with returning -1 is that it is a magic value. The fact that there is no unicode for -1 is pure coincidence, and not by design. In particular, any attempt to write "if (numericValue(c) < 0) fail" would also be wrong because:
http://unicode.org/cldr/utility/character.jsp?a=0F33
The TIBETAN DIGIT HALF ZERO returns -0.5

Do we *really* want to standardize the syntax of "if (numericValue(c) < -0.7)" ?

...

Damn you unicode!

(In reply to comment #10)
> (In reply to comment #5)
> > 
> > I'm wrapping up a revamp of std.uni that makes it piece of cake to create
> > character sets. And maps are converted to multi-staged tables that are faster
> > the binary search on a large set. I'd suggest to wait a bit on it (so as to not
> > duplicate work) and introduce only std.ascii version as the most useful.
> > 
> > The ongoing polishing, fixing and testing against ICU is going on here:
> > https://github.com/blackwhale/gsoc-bench-2012
> 
> OK: The thing I was having trouble though is that existing binary search
> returns a bool, whereas I need the actual entry, so I can do "value -
> entry[0]", eg:
> 
> //----
>     static immutable dchar[2][] table1 = [
>     [ 0x0030,  0x0039], //
>     [ 0x0660,  0x0669], //ARABIC-INDIC
>     [ 0x06F0,  0x06F9], //EXTENDED ARABIC-INDIC
> 
> ...
> //---
> That's because all the entries in [Nd] are consecutive numerals starting at 0.
> I can also cram a select couple of entries from [Nl] and [Po] that also use
> this scheme.
> 

Sometimes I was able to abuse the natural format of data and sometimes failed. But what proved to be quite good is varying sizes of multi-staged rable to match "periods" of data. In the end if the data has a lot of common "rows" a multi-staged table of certain size per stage is bound hit a sweet spot.

> So if I have the unicode 0x0665 (The ARABIC-INDIC numeral '6'), I'd want to
> find [ 0x0660,  0x0669], and then "return 0x0665 - 0x0660".
> 
> Well, I don't need the entire pair, but at least the lhs of the pair.
> 
> If you could keep that in mind during your re-write. Or not. Just throwing it
> out there.
> 
> For all other entries in [Nl] and [Po], I'd have:
>     static immutable dchar[2][] table1 = [
>     [ 0x261D,  100], //ROMAN NUMERAL ONE HUNDRED
> 
> So that's just basic dictionary. But I don't think you can statically allocate
> an AA. So yeah, just throwing that your direction too.
> 

Well, AA is a fat pig w.r.t RAM usage. But thanks anyway.

> > > The file is too large for std.xml to handle, so it's back to C++ for me :/
> > > 
> > http://www.unicode.org/Public/UNIDATA/UnicodeData.txt
> > 
> > Same thing but no useless XML trash. Description of fields is somewhere in the
> > middle of this document 
> > http://www.unicode.org/reports/tr44/
> 
> Nice, TY.
> 
> > > The only questions I have is:
> > > Return value: int or double?
> > 
> > Should be rational to acurately represent things like "1/5" character ;)
> > I do suspect some simple custom type could do (2 shorts packed in one struct
> > etc.).
> > 
> > > Input is not numeric: -1 or exception?
> > 
> > -1 is fine I think as this rather low level (per character) and it's not at all
> > convenient to throw (and then catch).
> 
> The only issue I have with returning -1 is that it is a magic value. The fact
> that there is no unicode for -1 is pure coincidence, and not by design. In
> particular, any attempt to write "if (numericValue(c) < 0) fail" would also be
> wrong because:
> http://unicode.org/cldr/utility/character.jsp?a=0F33
> The TIBETAN DIGIT HALF ZERO returns -0.5
> 
> Do we *really* want to standardize the syntax of "if (numericValue(c) < -0.7)"
> ?
> 
> ...
> 
> Damn you unicode!

Aye, and given there are things like "1e12" I don't think packing it would work any better... some kind of custom type is required.

(In reply to comment #9)
> int numericValue(dchar c) @safe pure nothrow

What about int->dchar?

We could call it toNumericChar or something, but it would probably have to throw on invalid input? Or can we also return -1? E.g.

char toNumericChar(int i) @safe pure nothrow
{
    return cast(char)((0 <= i && i <= 9) ? (i + '0') : -1);
}

(In reply to comment #12)
> (In reply to comment #9)
> > int numericValue(dchar c) @safe pure nothrow
> 
> What about int->dchar?
> 
> We could call it toNumericChar or something, but it would probably have to
> throw on invalid input? Or can we also return -1? E.g.
> 
> char toNumericChar(int i) @safe pure nothrow
> {
>     return cast(char)((0 <= i && i <= 9) ? (i + '0') : -1);
> }

-1 is char.init, so seems good to me. Although I'd go and write it as "char.init" explicitly in the code actually, so as to limit any possible confusion.

(In reply to comment #11)
> 
> Aye, and given there are things like "1e12" I don't think packing it would work
> any better... some kind of custom type is required.

Really? According to:

http://unicode.org/cldr/utility/properties.jsp?a=Numeric_Value#Numeric_Value

They only go from
-0.5 // TIBETAN DIGIT HALF ZERO
to
1_000_000 // ROMAN NUMERAL ONE HUNDRED THOUSAND

So I figured though we were in the number plane where there is a perfect "int <=> double" correlation. If this is not the case...

Having functions in std.ascii (and elsewhere) seems acceptable. But I think the name of such functions shouldn't be too much long.


to!int raises exceptions. Returning -1 in case of errors seems able to cause some problems. One common use case for the char->int conversion:

auto s = "123x456";
auto digits = s.map!numericValue().array();

Now I have to scan digits again looking for any -1.

(In reply to comment #15)
> Having functions in std.ascii (and elsewhere) seems acceptable. But I think the
> name of such functions shouldn't be too much long.
> 
> 
> to!int raises exceptions. Returning -1 in case of errors seems able to cause
> some problems. One common use case for the char->int conversion:
> 
> auto s = "123x456";
> auto digits = s.map!numericValue().array();
> 
> Now I have to scan digits again looking for any -1.

*But* you can wrap it inside a function which throws on -1 (pseudocode):

auto s = "123x456";
auto thr = (a) => a == -1 ? throw ConvException() : a;
auto digits = s.map!numericValue().array();

Whereas if it threw to begin with you're forced to catch exceptions.

(In reply to comment #14)
> (In reply to comment #11)
> > 
> > Aye, and given there are things like "1e12" I don't think packing it would work
> > any better... some kind of custom type is required.
> 
> Really? According to:
> 
> http://unicode.org/cldr/utility/properties.jsp?a=Numeric_Value#Numeric_Value
> 
> They only go from
> -0.5 // TIBETAN DIGIT HALF ZERO
> to
> 1_000_000 // ROMAN NUMERAL ONE HUNDRED THOUSAND
> 
> So I figured though we were in the number plane where there is a perfect "int
> <=> double" correlation. If this is not the case...

You missed the nice and cool 1.0e12 !

http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3AnumericValue%3D1.0E12%3A%5D&g=

(In reply to comment #16)

> Whereas if it threw to begin with you're forced to catch exceptions.

There is no perfect solution. Exceptions are safer than error codes because if you forget to test for a negative result, your program stops. On the other hand exceptions are less efficient, less handy to use in nothrow functions, and often require some try-catch wrapping.

In this enhancement request I was originally asking for an overload of to!(), this means a solution that throws exceptions when the input is wrong.

Efficiency is not a significant problem for me here because where I need to convert char digits to numerical digits with max efficientcy I use a '0' subtraction (or a vectorized version of it). So with this overload of to!() I was looking for safety.

(In reply to comment #17)
> (In reply to comment #14)
> > (In reply to comment #11)
> > > 
> > > Aye, and given there are things like "1e12" I don't think packing it would work
> > > any better... some kind of custom type is required.
> > 
> > Really? According to:
> > 
> > http://unicode.org/cldr/utility/properties.jsp?a=Numeric_Value#Numeric_Value
> > 
> > They only go from
> > -0.5 // TIBETAN DIGIT HALF ZERO
> > to
> > 1_000_000 // ROMAN NUMERAL ONE HUNDRED THOUSAND
> > 
> > So I figured though we were in the number plane where there is a perfect "int
> > <=> double" correlation. If this is not the case...
> 
> You missed the nice and cool 1.0e12 !
> 
> http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3AnumericValue%3D1.0E12%3A%5D&g=

Well, that still fits in both a long, and in a double with no loss, so we're still good. Crisis averted.

(In reply to comment #18)
> (In reply to comment #16)
> 
> > Whereas if it threw to begin with you're forced to catch exceptions.
> 
> There is no perfect solution. Exceptions are safer than error codes because if
> you forget to test for a negative result, your program stops. On the other hand
> exceptions are less efficient, less handy to use in nothrow functions, and
> often require some try-catch wrapping.
> 
> In this enhancement request I was originally asking for an overload of to!(),
> this means a solution that throws exceptions when the input is wrong.
> 
> Efficiency is not a significant problem for me here because where I need to
> convert char digits to numerical digits with max efficientcy I use a '0'
> subtraction (or a vectorized version of it). So with this overload of to!() I
> was looking for safety.

I think a good solution is to accept having different semantics:

std.ascii.numericValue:
int numericValue(dchar c) safe nothrow pure; returns -1 on failure

std.uni.numericValue:
double numericValue(dchar c) safe pure; Throws on failure

If you are doing anything with unicode, the exception's overhead will be mostly moot compared to the cost (I think). When operating with ASCII, then it's a different ballgame (IMO).

That's my opinion anyways.

THIS ISSUE HAS BEEN MOVED TO GITHUB

https://github.com/dlang/phobos/issues/9583

DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB