Created attachment 1200 [details]
Stripped down regex parser that shows the bug

Don I belive I've found something that of interest.

In short - I've got cut down sample, there is no _ctfe branching here, ~500 lines and most of imports are there to print stuff.

Up to last step in bytecode generation all is OK and the result is the same, then at the last step somehow even simple fill with 1 to n loop fails.
(see below)

See lines marked as CRITICAL POINT to see this bisection of the execution.

The output is:

RT version
0       Char   (0x0)
1       Char ☺ (0x1)
2       Char ☻ (0x2)
3       Char ♥ (0x3)
4       Char ♦ (0x4)
5       cast(IR)0
6       End

CT version
0       Char   (0x0)
1       Char ♥ (0x3)
2       Char ☻ (0x2)
3       Char ♥ (0x3)
4       Char ♦ (0x4)
5       cast(IR)0
6       End

Instead it should be the same in both versions.

(In reply to comment #1)
> Created an attachment (id=1200) [details]
> Stripped down regex parser that shows the bug
> 

I think I've pinpointed the issue in an even smaller test case (~90 LOCs).
It directly relates to dealing with arrays of structs at CTFE.

Interesting point is that the assertion in main passes in the code below if you switch 'Bytecode' struct that is nothing more then one int to simply int. Hence my thought about structs being the trigger.


struct Bytecode
{
    int raw;
}

struct Parser
{
    dchar _current;
    bool empty;
    string pat;      
    Bytecode[] ir;     
 
    this(string pattern)
    {
        pat = pattern;
        next();
        uint fix;//fixup pointer
        for(;;)
        {
            switch(current)
            {
            case '(':
                next();
                fix = cast(uint)ir.length;
                assert(current == '?');
                next();
                assert(current == ':');                    
                ir ~= Bytecode(-1);
                next();                    
                break;
            case ')': //CRITICAL POINT: the last closing paren
                //return; // up to this point generated bytecode is the same
                next();
                //return; //still OK
                { //CRITICAL POINT                    
                    size_t cnt = ir.length-fix-1;
                    //even simple write loop is failing with awful results
                    for(size_t i = 0; i < cnt; i++)
                    {
                        auto bc = Bytecode(i+10);
                        ir[fix+i] = bc;
                    }
                }
                return; // and here it differs            
            default:
                uint start = cast(uint)ir.length;
                ir ~= Bytecode(10*current);
                next();
                uint len = cast(uint)ir.length - start;
                next();
                ir ~= Bytecode(-4);
                ir ~= ir[start .. start+len];
                ir ~= Bytecode(-1);
            }
        }
    }

    @property dchar current(){ return _current; }

    bool next()
    {
        if(pat.length == 0)
        {
            empty = true;
            return false;
        }
        _current = pat[0];
        pat = pat[1..$];
        return true;
    }
}

public auto getIr(string pattern)
{
    auto ir = Parser(pattern).ir;    
    return ir;
}

void main()
{
    auto re = getIr("(?:a+)");
    static re2 = getIr("(?:a+)");  
//uncomment to see that it's a 3rd element of 2 arrays that differs
/*
    import std.stdio;
    writeln("RT version");
    writeln(re);
    writeln("\n\nCT version");
    writeln(re2);
*/
    assert(re == re2);
}

(In reply to comment #2)
> //uncomment to see that it's a 3rd element of 2 arrays that differs
> /*
>     import std.stdio;
>     writeln("RT version");
>     writeln(re);
>     writeln("\n\nCT version");
>     writeln(re2);
> */

For me, it's the second element that differs (DMD32 D Compiler v2.063-devel-0630526):
---
RT version
[Bytecode(10), Bytecode(11), Bytecode(12), Bytecode(13), Bytecode(-1)]


CT version
[Bytecode(10), Bytecode(13), Bytecode(12), Bytecode(13), Bytecode(-1)]
---

(In reply to comment #3)
> (In reply to comment #2)
> > //uncomment to see that it's a 3rd element of 2 arrays that differs
> > /*
> >     import std.stdio;
> >     writeln("RT version");
> >     writeln(re);
> >     writeln("\n\nCT version");
> >     writeln(re2);
> > */
> 
> For me, it's the second element that differs (DMD32 D Compiler
> v2.063-devel-0630526):
> ---
> RT version
> [Bytecode(10), Bytecode(11), Bytecode(12), Bytecode(13), Bytecode(-1)]
> 
> 
> CT version
> [Bytecode(10), Bytecode(13), Bytecode(12), Bytecode(13), Bytecode(-1)]
> ---

You're right that must a mistake on my part.
It's 2nd one that is different as I'm seeing this exact output.

Reduced the code from comment #2 further:

struct Bytecode
{
    int raw;
}

Bytecode[] getIr()
{
    Bytecode[] ir;
    
    ir ~= Bytecode(42);
    ir ~= ir[0 .. 1]; // add .dup and CTFE gets it
    assert(&ir[0] != &ir[1]); // fails in CTFE
    
    ir[0].raw = 13; // overwrites both ir[0] and ir[1]
    assert(ir[0].raw == 13);
    assert(ir[1].raw == 42); // fails in CTFE
    
    return ir;
}

void main()
{
    enum expected = [Bytecode(13), Bytecode(42)];
    assert(getIr() == expected); // passes
    static assert(getIr() == expected); // fails
}

(In reply to comment #5)
> Reduced the code from comment #2 further:
> 
> struct Bytecode
> {
>     int raw;
> }
> 
> Bytecode[] getIr()
> {
>     Bytecode[] ir;
> 
>     ir ~= Bytecode(42);
>     ir ~= ir[0 .. 1]; // add .dup and CTFE gets it

A-ha! That indicates that the bug is in appending slices of that same array...

That might get us workaround (without dup) and indeed if I change the above line to this:
    foreach(v; ir[0 .. 1])
        ir ~= v; 

it passes even for bigger slices.

>     assert(&ir[0] != &ir[1]); // fails in CTFE
> 
>     ir[0].raw = 13; // overwrites both ir[0] and ir[1]
>     assert(ir[0].raw == 13);
>     assert(ir[1].raw == 42); // fails in CTFE
> 
>     return ir;
> }
> 
> void main()
> {
>     enum expected = [Bytecode(13), Bytecode(42)];
>     assert(getIr() == expected); // passes
>     static assert(getIr() == expected); // fails
> }

(In reply to comment #5)
> Reduced the code from comment #2 further:
> 
> struct Bytecode
> {
>     int raw;
> }
> 
> Bytecode[] getIr()
> {
>     Bytecode[] ir;
> 
>     ir ~= Bytecode(42);
>     ir ~= ir[0 .. 1]; // add .dup and CTFE gets it
>     assert(&ir[0] != &ir[1]); // fails in CTFE
> 
>     ir[0].raw = 13; // overwrites both ir[0] and ir[1]
>     assert(ir[0].raw == 13);
>     assert(ir[1].raw == 42); // fails in CTFE
> 
>     return ir;
> }
> 
> void main()
> {
>     enum expected = [Bytecode(13), Bytecode(42)];
>     assert(getIr() == expected); // passes
>     static assert(getIr() == expected); // fails
> }

https://github.com/D-Programming-Language/phobos/pull/1290

*** Issue 7440 has been marked as a duplicate of this issue. ***

This is related to bug 6052.

This necessitates a workaround in Phobos.
https://github.com/D-Programming-Language/phobos/pull/1290

Commits pushed to master at https://github.com/D-Programming-Language/phobos

https://github.com/D-Programming-Language/phobos/commit/ed349e1a519e2245c9adc0789deb09350d8511c0
workaround issue 9634

https://github.com/D-Programming-Language/phobos/commit/db7e325f6d4a26294976b8fb9ebc69124d5056bc
Merge pull request #1290 from blackwhale/regex-9634

Workaround issue 9634

Reduced test case shows it doesn't even require ~=.
---------
struct Bug9634 {
    int raw;
}

bool bug9634()
{
    Bug9634[] jr = [Bug9634(42)];
    Bug9634[] ir = jr ~ jr;
    assert(&ir[0] != &ir[1]);
    return true;
}

static assert(bug9634());

Commit pushed to master at https://github.com/D-Programming-Language/dmd

https://github.com/D-Programming-Language/dmd/commit/baf2d607b6b723b498753b730b62b9b258a5659e
Fix bug 9634 [CTFE] wrong code concatenating arrays of structs

When concatenating, the array literals need to be duplicated, if
they are CTFE values.

(hopefully)

Commits pushed to master at https://github.com/D-Programming-Language/phobos

https://github.com/D-Programming-Language/phobos/commit/3e22b63a8b06c6687dc5ea29e8076b2c6ac802b1
CTFE bug 9634 was fixed, remove the workaround

https://github.com/D-Programming-Language/phobos/commit/a419cd25ffa0f7df43005429a9d659e98d5bf1ed
Merge pull request #1732 from blackwhale/revert-wa-regex

[TRIVIAL] CTFE bug 9634 was fixed, remove the workaround