I simplified the example a little bit:

import std.traits, std.range;
template ReturnTypeEx(alias A, B) {
	alias ReturnTypeEx = ReturnType!(A!B);
}
void a(S)(auto ref S i) { }
template b(alias R) {
	void b(S)(S i) {
		alias Ra = ReturnTypeEx!(R, S);
	}
}
void main() {
	alias xx = b!(a!string);
}

And one more thing I discovered: If I add a constraint to 'a', say 'if (isInputRange!S)', then dmd reports 'template instance something.a!string does not match template declaration a(S)(auto ref S i) if (isInputRange!S)', while instantiate 'a' directly with 'a!string' works.

Seems this bug is not related to nested templates at all, it's more likely an 'auto ref' bug:

import std.traits, std.range;

void a(S)(auto ref S i) { } 
void b(S)(auto ref S i) if (isInputRange!S) { }
void c(S)(ref S i) if (isInputRange!S) { }

void devil(alias S)() { }

void main() {
    a!string(""); //Works  <--- This line affects the result of devil!(a!string)
    b!string(""); //Works

    //Next line is weird, it:
    //1. Err, 'auto ref can only be used with template function', if 'a' is not
    //   instantiated with 'a!string' first
    //2. Works, if 'a!string' is done first
    alias x = devil!(a!string); 

    alias xx = devil!(b!string); //Err, template doesn't match

    alias xxx = devil!(c!string); //Works
}

(In reply to Yuxuan Shui from comment #2)
> Seems this bug is not related to nested templates at all, it's more likely
> an 'auto ref' bug:

`auto ref` parameter is allowed for template functions, AND needs actual function argument to deduce its ref-ness. For example:

void foo(auto ref int x) {}    // non-template function
// -> NG, auto can only be used for template function parameters

void bar(T)(auto ref T x) {}
void main() {
    int n;

    // IFTI (implicit function template instantiation)
    bar(1);      // OK, parameter x is deduced to non-ref
    bar(n);      // OK, parameter x is deduced to ref

    // partial template specialization + IFTI
    bar!int(1);  // also OK, equivalent with bar(1)
    bar!int(n);  // also OK, equivalent with bar(n)

    // explicit instantiation without IFTI
    alias b = bar!int;
    // NG! auto ref parameter x cannot deduce its ref-ness, therefore
    // sole auto ref parameter is rejected as same as foo definition.
}

(In reply to Kenji Hara from comment #3)
> (In reply to Yuxuan Shui from comment #2)
> > Seems this bug is not related to nested templates at all, it's more likely
> > an 'auto ref' bug:
> 
> `auto ref` parameter is allowed for template functions, AND needs actual
> function argument to deduce its ref-ness. For example:
> 
> void foo(auto ref int x) {}    // non-template function
> // -> NG, auto can only be used for template function parameters
> 
> void bar(T)(auto ref T x) {}
> void main() {
>     int n;
> 
>     // IFTI (implicit function template instantiation)
>     bar(1);      // OK, parameter x is deduced to non-ref
>     bar(n);      // OK, parameter x is deduced to ref
> 
>     // partial template specialization + IFTI
>     bar!int(1);  // also OK, equivalent with bar(1)
>     bar!int(n);  // also OK, equivalent with bar(n)
> 
>     // explicit instantiation without IFTI
>     alias b = bar!int;
>     // NG! auto ref parameter x cannot deduce its ref-ness, therefore
>     // sole auto ref parameter is rejected as same as foo definition.
> }

At least the error message here needs some improvement. 

Besides, this still doesn't work even if I call them in devil().

(In reply to Kenji Hara from comment #3)
> (In reply to Yuxuan Shui from comment #2)
> > Seems this bug is not related to nested templates at all, it's more likely
> > an 'auto ref' bug:
> 
> `auto ref` parameter is allowed for template functions, AND needs actual
> function argument to deduce its ref-ness.

Your code compiles for 2.067 (except for foo).

I also did my own test (which compiles on 2.067 and a recent head version):

void foo()(auto ref int i)
{
}

void bar(T)(auto ref T t)
{
}

void main()
{
    foo!()(1);
    foo(1);
    bar(1);
    bar!int(1);
    alias x = foo!();
    alias y = bar!(int);
    x(1);
    y(1);
}

Which compiles.

OP's code does not. I'm unsure of the rules, or how IFTI is supposed to work, so I don't want to mark this as rejects-valid, but it sure seems that way. Perhaps my code is wrong and should be rejected?

(In reply to Steven Schveighoffer from comment #5)
> Your code compiles for 2.067 (except for foo).
> 
> I also did my own test (which compiles on 2.067 and a recent head version):
> 
[snip]
> 
> Which compiles.
> 
> OP's code does not. I'm unsure of the rules, or how IFTI is supposed to
> work, so I don't want to mark this as rejects-valid, but it sure seems that
> way. Perhaps my code is wrong and should be rejected?

Ah, it's order dependent bug in compiler. Test case:

void bar(T)(auto ref T x) {}
void main() {
    int n;

    bar(1); // or bar(n), bar!int(1), or others
    // if you mask this first instantiation, the next alias will report error.

    alias b = bar!int;
}

(In reply to Kenji Hara from comment #6)
> Ah, it's order dependent bug in compiler. Test case:

Ok, I'll fix it in the auto-ref related PR:
https://github.com/D-Programming-Language/dmd/pull/4729

trusting https://github.com/dlang/dmd/pull/4729
errors on test case is issue 9204