When a type T is small enough, Variant allows assignment from it even if it is not implicitly convertible to Unqual!T. However, when the size of the type is larger than VariantN's size argument, a different path is chosen, and the compiler chokes. This example code showcases the effect - for small i there is no problem, but then there's an compile error when i > 16, 12, or 8 (depending on compiler and flags):

struct S(int padding) {
    byte[padding] _;
    int* p;
}

unittest {
    import std.variant;
    
    static foreach (i; 0..64) {{
        const S!i s;
        Variant a = s;
    }}
}

The code on which the compiler chokes is this:
https://github.com/dlang/phobos/blob/master/std/variant.d#L699-L701

I'm unsure why larger types are treated differently. The comment on lines 667-674 seems to indicate it has to do with whether they're passed on the stack or not, but this assumption does not seem to be documented elsewhere.