D issues are now tracked on GitHub. This Bugzilla instance remains as a read-only archive.
Issue 3425 - StdioException on end of stdin on Windows
Summary: StdioException on end of stdin on Windows
Status: RESOLVED FIXED
Alias: None
Product: D
Classification: Unclassified
Component: phobos (show other issues)
Version: D2
Hardware: Other Windows
: P2 major
Assignee: No Owner
URL:
Keywords: trivial
Depends on:
Blocks:
 
Reported: 2009-10-20 11:55 UTC by David Simcha
Modified: 2017-02-19 14:00 UTC (History)
7 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description David Simcha 2009-10-20 11:55:53 UTC
Code:

import std.stdio, std.algorithm;

void main(string[] args) {
    auto foo = stdin.byLine();
    foreach(elem; foo) {
        writeln(elem);
    }
}

Input file (foo.txt) :
foo
bar

Result:

F:\>cat foo.txt | test2.exe
foo
bar
std.stdio.StdioException: Bad file descriptor

Works perfectly on Linux.  Haven't tested on Mac, BSD or whatever the heck else DMD runs on lately.
Comment 1 David Simcha 2011-03-09 20:41:02 UTC
I've managed to figure out where this exception is coming from, but I don't know how to fix it because I don't know why errno is getting set.  It's on line 2271 in stdio.d:

    if (fp._flag & _IONBF)
    {
        /* Use this for unbuffered I/O, when running
         * across buffer boundaries, or for any but the common
         * cases.
         */
      L1:
        auto app = appender(buf);
        app.clear();
        if(app.capacity == 0)
            app.reserve(128); // get at least 128 bytes available

        int c;
        while((c = FGETC(fp)) != -1) {
            app.put(cast(char) c);
            if(c == terminator) {
                buf = app.data;
                return buf.length;
            }

        }

        if (ferror(fps))
            StdioException();

Can we **PLEASE** fix this one ASAP?  It's been in Bugzilla for over a year and a half and makes it impossible to write simple text filter programs on Windows without ugly and/or unsafe workarounds.
Comment 2 Jay Norwood 2011-10-12 20:03:13 UTC
You can work around the issue by testing for eof  on the first line in the loop. This works with no error.   

    foreach(elem; stdin.byLine()) {
        if (stdin.eof()) break;
        writeln(elem);
    }
Comment 3 Max Vilimpoc 2011-12-28 17:09:15 UTC
(In reply to comment #2)
> You can work around the issue by testing for eof  on the first line in the
> loop. This works with no error.   
> 
>     foreach(elem; stdin.byLine()) {
>         if (stdin.eof()) break;
>         writeln(elem);
>     }

This didn't work for me, when using the example program on the D homepage:

	import std.stdio;

	void main()
	{
		ulong lines = 0;
		double sumLength = 0;
		foreach(line; stdin.byLine())
		{
			if (stdin.eof()) break;

			++lines;
			sumLength += line.length;
		}
		
		writeln("Average line length = ", lines ? sumLength / lines : 0);
	}

I still get:

	std.stdio.StdioException@std\stdio.d(2159): Bad file descriptor
Comment 4 Max Vilimpoc 2011-12-28 17:17:14 UTC
However, I did find that if I modify stdio.d in Phobos to the following under the L1 label around line 2280, then the other form of piping, i.e. "type filename | executable", will work on Windows. 

	if (ferror(fps) && EPIPE != ferror(fps))
		StdioException();

In the "|" pipe case, it could be that perhaps the "type" command already closed down its end by the time ferror() was called on the receiving side.

On Windows, "executable < filename" piping worked fine, but is not ideal for stringing filters together.

Here's the writeln-debugged block of stdio.d code that works:

// Private implementation of readln
version (DIGITAL_MARS_STDIO)
private size_t readlnImpl(FILE* fps, ref char[] buf, dchar terminator = '\n')
{
    FLOCK(fps);
    scope(exit) FUNLOCK(fps);

	writeln("Entered readlnImpl()");
	scope(exit) writeln("Exited readlnImpl()");

    /* Since fps is now locked, we can create an "unshared" version
     * of fp.
     */
    auto fp = cast(_iobuf*)fps;

    if (__fhnd_info[fp._file] & FHND_WCHAR)
    {   
		/* Stream is in wide characters.
         * Read them and convert to chars.
         */
		writeln("Entered if (__fhnd_info[fp._file] & FHND_WCHAR)");

        static assert(wchar_t.sizeof == 2);
        auto app = appender(buf);
        app.clear();
        for (int c = void; (c = FGETWC(fp)) != -1; )
        {
            if ((c & ~0x7F) == 0)
            {   app.put(cast(char) c);
                if (c == terminator)
                    break;
            }
            else
            {
                if (c >= 0xD800 && c <= 0xDBFF)
                {
                    int c2 = void;
                    if ((c2 = FGETWC(fp)) != -1 ||
                            c2 < 0xDC00 && c2 > 0xDFFF)
                    {
                        StdioException("unpaired UTF-16 surrogate");
                    }
                    c = ((c - 0xD7C0) << 10) + (c2 - 0xDC00);
                }
                //std.utf.encode(buf, c);
                app.put(cast(dchar)c);
            }
        }
        if (ferror(fps))
            StdioException();
        buf = app.data;
        return buf.length;
    }

    auto sz = GC.sizeOf(buf.ptr);
    //auto sz = buf.length;
    buf = buf.ptr[0 .. sz];
    if (fp._flag & _IONBF)
    {
		writeln("Entered if (fp._flag & _IONBF)");
		
        /* Use this for unbuffered I/O, when running
         * across buffer boundaries, or for any but the common
         * cases.
         */
      L1:
        auto app = appender(buf);
        app.clear();
        if(app.capacity == 0)
            app.reserve(128); // get at least 128 bytes available

        int c;
		writeln("fp._cnt: ", fp._cnt);

        while((c = FGETC(fp)) != -1) {
			writeln("chars: ", cast(char) c);
            app.put(cast(char) c);
            if(c == terminator) {
				writeln("hit terminator");
                buf = app.data;
                return buf.length;
            }
        }

		writeln("feof(fps): ", feof(fps));
		writeln("ferror(fps): ", ferror(fps));

		// If EPIPE is seen then probably the other side has closed
		// already. This is the case when using, for example:
		//
		// "type filename | D-program" syntax on Windows.
        if (ferror(fps) && EPIPE != ferror(fps))
            StdioException();

        buf = app.data;
        return buf.length;
    }
    else
    {
		writeln("Entered if (!(fp._flag & _IONBF))");
	
        int u = fp._cnt;
        char* p = fp._ptr;
        int i;
	
		writeln("length of stdin fp: ", u);
		
        if (fp._flag & _IOTRAN)
        {   /* Translated mode ignores \r and treats ^Z as end-of-file
             */
			writeln("Entered if (fp._flag & _IOTRAN)");

            char c;
            while (1)
            {
                if (i == u)                // if end of buffer
                    goto L1;        // give up
                c = p[i];
                i++;
                if (c != '\r')
                {
                    if (c == terminator)
                        break;
                    if (c != 0x1A)
                        continue;
                    goto L1;
                }
                else
                {   if (i != u && p[i] == terminator)
                        break;
                    goto L1;
                }
            }
            if (i > sz)
            {
                buf = uninitializedArray!(char[])(i);
            }
            if (i - 1)
                memcpy(buf.ptr, p, i - 1);
            buf[i - 1] = cast(char)terminator;
            buf = buf[0 .. i];
            if (terminator == '\n' && c == '\r')
                i++;
        }
        else
        {
			writeln("Entered if !(fp._flag & _IOTRAN)");
	
            while (1)
            {
                if (i == u)                // if end of buffer
                    goto L1;        // give up
                auto c = p[i];
                i++;
                if (c == terminator)
                    break;
            }
            if (i > sz)
            {
                buf = uninitializedArray!(char[])(i);
            }
            memcpy(buf.ptr, p, i);
            buf = buf[0 .. i];
        }
        fp._cnt -= i;
        fp._ptr += i;
        return i;
    }
}

And here's what it outputs:

c:\>avg < index.html

Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
chars: <
chars: h
chars: t
chars: m
chars: l
chars: >
chars:

hit terminator
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 106
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 97
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 37
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 27
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 18
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 8
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
feof(fps): 16
ferror(fps): 0
Exited readlnImpl()
Average line length = 15.1429

c:\>type index.html | avg
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
chars: <
chars: h
chars: t
chars: m
chars: l
chars: >
chars:

hit terminator
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 106
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 97
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 37
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 27
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 18
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 8
Entered if !(fp._flag & _IOTRAN)
Exited readlnImpl()
Entered readlnImpl()
Entered if (!(fp._flag & _IONBF))
length of stdin fp: 0
Entered if !(fp._flag & _IOTRAN)
fp._cnt: 0
feof(fps): 0
ferror(fps): 32
Exited readlnImpl()
Average line length = 15.1429
Comment 5 Rainer Schuetze 2014-12-13 07:14:38 UTC
It seems this problem doesn't exist anymore.
Comment 6 Marc Schütz 2015-05-15 11:05:11 UTC
According to a user on the forums, this bug still occurs with DMD 2.067.1:

http://forum.dlang.org/thread/fqdukibittdcamnwqagy@forum.dlang.org
Comment 7 notna 2017-02-19 14:00:26 UTC
works well on:
Microsoft Windows [Version 10.0.14393]

with:
C:\Users\notna>dmd --version
DMD32 D Compiler v2.073.1
Copyright (c) 1999-2016 by Digital Mars written by Walter Bright