Issue 24524 - Very slow process fork if RLIMIT_NOFILE is too high
Summary: Very slow process fork if RLIMIT_NOFILE is too high
Status: RESOLVED FIXED
Alias: None
Product: D
Classification: Unclassified
Component: phobos (show other issues)
Version: D2
Hardware: x86_64 Linux
: P1 enhancement
Assignee: No Owner
URL:
Keywords: pull
Depends on:
Blocks:
 
Reported: 2024-04-27 09:30 UTC by Andrea Fontana
Modified: 2024-12-01 16:42 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description Andrea Fontana 2024-04-27 09:30:04 UTC
In `std.process`, the `spawnProcessPosix` function, used by all functions that start a process, has a flag `Config.Flags.inheritFDs`. If set to false, it reads the maximum limit of files that can be opened by a process, `RFILE_NOFILE`, and then creates a structure for each one and finally iterates through them.

Normally, the `RFILE_NOFILE` limit is set to 1024 or anyway a relatively low number. The hard limit depends on the system; in some, it is 2^20, in others even 2^30.

When starting a container with Docker, the limit is raised to the maximum allowed, and this inevitably slows down the process visibly (because it has to allocate 2^20 (or 2^30) elements and then iterate through them. An user reported that on a system with 2^30 as the hard limit, it becomes literally unusable.

There would be a faster way to do this by reading the open file descriptors from `/dev/fd` with the code below, but we are inside a `@nogc` block, so another solution must be found.

`
import std.file : exists, isDir;

if (exists("/dev/fd") && isDir("/dev/fd"))
{
    import std.file : dirEntries, SpanMode;
    import std.path : baseName;
    import std.algorithm : map, filter;
    import std.array : array;
    import std.string : indexOfNeither;
    import std.conv : to;

    // List all file descriptors except stdin, stdout, stderr, and the pipe
    auto fds = dirEntries("/dev/fd", SpanMode.shallow)
        .map!(d => d.name.baseName)
        .filter!(d => d.indexOfNeither("0123456789") < 0)
        .map!(d => d.to!int)
        .filter!(d => d >=3 && d != forkPipeOut)
        .array; // Copy to avoid closing while iterating

    // Close only the file descriptors that are open.
    foreach(fd; fds)
        close(fd);
}
`
Comment 1 Dlang Bot 2024-04-27 22:57:25 UTC
@trikko updated dlang/phobos pull request #8990 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" fixing this issue:

- Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high

https://github.com/dlang/phobos/pull/8990
Comment 2 Dlang Bot 2024-10-30 01:10:07 UTC
@schveiguy created dlang/phobos pull request #9077 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" fixing this issue:

- Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high

https://github.com/dlang/phobos/pull/9077
Comment 3 Dlang Bot 2024-11-01 03:45:22 UTC
dlang/phobos pull request #9077 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" was merged into master:

- 3c81b1ac8514ac17bef4375768eb9a0448bde9ac by Steven Schveighoffer:
  Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high

https://github.com/dlang/phobos/pull/9077
Comment 4 dlangBugzillaToGithub 2024-12-01 16:42:27 UTC
THIS ISSUE HAS BEEN MOVED TO GITHUB

https://github.com/dlang/phobos/issues/10548

DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB