Issue 19947 - OutOfMemoryError with GC-allocs
Summary: OutOfMemoryError with GC-allocs
Status: NEW
Alias: None
Product: D
Classification: Unclassified
Component: dmd (show other issues)
Version: D2
Hardware: x86 Windows
: P2 major
Assignee: No Owner
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-06-07 14:46 UTC by a11e99z
Modified: 2022-12-17 10:39 UTC (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description a11e99z 2019-06-07 14:46:30 UTC
program bellow crashed

// dmd -m32 -release -O gctest.d
import std, core.memory;
void main() {
    long r = 0; // using results = dont drop code
    for(int i = 0; i < 1000; i++) {
        auto x = new int[10000000];
        r += x.sum();
    }
    write( r );
}

C:\content\downloadz\dlang>gctest.exe
core.exception.OutOfMemoryError@src\core\exception.d(702): Memory allocation failed
----------------

with -m64 no crashes but prog has more memory than 2GB. 
it cannot be false positive coz array contains all 0s (int.init)
imo something wrong with gc-roots or gc-marks.
code by LDC works fine.
Comment 1 a11e99z 2019-06-07 14:50:17 UTC
Windows Server 2019 (x64)
DMD 2.086.0
LDC 1.16.0-beta2
Comment 2 anonymous4 2019-06-07 15:58:55 UTC
It can be false positive, see issue 15723
Comment 3 a11e99z 2019-06-07 21:30:19 UTC
maybe u right.
imo gc shouldnt scan all data segment for pointers:
when we assign to some globals some pointers or explicitly called GC.addRange/addRoot. and when I transfer something to C(or others)-libs.. well, its my responsibility only, not compiler or GC
Comment 4 Gregor Mückl 2019-06-08 17:40:28 UTC
I don't think that it can be false positives. Adding a GC.collect() call will clean up the allocations; so the issue must rather be that collection is not started for some reason. 

See also original forum discussion: https://forum.dlang.org/thread/obkpctudtnnfojrbrwiv@forum.dlang.org
Comment 5 Rainer Schuetze 2019-06-12 20:35:51 UTC
To avoid false pointers in the heap and the data segment, use
--DRT-gcopt=gc:precise --DRT-scanDataSeg=precise

With these the program runs to completion.
Comment 6 Gregor Mückl 2019-06-21 08:16:14 UTC
(In reply to Rainer Schuetze from comment #5)
> To avoid false pointers in the heap and the data segment, use
> --DRT-gcopt=gc:precise --DRT-scanDataSeg=precise
> 
> With these the program runs to completion.

How does this square with the fact that calling GC.collect() inside the for loop also makes the program run to completion? Also, the array is initialized with 0, so any false false pointers would be invisible to the user code.
Comment 7 Rainer Schuetze 2019-06-21 09:00:10 UTC
> How does this square with the fact that calling GC.collect()
> inside the for loop also makes the program run to completion?

When running GC.collect() after the allocation, you are changing the amount of memory that the GC overallocates before running the next collection (2 times the live memory at the time of the previous collection). This changes the probability of getting false pointers.

> Also, the array is initialized with 0, so any false false pointers
> would be invisible to the user code.

The content of the int[] array is not scanned, the false pointers in the data segment (and the stack, -gx might help) are the issue here. These can pin the large allocations. Address randomization (ASLR) makes it rather unpredictable, too.