D issues are now tracked on GitHub. This Bugzilla instance remains as a read-only archive.
Issue 16626 - [Reg 2.073] extreme CTFE memory usage with compile time regex
Summary: [Reg 2.073] extreme CTFE memory usage with compile time regex
Status: RESOLVED FIXED
Alias: None
Product: D
Classification: Unclassified
Component: dmd (show other issues)
Version: D2
Hardware: All All
: P1 regression
Assignee: No Owner
URL:
Keywords: pull
Depends on:
Blocks:
 
Reported: 2016-10-19 23:01 UTC by Martin Nowak
Modified: 2017-01-17 11:24 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description Martin Nowak 2016-10-19 23:01:38 UTC
Building a debug tests of Higgs[¹] doesn't finish building and runs out of memory with dmd-2016-10-10 (v2.073.0-master-9a935f6). Only takes about 2GB with dmd-2016-10-09 (v2.073.0-master-321c95e).
According to verbose output it hangs at `semantic3 parser`.

Digger says it's caused by https://github.com/dlang/phobos/pull/4286.

[¹]: https://github.com/higgsjs/Higgs
Comment 1 Martin Nowak 2016-10-19 23:07:58 UTC
Seems to happen for 2 ctRegex to filter out non-alphanumerical characters.

auto nonAlnum = ctRegex!(`[^\w]`w);
enum notAlnum = ctRegex!(`[^0-9|a-z|A-Z]`, "g");
Comment 2 Martin Nowak 2016-10-23 22:50:13 UTC
From Dmitry https://github.com/dlang/phobos/pull/4286#issuecomment-255182046

> The new version is more demanding on memory during CTFE so yeah, most likely it now fails for some patterns. Dunno what to do here would really love to see the new engine sometime soon.

That might still take a while and in the meantime such simple ctRegex eat up 16GiB (kill my project tester https://ci.dawg.eu/ btw).
If we can't really support the NFA kickstarter in CTFE, then we should just disable it for now.
Comment 3 Dmitry Olshansky 2016-10-24 19:21:08 UTC
https://github.com/dlang/phobos/pull/4877
Comment 4 Martin Nowak 2016-12-24 10:55:31 UTC
Wasn't fully resolved by the PR.
https://github.com/dlang/phobos/pull/4877#issuecomment-257186040
Comment 5 Martin Nowak 2016-12-24 16:06:49 UTC
More specifically this commit https://github.com/dlang/phobos/commit/e98fa4ad5ad39487844c91357cfec4f698e88230 is responsible for the high memory usage.

Also both, regex and ctRegex, trigger the bug if initialized during CTFE.

enum notAlnum = regex(`[^0-9|a-z|A-Z]`);
enum ctNotAlnum = ctRegex!(`[^0-9|a-z|A-Z]`);

https://github.com/dlang/phobos/pull/4995
Comment 6 github-bugzilla 2016-12-26 23:33:37 UTC
Commits pushed to master at https://github.com/dlang/phobos

https://github.com/dlang/phobos/commit/07090ae2fdaddd1c40b8c149f32733e908094ac3
disable kickstart in ctfe to workaround Issue 16626

- consumes too much memory, introduced by
  e98fa4ad5ad39487844c91357cfec4f698e88230 (#4286)

https://github.com/dlang/phobos/commit/9cbc862544a9a4afa14bb526ec4aff7792a71e49
Merge pull request #4995 from MartinNowak/fix16626

disable kickstart in ctfe to workaround Issue 16626
Comment 7 github-bugzilla 2017-01-07 03:03:20 UTC
Commits pushed to stable at https://github.com/dlang/phobos

https://github.com/dlang/phobos/commit/07090ae2fdaddd1c40b8c149f32733e908094ac3
disable kickstart in ctfe to workaround Issue 16626

https://github.com/dlang/phobos/commit/9cbc862544a9a4afa14bb526ec4aff7792a71e49
Merge pull request #4995 from MartinNowak/fix16626
Comment 8 github-bugzilla 2017-01-16 23:26:14 UTC
Commits pushed to newCTFE at https://github.com/dlang/phobos

https://github.com/dlang/phobos/commit/07090ae2fdaddd1c40b8c149f32733e908094ac3
disable kickstart in ctfe to workaround Issue 16626

https://github.com/dlang/phobos/commit/9cbc862544a9a4afa14bb526ec4aff7792a71e49
Merge pull request #4995 from MartinNowak/fix16626