Implement underscores in numeric literals (NumericUnderscores extension)
authorTakenobu Tani <takenobu.hs@gmail.com>
Sun, 21 Jan 2018 17:08:59 +0000 (12:08 -0500)
committerBen Gamari <ben@smart-cactus.org>
Sun, 21 Jan 2018 17:09:00 +0000 (12:09 -0500)
commit4a13c5b1f4beb53cbf1f3529acdf3ba37528e694
treecfce02f26d602175f99699763b0c654b4b65033e
parent180ca65ff6d1b4f3f4cdadc569fd4de107be14db
Implement underscores in numeric literals (NumericUnderscores extension)

Implement the proposal of underscores in numeric literals.
Underscores in numeric literals are simply ignored.

The specification of the feature is available here:
https://github.com/ghc-proposals/ghc-proposals/blob/master/proposals/000
9-numeric-underscores.rst

For a discussion of the various choices:
https://github.com/ghc-proposals/ghc-proposals/pull/76

Implementation detail:

* Added dynamic flag
  * `NumericUnderscores` extension flag is added for this feature.

* Alex "Regular expression macros" in Lexer.x
  * Add `@numspc` (numeric spacer) macro to represent multiple
    underscores.
  * Modify `@decimal`, `@decimal`, `@binary`, `@octal`, `@hexadecimal`,
    `@exponent`, and `@bin_exponent` macros to include `@numspc`.

* Alex "Rules" in Lexer.x
  * To be simpler, we have only the definitions with underscores.
    And then we have a separate function (`tok_integral` and `tok_frac`)
    that validates the literals.

* Validation functions in Lexer.x
  * `tok_integral` and `tok_frac` functions validate
    whether contain underscores or not.
    If `NumericUnderscores` extensions are not enabled,
    check that there are no underscores.
  * `tok_frac` function is created by merging `strtoken` and
    `init_strtoken`.
  * `init_strtoken` is deleted. Because it is no longer used.

* Remove underscores from target literal string
  * `parseUnsignedInteger`, `readRational__`, and `readHexRational} use
    the customized `span'` function to remove underscores.

* Added Testcase
  * testcase for NumericUnderscores enabled.
      NumericUnderscores0.hs and NumericUnderscores1.hs
  * testcase for NumericUnderscores disabled.
      NoNumericUnderscores0.hs and NoNumericUnderscores1.hs
  * testcase to invalid pattern for NumericUnderscores enabled.
      NumericUnderscoresFail0.hs and NumericUnderscoresFail1.hs

Test Plan: `validate` including the above testcase

Reviewers: goldfire, bgamari

Reviewed By: bgamari

Subscribers: carter, rwbarton, thomie

GHC Trac Issues: #14473

Differential Revision: https://phabricator.haskell.org/D4235
21 files changed:
compiler/main/DynFlags.hs
compiler/parser/Lexer.x
compiler/utils/StringBuffer.hs
compiler/utils/Util.hs
docs/users_guide/glasgow_exts.rst
libraries/ghc-boot-th/GHC/LanguageExtensions/Type.hs
testsuite/tests/driver/T4437.hs
testsuite/tests/parser/should_fail/NoNumericUnderscores0.hs [new file with mode: 0644]
testsuite/tests/parser/should_fail/NoNumericUnderscores0.stderr [new file with mode: 0644]
testsuite/tests/parser/should_fail/NoNumericUnderscores1.hs [new file with mode: 0644]
testsuite/tests/parser/should_fail/NoNumericUnderscores1.stderr [new file with mode: 0644]
testsuite/tests/parser/should_fail/NumericUnderscoresFail0.hs [new file with mode: 0644]
testsuite/tests/parser/should_fail/NumericUnderscoresFail0.stderr [new file with mode: 0644]
testsuite/tests/parser/should_fail/NumericUnderscoresFail1.hs [new file with mode: 0644]
testsuite/tests/parser/should_fail/NumericUnderscoresFail1.stderr [new file with mode: 0644]
testsuite/tests/parser/should_fail/all.T
testsuite/tests/parser/should_run/NumericUnderscores0.hs [new file with mode: 0644]
testsuite/tests/parser/should_run/NumericUnderscores0.stdout [new file with mode: 0644]
testsuite/tests/parser/should_run/NumericUnderscores1.hs [new file with mode: 0644]
testsuite/tests/parser/should_run/NumericUnderscores1.stdout [new file with mode: 0644]
testsuite/tests/parser/should_run/all.T