ApiAnnotations: Add SourceText for unicode tokens
[ghc.git] / compiler / parser / ApiAnnotation.hs
1 {-# LANGUAGE DeriveDataTypeable #-}
2
3 module ApiAnnotation (
4 getAnnotation, getAndRemoveAnnotation,
5 getAnnotationComments,getAndRemoveAnnotationComments,
6 ApiAnns,
7 ApiAnnKey,
8 AnnKeywordId(..),
9 AnnotationComment(..),
10 IsUnicodeSyntax(..),
11 unicodeAnn,
12 LRdrName -- Exists for haddocks only
13 ) where
14
15 import RdrName
16 import Outputable
17 import SrcLoc
18 import qualified Data.Map as Map
19 import Data.Data
20
21
22 {-
23 Note [Api annotations]
24 ~~~~~~~~~~~~~~~~~~~~~~
25 In order to do source to source conversions using the GHC API, the
26 locations of all elements of the original source needs to be tracked.
27 This includes keywords such as 'let' / 'in' / 'do' etc as well as
28 punctuation such as commas and braces, and also comments.
29
30 These are captured in a structure separate from the parse tree, and
31 returned in the pm_annotations field of the ParsedModule type.
32
33 The non-comment annotations are stored indexed to the SrcSpan of the
34 AST element containing them, together with a AnnKeywordId value
35 identifying the specific keyword being captured.
36
37 > type ApiAnnKey = (SrcSpan,AnnKeywordId)
38 >
39 > Map.Map ApiAnnKey SrcSpan
40
41 So
42
43 > let x = 1 in 2 *x
44
45 would result in the AST element
46
47 L span (HsLet (binds for x = 1) (2 * x))
48
49 and the annotations
50
51 (span,AnnLet) having the location of the 'let' keyword
52 (span,AnnIn) having the location of the 'in' keyword
53
54
55 The comments are indexed to the SrcSpan of the lowest AST element
56 enclosing them
57
58 > Map.Map SrcSpan [Located AnnotationComment]
59
60 So the full ApiAnns type is
61
62 > type ApiAnns = ( Map.Map ApiAnnKey SrcSpan
63 > , Map.Map SrcSpan [Located AnnotationComment])
64
65
66 This is done in the lexer / parser as follows.
67
68
69 The PState variable in the lexer has the following variables added
70
71 > annotations :: [(ApiAnnKey,[SrcSpan])],
72 > comment_q :: [Located AnnotationComment],
73 > annotations_comments :: [(SrcSpan,[Located AnnotationComment])]
74
75 The first and last store the values that end up in the ApiAnns value
76 at the end via Map.fromList
77
78 The comment_q captures comments as they are seen in the token stream,
79 so that when they are ready to be allocated via the parser they are
80 available.
81
82 The parser interacts with the lexer using the function
83
84 > addAnnotation :: SrcSpan -> AnnKeywordId -> SrcSpan -> P ()
85
86 which takes the AST element SrcSpan, the annotation keyword and the
87 target SrcSpan.
88
89 This adds the annotation to the `annotations` field of `PState` and
90 transfers any comments in `comment_q` to the `annotations_comments`
91 field.
92
93 Parser
94 ------
95
96 The parser implements a number of helper types and methods for the
97 capture of annotations
98
99 > type AddAnn = (SrcSpan -> P ())
100 >
101 > mj :: AnnKeywordId -> Located e -> (SrcSpan -> P ())
102 > mj a l = (\s -> addAnnotation s a (gl l))
103
104 AddAnn represents the addition of an annotation a to a provided
105 SrcSpan, and `mj` constructs an AddAnn value.
106
107 > ams :: Located a -> [AddAnn] -> P (Located a)
108 > ams a@(L l _) bs = (mapM_ (\a -> a l) bs) >> return a
109
110 So the production in Parser.y for the HsLet AST element is
111
112 | 'let' binds 'in' exp {% ams (sLL $1 $> $ HsLet (snd $ unLoc $2) $4)
113 (mj AnnLet $1:mj AnnIn $3
114 :(fst $ unLoc $2)) }
115
116 This adds an AnnLet annotation for 'let', an AnnIn for 'in', as well
117 as any annotations that may arise in the binds. This will include open
118 and closing braces if they are used to delimit the let expressions.
119
120 The wiki page describing this feature is
121 https://ghc.haskell.org/trac/ghc/wiki/ApiAnnotations
122
123 -}
124 -- ---------------------------------------------------------------------
125
126 type ApiAnns = ( Map.Map ApiAnnKey [SrcSpan]
127 , Map.Map SrcSpan [Located AnnotationComment])
128
129 type ApiAnnKey = (SrcSpan,AnnKeywordId)
130
131
132 -- | Retrieve a list of annotation 'SrcSpan's based on the 'SrcSpan'
133 -- of the annotated AST element, and the known type of the annotation.
134 getAnnotation :: ApiAnns -> SrcSpan -> AnnKeywordId -> [SrcSpan]
135 getAnnotation (anns,_) span ann
136 = case Map.lookup (span,ann) anns of
137 Nothing -> []
138 Just ss -> ss
139
140 -- | Retrieve a list of annotation 'SrcSpan's based on the 'SrcSpan'
141 -- of the annotated AST element, and the known type of the annotation.
142 -- The list is removed from the annotations.
143 getAndRemoveAnnotation :: ApiAnns -> SrcSpan -> AnnKeywordId
144 -> ([SrcSpan],ApiAnns)
145 getAndRemoveAnnotation (anns,cs) span ann
146 = case Map.lookup (span,ann) anns of
147 Nothing -> ([],(anns,cs))
148 Just ss -> (ss,(Map.delete (span,ann) anns,cs))
149
150 -- |Retrieve the comments allocated to the current 'SrcSpan'
151 --
152 -- Note: A given 'SrcSpan' may appear in multiple AST elements,
153 -- beware of duplicates
154 getAnnotationComments :: ApiAnns -> SrcSpan -> [Located AnnotationComment]
155 getAnnotationComments (_,anns) span =
156 case Map.lookup span anns of
157 Just cs -> cs
158 Nothing -> []
159
160 -- |Retrieve the comments allocated to the current 'SrcSpan', and
161 -- remove them from the annotations
162 getAndRemoveAnnotationComments :: ApiAnns -> SrcSpan
163 -> ([Located AnnotationComment],ApiAnns)
164 getAndRemoveAnnotationComments (anns,canns) span =
165 case Map.lookup span canns of
166 Just cs -> (cs,(anns,Map.delete span canns))
167 Nothing -> ([],(anns,canns))
168
169 -- --------------------------------------------------------------------
170
171 -- | API Annotations exist so that tools can perform source to source
172 -- conversions of Haskell code. They are used to keep track of the
173 -- various syntactic keywords that are not captured in the existing
174 -- AST.
175 --
176 -- The annotations, together with original source comments are made
177 -- available in the @'pm_annotations'@ field of @'GHC.ParsedModule'@.
178 -- Comments are only retained if @'Opt_KeepRawTokenStream'@ is set in
179 -- @'DynFlags.DynFlags'@ before parsing.
180 --
181 -- The wiki page describing this feature is
182 -- https://ghc.haskell.org/trac/ghc/wiki/ApiAnnotations
183 --
184 -- Note: in general the names of these are taken from the
185 -- corresponding token, unless otherwise noted
186 -- See note [Api annotations] above for details of the usage
187 data AnnKeywordId
188 = AnnAs
189 | AnnAt
190 | AnnBang -- ^ '!'
191 | AnnBackquote -- ^ '`'
192 | AnnBy
193 | AnnCase -- ^ case or lambda case
194 | AnnClass
195 | AnnClose -- ^ '\#)' or '\#-}' etc
196 | AnnCloseC -- ^ '}'
197 | AnnCloseP -- ^ ')'
198 | AnnCloseS -- ^ ']'
199 | AnnColon
200 | AnnComma -- ^ as a list separator
201 | AnnCommaTuple -- ^ in a RdrName for a tuple
202 | AnnDarrow -- ^ '=>'
203 | AnnDarrowU -- ^ '=>', unicode variant
204 | AnnData
205 | AnnDcolon -- ^ '::'
206 | AnnDcolonU -- ^ '::', unicode variant
207 | AnnDefault
208 | AnnDeriving
209 | AnnDo
210 | AnnDot -- ^ '.'
211 | AnnDotdot -- ^ '..'
212 | AnnElse
213 | AnnEqual
214 | AnnExport
215 | AnnFamily
216 | AnnForall
217 | AnnForallU -- ^ Unicode variant
218 | AnnForeign
219 | AnnFunId -- ^ for function name in matches where there are
220 -- multiple equations for the function.
221 | AnnGroup
222 | AnnHeader -- ^ for CType
223 | AnnHiding
224 | AnnIf
225 | AnnImport
226 | AnnIn
227 | AnnInfix -- ^ 'infix' or 'infixl' or 'infixr'
228 | AnnInstance
229 | AnnLam
230 | AnnLarrow -- ^ '<-'
231 | AnnLarrowU -- ^ '<-', unicode variant
232 | AnnLet
233 | AnnMdo
234 | AnnMinus -- ^ '-'
235 | AnnModule
236 | AnnNewtype
237 | AnnName -- ^ where a name loses its location in the AST, this carries it
238 | AnnOf
239 | AnnOpen -- ^ '(\#' or '{-\# LANGUAGE' etc
240 | AnnOpenC -- ^ '{'
241 | AnnOpenP -- ^ '('
242 | AnnOpenPE -- ^ '$('
243 | AnnOpenPTE -- ^ '$$('
244 | AnnOpenS -- ^ '['
245 | AnnPackageName
246 | AnnPattern
247 | AnnProc
248 | AnnQualified
249 | AnnRarrow -- ^ '->'
250 | AnnRarrowU -- ^ '->', unicode variant
251 | AnnRec
252 | AnnRole
253 | AnnSafe
254 | AnnStar -- ^ '*'
255 | AnnStarU -- ^ '*', unicode variant.
256 | AnnSemi -- ^ ';'
257 | AnnSimpleQuote -- ^ '''
258 | AnnStatic -- ^ 'static'
259 | AnnThen
260 | AnnThIdSplice -- ^ '$'
261 | AnnThIdTySplice -- ^ '$$'
262 | AnnThTyQuote -- ^ double '''
263 | AnnTilde -- ^ '~'
264 | AnnTildehsh -- ^ '~#'
265 | AnnType
266 | AnnUnit -- ^ '()' for types
267 | AnnUsing
268 | AnnVal -- ^ e.g. INTEGER
269 | AnnValStr -- ^ String value, will need quotes when output
270 | AnnVbar -- ^ '|'
271 | AnnWhere
272 | Annlarrowtail -- ^ '-<'
273 | AnnlarrowtailU -- ^ '-<', unicode variant
274 | Annrarrowtail -- ^ '->'
275 | AnnrarrowtailU -- ^ '->', unicode variant
276 | AnnLarrowtail -- ^ '-<<'
277 | AnnLarrowtailU -- ^ '-<<', unicode variant
278 | AnnRarrowtail -- ^ '>>-'
279 | AnnRarrowtailU -- ^ '>>-', unicode variant
280 | AnnEofPos
281 deriving (Eq, Ord, Data, Typeable, Show)
282
283 instance Outputable AnnKeywordId where
284 ppr x = text (show x)
285
286 -- ---------------------------------------------------------------------
287
288 data AnnotationComment =
289 -- Documentation annotations
290 AnnDocCommentNext String -- ^ something beginning '-- |'
291 | AnnDocCommentPrev String -- ^ something beginning '-- ^'
292 | AnnDocCommentNamed String -- ^ something beginning '-- $'
293 | AnnDocSection Int String -- ^ a section heading
294 | AnnDocOptions String -- ^ doc options (prune, ignore-exports, etc)
295 | AnnDocOptionsOld String -- ^ doc options declared "-- # ..."-style
296 | AnnLineComment String -- ^ comment starting by "--"
297 | AnnBlockComment String -- ^ comment in {- -}
298 deriving (Eq, Ord, Data, Typeable, Show)
299 -- Note: these are based on the Token versions, but the Token type is
300 -- defined in Lexer.x and bringing it in here would create a loop
301
302 instance Outputable AnnotationComment where
303 ppr x = text (show x)
304
305 -- | - 'ApiAnnotation.AnnKeywordId' : 'ApiAnnotation.AnnOpen',
306 -- 'ApiAnnotation.AnnClose','ApiAnnotation.AnnComma',
307 -- 'ApiAnnotation.AnnRarrow','ApiAnnotation.AnnTildehsh',
308 -- 'ApiAnnotation.AnnTilde'
309 -- - May have 'ApiAnnotation.AnnComma' when in a list
310 type LRdrName = Located RdrName
311
312
313 -- | Certain tokens can have alternate representations when unicode syntax is
314 -- enabled. This flag is attached to those tokens in the lexer so that the
315 -- original source representation can be reproduced in the corresponding
316 -- 'ApiAnnotation'
317 data IsUnicodeSyntax = UnicodeSyntax | NormalSyntax
318 deriving (Eq, Ord, Data, Typeable, Show)
319
320 -- | Convert a normal annotation into its unicode equivalent one
321 unicodeAnn :: AnnKeywordId -> AnnKeywordId
322 unicodeAnn AnnForall = AnnForallU
323 unicodeAnn AnnDcolon = AnnDcolonU
324 unicodeAnn AnnLarrow = AnnLarrowU
325 unicodeAnn AnnRarrow = AnnRarrowU
326 unicodeAnn AnnDarrow = AnnDarrowU
327 unicodeAnn Annlarrowtail = AnnLarrowtailU
328 unicodeAnn Annrarrowtail = AnnrarrowtailU
329 unicodeAnn AnnLarrowtail = AnnLarrowtailU
330 unicodeAnn AnnRarrowtail = AnnRarrowtailU
331 unicodeAnn AnnStar = AnnStarU
332 unicodeAnn ann = ann
333 -- What about '*'?