token
packagestandard library
This package is not in the latest version of its module.
Details
Validgo.mod file
The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
- Redistributable license
Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
- Tagged version
Modules with tagged versions give importers more predictable builds.
- Stable version
When a project reaches major version v1 it is considered stable.
- Learn more about best practices
Repository
Links
Report a Vulnerability
Documentation¶
Overview¶
Package token defines constants representing the lexical tokens of the Goprogramming language and basic operations on tokens (printing, predicates).
Example (RetrievePositionInfo)¶
package mainimport ("fmt""go/ast""go/parser""go/token")func main() {fset := token.NewFileSet()const src = `package mainimport "fmt"import "go/token"//line :1:5type p = token.Posconst bad = token.NoPos//line fake.go:42:11func ok(pos p) bool {return pos != bad}/*line :7:9*/func main() {fmt.Println(ok(bad) == bad.IsValid())}`f, err := parser.ParseFile(fset, "main.go", src, 0)if err != nil {fmt.Println(err)return}// Print the location and kind of each declaration in f.for _, decl := range f.Decls {// Get the filename, line, and column back via the file set.// We get both the relative and absolute position.// The relative position is relative to the last line directive.// The absolute position is the exact position in the source.pos := decl.Pos()relPosition := fset.Position(pos)absPosition := fset.PositionFor(pos, false)// Either a FuncDecl or GenDecl, since we exit on error.kind := "func"if gen, ok := decl.(*ast.GenDecl); ok {kind = gen.Tok.String()}// If the relative and absolute positions differ, show both.fmtPosition := relPosition.String()if relPosition != absPosition {fmtPosition += "[" + absPosition.String() + "]"}fmt.Printf("%s: %s\n", fmtPosition, kind)}}Output:main.go:3:1: importmain.go:5:1: importmain.go:1:5[main.go:8:1]: typemain.go:3:1[main.go:10:1]: constfake.go:42:11[main.go:13:1]: funcfake.go:7:9[main.go:17:14]: func
Index¶
- Constants
- func IsExported(name string) bool
- func IsIdentifier(name string) bool
- func IsKeyword(name string) bool
- type File
- func (f *File) AddLine(offset int)
- func (f *File) AddLineColumnInfo(offset int, filename string, line, column int)
- func (f *File) AddLineInfo(offset int, filename string, line int)
- func (f *File) Base() int
- func (f *File) Line(p Pos) int
- func (f *File) LineCount() int
- func (f *File) LineStart(line int) Pos
- func (f *File) Lines() []int
- func (f *File) MergeLine(line int)
- func (f *File) Name() string
- func (f *File) Offset(p Pos) int
- func (f *File) Pos(offset int) Pos
- func (f *File) Position(p Pos) (pos Position)
- func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)
- func (f *File) SetLines(lines []int) bool
- func (f *File) SetLinesForContent(content []byte)
- func (f *File) Size() int
- type FileSet
- func (s *FileSet) AddExistingFiles(files ...*File)
- func (s *FileSet) AddFile(filename string, base, size int) *File
- func (s *FileSet) Base() int
- func (s *FileSet) File(p Pos) (f *File)
- func (s *FileSet) Iterate(yield func(*File) bool)
- func (s *FileSet) Position(p Pos) (pos Position)
- func (s *FileSet) PositionFor(p Pos, adjusted bool) (pos Position)
- func (s *FileSet) Read(decode func(any) error) error
- func (s *FileSet) RemoveFile(file *File)
- func (s *FileSet) Write(encode func(any) error) error
- type Pos
- type Position
- type Token
Examples¶
Constants¶
const (LowestPrec = 0// non-operatorsUnaryPrec = 6HighestPrec = 7)
A set of constants for precedence-based expression parsing.Non-operators have lowest precedence, followed by operatorsstarting with precedence 1 up to unary operators. The highestprecedence serves as "catch-all" precedence for selector,indexing, and other operator and delimiter tokens.
Variables¶
This section is empty.
Functions¶
funcIsExported¶added ingo1.13
IsExported reports whether name starts with an upper-case letter.
funcIsIdentifier¶added ingo1.13
IsIdentifier reports whether name is a Go identifier, that is, a non-emptystring made up of letters, digits, and underscores, where the first characteris not a digit. Keywords are not identifiers.
Types¶
typeFile¶
type File struct {// contains filtered or unexported fields}A File is a handle for a file belonging to aFileSet.A File has a name, size, and line offset table.
UseFileSet.AddFile to create a File.A File may belong to more than one FileSet; seeFileSet.AddExistingFiles.
func (*File)AddLine¶
AddLine adds the line offset for a new line.The line offset must be larger than the offset for the previous lineand smaller than the file size; otherwise the line offset is ignored.
func (*File)AddLineColumnInfo¶added ingo1.11
AddLineColumnInfo adds alternative file, line, and column numberinformation for a given file offset. The offset must be largerthan the offset for the previously added alternative line infoand smaller than the file size; otherwise the information isignored.
AddLineColumnInfo is typically used to register alternative positioninformation for line directives such as //line filename:line:column.
func (*File)AddLineInfo¶
AddLineInfo is likeFile.AddLineColumnInfo with a column = 1 argument.It is here for backward-compatibility for code prior to Go 1.11.
func (*File)Line¶
Line returns the line number for the given file position p;p must be aPos value in that file orNoPos.
func (*File)LineStart¶added ingo1.12
LineStart returns thePos value of the start of the specified line.It ignores any alternative positions set usingFile.AddLineColumnInfo.LineStart panics if the 1-based line number is invalid.
func (*File)Lines¶added ingo1.21.0
Lines returns the effective line offset table of the form described byFile.SetLines.Callers must not mutate the result.
func (*File)MergeLine¶added ingo1.2
MergeLine merges a line with the following line. It is akin to replacingthe newline character at the end of the line with a space (to not change theremaining offsets). To obtain the line number, consult e.g. [Position.Line].MergeLine will panic if given an invalid line number.
func (*File)Offset¶
Offset returns the offset for the given file position p.
If p is before the file's start position (or if p is NoPos),the result is 0; if p is past the file's end position,the result is the file size (see also go.dev/issue/57490).
The following invariant, though not true for offset valuesin general, holds for the result offset:f.Offset(f.Pos(offset)) == offset
func (*File)Pos¶
Pos returns the Pos value for the given file offset.
If offset is negative, the result is the file's startposition; if the offset is too large, the result isthe file's end position (see also go.dev/issue/57490).
The following invariant, though not true for Pos valuesin general, holds for the result p:f.Pos(f.Offset(p)) == p.
func (*File)Position¶
Position returns the Position value for the given file position p.If p is out of bounds, it is adjusted to match the File.Offset behavior.Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).
func (*File)PositionFor¶added ingo1.4
PositionFor returns the Position value for the given file position p.If p is out of bounds, it is adjusted to match the File.Offset behavior.If adjusted is set, the position may be adjusted by position-altering//line comments; otherwise those comments are ignored.p must be a Pos value in f or NoPos.
func (*File)SetLines¶
SetLines sets the line offsets for a file and reports whether it succeeded.The line offsets are the offsets of the first character of each line;for instance for the content "ab\nc\n" the line offsets are {0, 3}.An empty file has an empty line offset table.Each line offset must be larger than the offset for the previous lineand smaller than the file size; otherwise SetLines fails and returnsfalse.Callers must not mutate the provided slice after SetLines returns.
func (*File)SetLinesForContent¶
SetLinesForContent sets the line offsets for the given file content.It ignores position-altering //line comments.
typeFileSet¶
type FileSet struct {// contains filtered or unexported fields}A FileSet represents a set of source files.Methods of file sets are synchronized; multiple goroutinesmay invoke them concurrently.
The byte offsets for each file in a file set are mapped intodistinct (integer) intervals, one interval [base, base+size]per file.FileSet.Base represents the first byte in the file, and sizeis the corresponding file size. APos value is a value in suchan interval. By determining the interval aPos value belongsto, the file, its file base, and thus the byte offset (position)thePos value is representing can be computed.
When adding a new file, a file base must be provided. That canbe any integer value that is past the end of any interval of anyfile already in the file set. For convenience,FileSet.Base providessuch a value, which is simply the end of the Pos interval of the mostrecently added file, plus one. Unless there is a need to extend aninterval later, using theFileSet.Base should be used as argumentforFileSet.AddFile.
AFile may be removed from a FileSet when it is no longer needed.This may reduce memory usage in a long-running application.
func (*FileSet)AddExistingFiles¶added ingo1.25.0
AddExistingFiles adds the specified files to theFileSet if they are not already present.The caller must ensure that no pair of Files thatwould appear in the resulting FileSet overlap.
func (*FileSet)AddFile¶
AddFile adds a new file with a given filename, base offset, and file sizeto the file set s and returns the file. Multiple files may have the samename. The base offset must not be smaller than theFileSet.Base, andsize must not be negative. As a special case, if a negative base is provided,the current value of theFileSet.Base is used instead.
Adding the file will set the file set'sFileSet.Base value to base + size + 1as the minimum base value for the next file. The following relationshipexists between aPos value p for a given file offset offs:
int(p) = base + offs
with offs in the range [0, size] and thus p in the range [base, base+size].For convenience,File.Pos may be used to create file-specific positionvalues from a file offset.
func (*FileSet)Base¶
Base returns the minimum base offset that must be provided toFileSet.AddFile when adding the next file.
func (*FileSet)File¶
File returns the file that contains the position p.If no such file is found (for instance for p ==NoPos),the result is nil.
func (*FileSet)Iterate¶
Iterate calls yield for the files in the file set in ascending Baseorder until yield returns false.
func (*FileSet)Position¶
Position converts aPos p in the fileset into a Position value.Calling s.Position(p) is equivalent to calling s.PositionFor(p, true).
func (*FileSet)PositionFor¶added ingo1.4
PositionFor converts aPos p in the fileset into aPosition value.If adjusted is set, the position may be adjusted by position-altering//line comments; otherwise those comments are ignored.p must be aPos value in s orNoPos.
func (*FileSet)RemoveFile¶added ingo1.20
RemoveFile removes a file from theFileSet so that subsequentqueries for itsPos interval yield a negative result.This reduces the memory usage of a long-livedFileSet thatencounters an unbounded stream of files.
Removing a file that does not belong to the set has no effect.
typePos¶
type Posint
Pos is a compact encoding of a source position within a file set.It can be converted into aPosition for a more convenient, but muchlarger, representation.
The Pos value for a given file is a number in the range [base, base+size],where base and size are specified when a file is added to the file set.The difference between a Pos value and the corresponding file basecorresponds to the byte offset of that position (represented by the Pos value)from the beginning of the file. Thus, the file base offset is the Pos valuerepresenting the first byte in the file.
To create the Pos value for a specific source offset (measured in bytes),first add the respective file to the current file set usingFileSet.AddFileand then callFile.Pos(offset) for that file. Given a Pos value pfor a specific file set fset, the correspondingPosition value isobtained by calling fset.Position(p).
Pos values can be compared directly with the usual comparison operators:If two Pos values p and q are in the same file, comparing p and q isequivalent to comparing the respective source file offsets. If p and qare in different files, p < q is true if the file implied by p was addedto the respective file set before the file implied by q.
typePosition¶
type Position struct {Filenamestring// filename, if anyOffsetint// offset, starting at 0Lineint// line number, starting at 1Columnint// column number, starting at 1 (byte count)}Position describes an arbitrary source positionincluding the file, line, and column location.A Position is valid if the line number is > 0.
func (Position)String¶
String returns a string in one of several forms:
file:line:column valid position with file namefile:line valid position with file name but no column (column == 0)line:column valid position without file nameline valid position without file name and no column (column == 0)file invalid position with file name- invalid position without file name
typeToken¶
type Tokenint
Token is the set of lexical tokens of the Go programming language.
const (// Special tokensILLEGALToken =iotaEOFCOMMENT// Identifiers and basic type literals// (these tokens stand for classes of literals)IDENT// mainINT// 12345FLOAT// 123.45IMAG// 123.45iCHAR// 'a'STRING// "abc"// Operators and delimitersADD// +SUB// -MUL// *QUO// /REM// %AND// &OR// |XOR// ^SHL// <<SHR// >>AND_NOT// &^ADD_ASSIGN// +=SUB_ASSIGN// -=MUL_ASSIGN// *=QUO_ASSIGN// /=REM_ASSIGN// %=AND_ASSIGN// &=OR_ASSIGN// |=XOR_ASSIGN// ^=SHL_ASSIGN// <<=SHR_ASSIGN// >>=AND_NOT_ASSIGN// &^=LAND// &&LOR// ||ARROW// <-INC// ++DEC// --EQL// ==LSS// <GTR// >ASSIGN// =NOT// !NEQ// !=LEQ// <=GEQ// >=DEFINE// :=ELLIPSIS// ...LPAREN// (LBRACK// [LBRACE// {COMMA// ,PERIOD// .RPAREN// )RBRACK// ]RBRACE// }SEMICOLON// ;COLON// :// KeywordsBREAKCASECHANCONSTCONTINUEDEFAULTDEFERELSEFALLTHROUGHFORFUNCGOGOTOIFIMPORTINTERFACEMAPPACKAGERANGERETURNSELECTSTRUCTSWITCHTYPEVAR// additional tokens, handled in an ad-hoc mannerTILDE)
The list of tokens.
func (Token)IsKeyword¶
IsKeyword returns true for tokens corresponding to keywords;it returns false otherwise.
func (Token)IsLiteral¶
IsLiteral returns true for tokens corresponding to identifiersand basic type literals; it returns false otherwise.
func (Token)IsOperator¶
IsOperator returns true for tokens corresponding to operators anddelimiters; it returns false otherwise.
func (Token)Precedence¶
Precedence returns the operator precedence of the binaryoperator op. If op is not a binary operator, the resultis LowestPrecedence.
func (Token)String¶
String returns the string corresponding to the token tok.For operators, delimiters, and keywords the string is the actualtoken character sequence (e.g., for the tokenADD, the string is"+"). For all other tokens the string corresponds to the tokenconstant name (e.g. for the tokenIDENT, the string is "IDENT").
Source Files