Package mars.venus.editor.jeditsyntax.tokenmarker

Class Token

java.lang.Object
mars.venus.editor.jeditsyntax.tokenmarker.Token
public class Token extends Object
A linked list of tokens. Each token has three fields - a token identifier, which is a byte value that can be looked up in the array returned by SyntaxDocument.getColors() to get a color value, a length value which is the length of the token in the text, and a pointer to the next token in the list.
Version:
$Id: Token.java,v 1.12 1999/12/13 03:40:30 sp Exp $
Author:
Slava Pestov

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final byte
    CHAR_LITERAL
    Literal 2 token id.
    static final byte
    COMMENT
    Comment 1 token id.
    static final byte
    COMMENT2
    Comment 2 token id.
    static final byte
    DIRECTIVE
    Keyword 2 token id.
    static final byte
    END
    The token type, that along with a length of 0 marks the end of the token list.
    byte
    id
    The id of this token.
    static final byte
    ID_COUNT
    The total number of defined token ids.
    static final byte
    INSTRUCTION
    Keyword 1 token id.
    static final byte
    INTERNAL_FIRST
    The first id that can be used for internal state in a token marker.
    static final byte
    INTERNAL_LAST
    The last id that can be used for internal state in a token marker.
    static final byte
    INVALID
    Invalid token id.
    static final byte
    LABEL
    Label token id.
    int
    length
    The length of this token.
    static final byte
    MACRO_ARGUMENT
    Macro parameter token.
    Token
    next
    The next token in the linked list.
    static final byte
    NULL
    Normal text token id.
    static final byte
    OPERATOR
    Operator token id.
    static final byte
    REGISTER
    Keyword 3 token id.
    static final byte
    STRING_LITERAL
    Literal 1 token id.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Token(int length, byte id)
    Creates a new token.

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    toString()
    Returns a string representation of this token.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • NULL

      public static final byte NULL
      Normal text token id. This should be used to mark normal text.
      See Also:

    • COMMENT

      public static final byte COMMENT
      Comment 1 token id. This can be used to mark a comment.
      See Also:

    • COMMENT2

      public static final byte COMMENT2
      Comment 2 token id. This can be used to mark a comment.
      See Also:

    • STRING_LITERAL

      public static final byte STRING_LITERAL
      Literal 1 token id. This can be used to mark a string literal (eg, C mode uses this to mark "..." literals)
      See Also:

    • CHAR_LITERAL

      public static final byte CHAR_LITERAL
      Literal 2 token id. This can be used to mark an object literal (eg, Java mode uses this to mark true, false, etc)
      See Also:

    • LABEL

      public static final byte LABEL
      Label token id. This can be used to mark labels (eg, C mode uses this to mark ...: sequences)
      See Also:

    • INSTRUCTION

      public static final byte INSTRUCTION
      Keyword 1 token id. This can be used to mark a keyword. This should be used for general language constructs.
      See Also:

    • DIRECTIVE

      public static final byte DIRECTIVE
      Keyword 2 token id. This can be used to mark a keyword. This should be used for preprocessor commands, or variables.
      See Also:

    • REGISTER

      public static final byte REGISTER
      Keyword 3 token id. This can be used to mark a keyword. This should be used for data types.
      See Also:

    • OPERATOR

      public static final byte OPERATOR
      Operator token id. This can be used to mark an operator. (eg, SQL mode marks +, -, etc with this token type)
      See Also:

    • INVALID

      public static final byte INVALID
      Invalid token id. This can be used to mark invalid or incomplete tokens, so the user can easily spot syntax errors.
      See Also:

    • MACRO_ARGUMENT

      public static final byte MACRO_ARGUMENT
      Macro parameter token. Added for MARS 4.3.
      See Also:

    • ID_COUNT

      public static final byte ID_COUNT
      The total number of defined token ids.
      See Also:

    • INTERNAL_FIRST

      public static final byte INTERNAL_FIRST
      The first id that can be used for internal state in a token marker.
      See Also:

    • INTERNAL_LAST

      public static final byte INTERNAL_LAST
      The last id that can be used for internal state in a token marker.
      See Also:

    • END

      public static final byte END
      The token type, that along with a length of 0 marks the end of the token list.
      See Also:

    • length

      public int length
      The length of this token.

    • id

      public byte id
      The id of this token.

    • next

      public Token next
      The next token in the linked list.

  • Constructor Details

    • Token

      public Token(int length, byte id)
      Creates a new token.
      Parameters:
      length - The length of the token
      id - The id of the token

  • Method Details

    • toString

      public String toString()
      Returns a string representation of this token.
      Overrides:
      toString in class Object