With this example we are going to demonstrate how to tokenize a java source file.
In short, to tokenize a java source file you should:

• Create a newFileReader.
• Create a newStreamTokenizer that parses the given FileReader.
• UseparseNumbers() API method of StreamTokenizer that specifies that numbers should be parsed by this tokenizer.
• UsewordChars(int low, int hi) API method that specifies that all characters c in the range low <= c <= high are word constituents.
• UseeolIsSignificant(boolean flag) method that determines whether or not ends of line are treated as tokens.
• UseordinaryChars(int low, int hi) that specifies that all characters c in the range low <= c <= high are “ordinary” in this tokenizer.
• UseslashSlashComments(boolean flag) method that determines whether or not the tokenizer recognizes C++-style comments.
• UseslashStarComments(boolean flag) API method that determines whether or not the tokenizer recognizes C-style comments.
• Iterate over the tokens of the tokenizer and for every token of the tokenizer, and check if it a String, the end of a line, a number, a word or something else,
• Close the fileReader using itsclose() API method.

Let’s take a look at the code snippet that follows:

package com.javacodegeeks.snippets.core;import java.io.FileReader;import java.io.StreamTokenizer;public class Main {    public static void main(String[] argv) throws Exception {    FileReader fileReader = new FileReader("C:/Users/nikos7/Desktop/Main.java");  StreamTokenizer tokenizer = new StreamTokenizer(fileReader);  tokenizer.parseNumbers();  tokenizer.wordChars('_', '_');  tokenizer.eolIsSignificant(true);  tokenizer.ordinaryChars(0, ' ');  tokenizer.slashSlashComments(true);  tokenizer.slashStarComments(true);  int tok = tokenizer.nextToken();  while (tok != StreamTokenizer.TT_EOF) {tok = tokenizer.nextToken();switch (tok) {    case StreamTokenizer.TT_NUMBER:  double n = tokenizer.nval;  System.out.println(n);  break;    case StreamTokenizer.TT_WORD:  String word = tokenizer.sval;  System.out.println(word);  break;    case '"':  String doublequote = tokenizer.sval;  System.out.println(doublequote);  break;    case ''':  String singlequote = tokenizer.sval;  System.out.println(singlequote);  break;    case StreamTokenizer.TT_EOL:  break;    case StreamTokenizer.TT_EOF:  break;    default:  char character = (char) tokenizer.ttype;  System.out.println(character);  break;}  }  fileReader.close();    }}

Output:

ch = (char) tokenizer.ttype;

 
This was an example of how to tokenize a java source file in Java.

Do you want to know how to develop your skillset to become aJava Rockstar?

Subscribe to our newsletter to start Rockingright now!

To get you started we give you our best selling eBooks forFREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to theTerms andPrivacy Policy

Sign up

Thank you!

We will contact you soon.