23-1

Lesson 23….. StringTokenizer Class Consider the String, “The quick brown fox jumped over the lazy dogs.” Suppose we wanted to take this sentence apart, word-by-word, and then test or process each word? Right now it’s a little hard to see what might be the application of doing something like that or why it might be practical. Rest assured there are plenty of applications (for example, virus detection, email filtering, etc); however, those apps are a bit advanced for us right now. Instead, let’s look at another, more practical String for us: “128 65 1 586 108 79222” What we have above are several numbers, all separated by spaces. We might also have the same list in which commas separate the numbers as follows: “128,65,1,586,108,79222” Tokens: In either case, what can we do to separate the individual words of a sentence …or as in the last example, the individual numbers? We call the individual words or numbers, tokens. The characters that separate them (spaces, commas, etc), are called delimiters. A class that lets us choose delimiters and then automatically separates out the tokens is called the StringTokenizer class. StringTokenizer methods: Let’s take a look at the various methods available to us from the StringTokenizer class (we must import java.util.*). For each method, we will give a description and then the signature of the method. These are not static methods, so we must first show how to create an object: String theString = “128,65,1,586,108,79222”; StringTokenizer t = new StringTokenizer(theString, “, \n”); //comma, space, & \n //are delimiters countTokens( ) Returns the number of tokens in the specified String that remain to be processed. public int countTokens( ) //Signature Example: System.out.println ( t.countTokens( ) ); //6 //See next example for more on countTokens nextToken( ) Returns the next token in the specified string. Gives an exception (error) if no more tokens. public String nextToken( ) //Signature Example: String x = t.nextToken( ); //x is now equal to “128” String y = t.nextToken( ); //y is now equal to “65” System.out.println ( t.countTokens( ) ); //4 remain to be processed

23-2 Also, see the example for hasMoreTokens( ) below for how to prevent nextToken from throwing an exception. nextToken(delim) Establishes a new set of delimiters via String delim and then returns the next resulting token. Gives an exception (error) if no more tokens. public String nextToken(String delim) //Signature Example: (Assume the code in the previous examples has already executed) String m = t.nextToken(“ ,8\n”); //space, comma, 8, and \n specified as new //delimiters. Returns a 1. System.out.println( t.nextToken( ) ); //5… as a result of 8 also acting as a //delimiter. hasMoreTokens( ) Returns either a true or false depending on the presence of more unprocessed tokens. public boolean hasMoreTokens( ) //Signature Example: (Assume the code in the previous examples has already executed) if ( t.hasMoreTokens( ) ) System.out.println( t.nextToken( ) ); //6 Note that the above if statement is equivalent to: if ( t.countTokens( ) > 0 ) System.out.println( t.nextToken( ) ); //6 Constructors: There are two constructors that you should be aware of when creating StringTokenizer objects. Each type is illustrated below: One parameter: StringTokenizer stok = new StringTokenizer(myString); There is only one parameter, myString (the String we desire to tokenize). The delimiters are by default: “ \t\n\r\f” i.e., the space, tab, new line, carriage-return, and form-feed characters. Two parameters: StringTokenizer stok = new StringTokenizer(myString, delimString); Again, myString is the String we desire to tokenize. String delimString is a list of desired delimiter characters. For example, if we desire a space, a plus sign, and the letter “p” as delimiters, then set delimString = “ + p”. You should be aware that another way to tokenize Strings is by using the split method of the String class. See Lesson 18, Appendix J, and Appendix AC for more on the split method.

23-3

Exercise on Lesson 23 1. Create a StringTokenizer object called st. We wish to tokenize String zulu and specify only a plus sign as a delimiter.

2. What are the “things” called that separate the “words” within a String that is to be tokenized?

3. What are the individual parts or “words” called in a String that is to be tokenized?

4. What is the import we need in order to get the StringTokenizer to work?

5. What is the output of the following code? StringTokenizer t = new StringTokenizer(“Hello there good buddy”); String m = t.nextToken( ); System.out.println(m + “>>>” + t.countTokens( ) + “ tokens left.”);

6. Rewrite the following if statement using countTokens( ) rather than hasMoreTokens( ). if ( jj.hasMoreTokens( ) ) { …. }

7. What is the output of the following code? StringTokenizer g = new StringTokenizer(“Rumplestillskin”, “me”); System.out.println( g.nextToken( ) ); System.out.println( g.nextToken(“s”) );

8. Write a class called SpecialToken that has a static method called thirdToken. This method should return as a String, the third token of a String that you pass as a parameter. You may assume that spaces will serve as delimiters.

9. Which constructor for the StringTokenizer class would be simplest to use if you wanted spaces and tabs as delimiters? In problems 10 – 13 state what’s printed. Use the following code and assume for each question that the code in the previous questions has been executed. StringTokenizer gt = new StringTokenizer(“Humpty Dumpty”, “ pu\n\t”);

10. System.out.println(gt.countTokens( ));

23-4 11. String radString = gt.nextToken( ); System.out.println( gt.nextToken( ) + radString );

12. System.out.println( gt.countTokens( ) );

13. What should replace ??? below in order to insure that we don’t get an exception? while ( ??? ) { System.out.println( gt.nextToken( ) ); }

14. What is output by the following code? StringTokenizer tux = new StringTokenizer(“Ignoramus”); System.out.println( tux.countTokens( ) ); System.out.println( tux.nextToken( ) ); System.out.println( tux.nextToken( ) );

23-5

Project… Military Censor You are in the Army and have been assigned the task of censoring soldiers’ outgoing mail for security reasons. Let’s assume that the troops all know about an upcoming offensive that will involve an assault on the Hermes bridge that crosses the Muddy River. Develop an algorithm that uses the StringTokenizer to examine each word of outgoing email. If any of the following words are found, print the word REJECTED. If none are found, then print OK. Taboo words are: Hermes, bridge, Muddy, River, assault, and offensive

Call your class Censor and use the following sentences for testing. “I hope I survive the assault tomorrow.” “I want to talk to you about Bobby, but we’ll cross that bridge later.” “Tell sis and Larry that I’ll be Ok and I will see them in 6 months” “Your last letter was a little muddy on exactly what you meant.” “I see no point in us trying to take the hermes crossing.”

Notice the last sentence uses “hermes” instead of “Hermes”. Your code should not be sensitive to case and should reject this sentence. You should input these sentences via the keyboard using the Scanner class. Your output screen should look like the following after testing all the sentences: Enter next sentence: I hope I survive the assault tomorrow. I hope I survive the assault tomorrow.>>>REJECTED Enter next sentence: I want to ask about Bobby, but we'll cross that bridge later. I want to ask about Bobby, but we'll cross that bridge later.>>>REJECTED Enter next sentence: Tell sis and Larry that I'll be ok and I will see them in 6 months. Tell sis and Larry that I'll be ok and I will see them in 6 months.>>>OK Enter next sentence: Your last letter was a little muddy on exactly what you meant. Your last letter was a little muddy on exactly what you meant.>>>REJECTED Enter next sentence: I see no point in us trying to take the hermes crossing. I see no point in us trying to take the hermes crossing.>>>REJECTED

23-6

StringTokenizer… Contest Type Problems 1. What is output? A. B. C. D. E.

54593 54+593 54+5=93 59=93 None of these

import java.util.*; public class MyTester { public static void main(String args[]){ int m=3, n=4, p=5; String s="5;4+5=9;3"; StringTokenizer st=new StringTokenizer(s,";"); while(st.hasMoreTokens( )) { System.out.print(st.nextToken( )); } } }

public static int total(String str) { 2. What should be passed as a StringTokenizer t = new StringTokenizer(str); parameter to the total method in order int sum = 5; for it to return a 17? while(t.hasMoreTokens( )) { A. “11” sum = sum+Integer.parseInt(t.nextToken( )); B. “3 8 0” } C. “7\n2 1\n1” return sum+1; D. “15 –1 –2 –1” } E. All of the above 3. What is output? A. B. C. D. E.

100.4 101.4 101.5 102.5 None of these

import java.util.*; public class MyTester { public static void main(String args[]){ double [] md = {100.3, 100.4, 100.5, 100.6}; int k=0; String b = "0 1 2"; StringTokenizer st=new StringTokenizer(b); while(st.hasMoreTokens( )) { String str = new String(st.nextToken( )); //System.out.println(str); double val = Double.parseDouble(str); ++k; md[k]+=val; } System.out.print(md[1]); } }

23-7 4. What is output? A. B. C. D. E.

6 8 2 Throws exception None of these

5. What is output? A. B. C. D. E.

Fourscoreandsevenyearsago Four-scoreandsevenyears Fourscoreandsevenyears Fourscore and seven years ago None of these

6. How would the output change if the for-loop was changed to for(int j=1;j