Geeks With Blogs

News

Google My Blog

Catch me at: The List!


My InstallScript Utility Belt My Amazon Wishlist
My Standard Disclaimer


Archives
Chris G. Williams Beware: I mix tech and personal interests here.

The players…

I have 2 files, File1.txt is 9.6mb, File2.txt is 42k.  Both are just regular text files with no special characters other than the usual \r\n at the end of each line. I also have a Win Form with 2 buttons that have the code below attached.

 

The puzzle…

Button1 reads in File1 via a StreamReader and works perfectly.
Button2 reads in File2 via a StreamReader and does not work.

 

The code…

private void button1_Click(object sender, EventArgs e)
{
    SortedDictionary<string, string>[] SD;

    StreamReader sr = new StreamReader("file1.txt", System.Text.Encoding.UTF8);
    StringBuilder sb = new StringBuilder();
    while (!sr.EndOfStream)
    {
        char[] buffer = new char[10 * 1024 * 1024];
        int xx = sr.ReadBlock(buffer, 0, buffer.Length);
        sb.Append(buffer);
    }

    SD = Parser.ParseFile1(sb.ToString());

    //do something with it
    --snip--

}

private void button2_Click(object sender, EventArgs e)
{
    SortedDictionary<string, string>[] SD;

    StreamReader sr = new StreamReader("file2.txt", System.Text.Encoding.UTF8);
    StringBuilder sb = new StringBuilder();
    while (!sr.EndOfStream)
    {
        char[] buffer = new char[10 * 1024 * 1024];
        int xx = sr.ReadBlock(buffer, 0, buffer.Length);
        sb.Append(buffer);
    }

    SD = Parser.ParseFile2(sb.ToString());

    //do something with it
    --snip--

}

 

If I click Button1, the code runs as expected. The file is read in, dumped into the StringBuilder and then into the ParseFile1 routine as a string. Life is good.

If I click Button2, the file appears to be read in ok, I can examine the contents of the StringBuilder and the entire file is there, but once I hit the ToString() method on the StringBuilder and try to examine the contents of the string, I get a “serious error in the debugger” with no other information, and have to restart Visual Studio.

What’s the problem?

 

The answer…

I did eventually figure it out.  This one wasn’t obvious (to me, at least) so I’m hoping it stumps at least a few of you.

Select the text to read the answer.

I’m not sure why, but the buffer size for the StreamReader seems to matter. For File2, the buffer was way too big at 10MB (10 * 1024 * 1024) so I changed it to 100K (100 * 1024) and that did the trick.

Now, can someone tell me why this matters?

 

 

UPDATE:

After getting a fair amount of feedback on this post, and some suggestions of different code to try, I’m happy to say the “problem” is no longer a problem, and the code is MUCH cleaner as a result.

Instead of:

    SortedDictionary<string, string>[] SD;

    StreamReader sr = new StreamReader("file2.txt", System.Text.Encoding.UTF8);
    StringBuilder sb = new StringBuilder();
    while (!sr.EndOfStream)
    {
        char[] buffer = new char[10 * 1024 * 1024];
        int xx = sr.ReadBlock(buffer, 0, buffer.Length);
        sb.Append(buffer);
    }

    SD = Parser.ParseFile2(sb.ToString());

I now have:

    SortedDictionary<string, string>[] SD = Parser.ParseFile2(File.ReadAllText("file2.txt", Encoding.UTF8));

No more memory errors, nice and clean. Thanks guys!!  (I’m a little embarrassed to admit that I didn’t know File.ReadAllText existed. I would have certainly started with that, had I known.)

Posted on Wednesday, April 29, 2009 6:54 PM | Back to top


Comments on this post: Odd StreamReader/StringBuilder behavior…

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
Maybe you should be looking at the number of bytes read (with ReadBlock) rather than appending the exact buffer size to the end of StringBuilder. To avoid the random junk at the end.

Actually, there are quite a few things "wrong" with the code.
Left by joe on Apr 29, 2009 8:12 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
Chris,

Delete all that code and replace with:

SD = Parser.ParseFileX(File.ReadAllText(filename, Encoding.UTF8);

All that code makes my eyes hurt :-)
Left by Bill on Apr 29, 2009 8:17 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
Not to be pick, but "ParseFile" is kind of a bad name for a method that parses strings, so maybe move that one liner down into the parser class and have both a real ParseFile (as well as a ParseString).

SD = Parser.ParseFile(filename, encoding);

PS: Reguarding the "Button1 reads in File 1 via stream reader and works perfectly" comment in your original post - better check on that! It looks to me like if "9.6Mb" is not the *exact* byte multiple of 1024*1024*10, then the "string" built in the string builder will have gobbly-gook at the end, and the code works by accident (I would guess) because the Parser can handle a bit of random junk at the end.

Cheers!
Left by Bill on Apr 29, 2009 8:23 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
"pick" was meant to be written as "picky". Whew! Didn't want anyone to think I meant "prick".

:-)
Left by Bill on Apr 29, 2009 8:30 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
ParseFile isn't the actual name of the routine. Names were changed to be more generic, but that's irrelevant. :)
Left by Chris G. Williams on Apr 29, 2009 8:43 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
I realize the code could be tighter, but I'm not terribly worried about that at the moment. I'm much more interested in why ToString() blows up...

Left by Chris G. Williams on Apr 29, 2009 9:00 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
Justin: yes, but not relevant in this example. This is just a snippet to illustrate a concept. I'm more interested in why ToString() would blow up when there is extra space at the end.
Left by Chris G. Williams on Apr 29, 2009 11:13 PM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
I think when you get the error depends a little on the size of your files, but I managed to get it to hang on the very first call to Parse.ParseFile1 (I made this method simply write the string to the Console).

I fixed it by changing the Append line to the following as 'joe' suggested:

sb.Append(buffer, 0, xx);
Left by Sam Judson on Apr 30, 2009 3:29 AM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
Chris,

Your answer is in my posts. And Joe was the first to point it out. [The "PS" explains why even the case you stated "worked" really was not working.]

I assumed you knew what joe meant. I think the term "maybe" in hist post had a little sarcasm in it, because it is obviously not a "maybe". The size of the receive buffer is not the number of bytes read unless you are not done reading the file. Odds are, you will almost never have a file exactly the same number of bytes as the allocated buffer.

My one-liner was not intended to "clean up the code", but rather to "fix the bugs in the code". Point being that if you don't type in so much code, you won't type in so many bugs! (File.ReadAllText(), for example, does not have the bug - it also properly calls dispose(), as Justin pointed out).

I apoligize for being misinterpreted, it was my own fault (typing in the evening).

Let's move to more interesting questions!

Q. Why are we reading a structured text file and building an array of sorted Dictionary of simple objects (strings)?

Sound like we are trading one complicated data structure (file) for another (array of dictionaries), and later code will simply parse the dictionaries instead of the file. Reminds me of the old DataSet object - trade one data structure (database) for another (dataset). What have you gained?

Is this a settings file, like an "ini", with grouped sections of key/value pairs? And the code is building an array (groups) with the key/values? If so, there are much more interesting ideas to talk about.
Left by Bill on Apr 30, 2009 5:36 AM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...
Bill, Sam, Joe, Justin:

Thanks so much for the feedback. The good news is I swapped out the StreamReader/StringBuilder code with the single line of code using File.ReadAllText (how the heck did I miss that?!?) and it works great.

As for the question about why I'm returning a complex String Dictionary, that has more to do with the request of the person consuming the final result. The code in this example was just so I could test the parser before handing it over. He wanted the parse results as a String Dictionary, so that's what he gets. :)

In regards to the type of file I'm parsing, without going into too much detail, it's a rather loosely formatted dump of data from a device. The parser digs through the file, line by line and looks for certain tokens and grabs the value that goes with them, sticking token and value as strings in the dictionary. 1 file can contain multiple sets of data, which is why I end up returning a collection of dictionaries. That's about all I can say on that. :)

Thanks so much for everyone's help and feedback. The code (which I will update in the post) is better as a result!
Left by Chris G. Williams on Apr 30, 2009 8:41 AM

# re: Odd StreamReader/StringBuilder behavior…
Requesting Gravatar...

I realize the code could be tighter, but I'm not terribly worried about that at the moment. I'm much more interested in why ToString() blows up...

Poker
Left by hatun site on Jul 25, 2009 3:41 PM

Your comment:
 (will show your gravatar)


Copyright © Chris G. Williams | Powered by: GeeksWithBlogs.net