A very strange problem with SSIS data dump into a pipe delimited file

[Update: 2010-07-14]

Don't know what difference does one day make, but I re-tried this today and the file on the network looks fine.  I know I'm not crazy because I tried the same steps yesterday twice and got screenshots in the email to prove it.  At this point, I'm willing to take anyone's theory.

 

[Original post]

I ran into this problem and have yet to solve it.

We needed to create a data dump pipe delimited flat file out of a table of 4 million+ rows.  The output was created just fine on my local machine via SSIS.  However, when the file is copied over to a network share location (whether it is in zipped format or not), the {CR}{LF} got stripped for every record following right around the 1 millionth line.  Basically all the records following that mark became a single line, while all records before that still retain their line breaks.

Has anybody seen this?  I can only isolate this down to a network issue, but that does not really make sense to me.  I thought if anything it is a problem with my text editor, but I can open the copy on my local drive and see the entire file formatted properly.  I am a little perplexed to say the least.

The current workaround is to output one file for every 900k records and send over the network in zipped format.  We are going to try FTP and see if that solves our problem.  Nevertheless, I would really like to know whether there is anything I can do on my network settings to solve it, or whether this is even a network issue to begin with.

Print | posted on Wednesday, July 14, 2010 11:34 AM

Feedback

No comments posted yet.

Your comment:





 

Copyright © Kevin Shyr

Design by Bartosz Brzezinski

Design by Phil Haack Based On A Design By Bartosz Brzezinski