Geeks With Blogs
SOA & Integration Services - BizTalk, WCF, WF, AppFabric etc Why is it drug addicts and computer afficionados are both called users?

I got got an email from Chirag in Nigeria with the following problem. He had a positional flat file which he needed to parse which looked like this (the ` marks the carrage return and like feed, EOF is the end of file marker remove these to test the file):

0120200300 01                                      THORNTON                                                           `
0120200400 01                                      OGIHARA                                                            `
0120200500 01                                      WILSON                                                             `
0080300101 JENNIFERMRS                                                        `
0080300201 BENJAMINMSTR                                                       EOF

There are two different schemas or 'Sections' of data here and they are identified by the first 3 digits at the start of each line 012 and 008. Cirag had written a parser in C# but he was concerned it was too slow and this was not the "BizTalk" way to do things.

1) Create a schema that looks like the one shown below:

 


2) Set the properties on the following nodes. The key properties which make this solution work are the Parser Optimisation, Lookahead Depth for performance the Lookahead Depth must be set to allow the parser to identify the first 3 characters of each record i.e. 012 or 008. This works in turn with the Tag Identifier property which matches different records with different areas in the schema i.e. the 012 records should be parsed into the K20 parent node area and the 008 into the K30 parent node area, this is quite a powerful piece of functionality. The most common use for the Tag Identifier property I have seen is with correctly parsing header and footer records into a schema.

Note you will need the BizTalk 2004 SP1 installed to set the Parser Optimisation, Lookahead Depth and Allow Early Termination properties in the schema properties dialog or you will need to open the schema and set these using notepad or some other editor.

Node                   Property Name                        Property Value
Schema                 Schema Editor Extension              Flat File Extension 
                       Parser Optimisation                  Speed
                       Lookahead Depth                      3
                       Allow Early Termination              No

 

AirlineMessage         Structure                            Delimited
                       Child Delimiter Type                 Hexadecimal
                       Child Delimiter                      0x0D 0x0A
                       Child Order                          Infix

 

K20                    Structure                            Positional
                       Tag Identifier                       012
                       Tag Offset                           0
                       Max Occurs                           *
                       Min Occurs                           0

PLen                   Positional Length                    4

Key2                   Positional Length                    2

Key3                   Positional Length                    2

ChangedFlag            Positional Length                    1

PartyNo                Positional Length                    2

Remark                 Positional Length                    32

SplName                Positional Length                    6

SurName                Positional Length                    67

 

K30                    Structure                            Positional
                       Tag Identifier                       008
                       Tag Offset                           0
                       Max Occurs                           *
                       Min Occurs                           0

PLen                   Positional Length                    4

Key2                   Positional Length                    2

Key3                   Positional Length                    2

ChangedFlag            Positional Length                    1

Name                   Positional Length                    67

 

3) Now you've got the schema lets test it there are two easy ways to test the schema either:
a) Right click the schema in the solution explorer choose properties set the entries in the dialog to the following

Now Right click the schema in the solution explorer again and select validate instance. If all goes ok the the output window should display something like the following:

Invoking component...
Validation generated XML output <file:///C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\_SchemaData\TestElement_output.xml
>.
Validate Instance succeeded for schema TestElement.xsd, file: <file:///C:\Project\Artifacts\Biztalk
Projects\UnuniformPositionalFlatFile\Test.pos>.
Component invocation succeeded.
 
 
Double click on the Validation generated XML output link and the xml output result of the file will be  displayed. Then
click in the pane in which the XML is displayed hold down the Ctrl key and press K and D this will format the result xml correctly so you can view it.

b) Call the the Flat File Dissasembly utility FFDASM.exe. the easiest way to do this is to create a bat file with the contents like the following:

c:
cd C:\Program Files\Microsoft BizTalk Server 2004\SDK\Utilities\PipelineTools
ffdasm.exe "C:\path\inputFlatFileName.pos" -bs "C:\path\schemaName.xsd" -c -v
pause..

The output should look like:

 <AirlineMessage xmlns="http://UnuniformPositionalFlatFile.Test">
   <K20 PLen="0120" Key2="20" Key3="02" ChangedFlag="0" PartyNo="0" Remark="01" SplName="" SurName="THORNTON" xmlns="" />
   <K20 PLen="0120" Key2="20" Key3="04" ChangedFlag="0" PartyNo="0" Remark="01" SplName="" SurName="OGIHARA" xmlns="" />
   <K20 PLen="0120" Key2="20" Key3="05" ChangedFlag="0" PartyNo="0" Remark="01" SplName="" SurName="WILSON" xmlns="" />
   <K30 PLen="0080" Key2="30" Key3="01" ChangedFlag="0" Name="JENNIFERMRS" xmlns="" />
   <K30 PLen="0080" Key2="30" Key3="02" ChangedFlag="0" Name="BENJAMINMSTR" xmlns="" />
 <AirlineMessage>
 
 R. Addis
 
 
Posted on Tuesday, May 31, 2005 12:06 AM BizTalk | Back to top


Comments on this post: Another BizTalk 2004 Positional Flat File Schema Example

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi,

Great article. Keep up the good work.

Anyhow, I've got a map from an XML to a positional file.

the map works fine when passing a single record but when the XML contains
multiple records it comes up with the following error:

===============================================
Native serialization error: Cannot find definition for the input: InsNumber
===============================================

The schema stracture is as follows:
===============================================
Schema
Root Record
Middle Record
Sub Record(s)
===============================================

and the properties are as follows:

===============================================
Schema
Schema Editor Extensions = Flat File Extension

Root Record
Child Delimiter = 0x0D 0x0A 0x2E 0x0A
Child Delimiter Type = Hex
Child Order = Infix
Structure = Delimited

Middle Record
Child Delimiter = 0x0D 0x0A
Child Delimiter Type = Hex
Child Order = Infix
Structure = Delimited
Group Order Type = Sequence
Max Occurs = unbounded
Min Occurs = 0
Taq Identifier = G

Sub Record(s)
Structure = Positional
Group Order Type = Sequence
Max Occurs = 1
Min Occurs = 0
Taq Identifier = G
===============================================

Note: InsNumber is the first fields of all the records and sub records.

Any help or comments would be very much appreciated. It's quite urgent so
please help....

Kamal
kali@bca-group.com
Left by Kamal on Jun 01, 2005 2:48 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Oki, I've added a Looping Functoid between the Middle Record of the source schema (XML) and the Middle Record of the Destination schema (Positional flat file) and that generated a very strange result.

---------------------------
2186 01...
2186 02...
2186 03....
2186 04...
5423 04....
5423 04 .....
2186 05
2186 05
2186 05
2186 05
2186 05
.....

--------------------------

so the records are not displayed in the correct order. so you get some of the
records and nodes of the record (2186) and some of the other records such as
(5423) and some of the records are missing as well..

I really need some help.

I hope I'm explained this clearly.

I'd really appreciate your assistance.

Thank you
Left by Kam on Jun 01, 2005 4:18 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi Kamal,
What I need from you is:
1) The xml input
2) The whole postional schema
3) The correct native positional file output i.e. what you want the result to be

For others wanting my help I am busy and need at least the above to solve your BizTalk flat file \ mapping problems. Post your email as well. The last condition is that if I solve you problems then I get to post the solution to this Blog so other's can see etc.
Left by Rob on Jun 03, 2005 4:32 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi,

We've the below flat file.

ABCDEF 123 888
11122233399 111222333
11122233399 111222333
11122233399 111222333
11122233399 111222333

We need to convert it in the below XML format.

</Notes>
<Header>
<Field1>ABCDEF</Field1>
<Field2>123</Field2>
<Field3>888</Field3>
</Header>
<Record Code="99">11122233399 111222333</Record>
<Record Code="99">11122233399 111222333</Record>
<Record Code="99">11122233399 111222333</Record>
<Record Code="99">11122233399 111222333</Record>
</Notes>

Only issue we face is we need to seperate the 11th and 12th positions and have it in Code attribute.

Yours
Jinishans
Left by JiniShans on Jun 28, 2005 1:11 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Sorry the root not should start as <Notes>. I've given as </Notes>
Left by Jinishans on Jun 28, 2005 1:12 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi Jinishans, works been flat out this week I'll have a look at your issue next week. Thanks for sending it in!
Left by Rob on Jun 30, 2005 10:37 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
If your flat file remains the same always thn u can use my custom receive pipeline component available on gotdotnet.
If it changes always then u just need to modify my component.My component converts your input to string and thn u can do anything with the string.
Regards
Nishil
Left by NISHIL on Sep 29, 2005 2:24 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
mail me at nishil@rediffmail.com if u need any help.I may not visit these blogs at times.
Regards
Nishil
Left by Nishil on Sep 29, 2005 2:26 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
This example work fine but when i use a flat file whith mixed order like folowing :
0120200300 01 ...
0120200400 01 ...
0080300101 JENNIFERMRS ...
0120200400 01 ...
0080300101 JENNIFERMRS ...

Thas not work and i have this error message :
There was a failure executing the receive pipeline: "PPRMTL_Schemas.MTLReceivePipeline" Source: "Flat file disassembler" Receive Location: "C:\AcredBiztalkSolution\PPRFlows\PPR_MTL\MTLIn\*.txt" Reason: Unexpected data found while looking for:
'149'
'249'
The current definition being parsed is MTL. The stream offset where the error occured is 228. The line number where the error occured is 3. The column where the error occured is 0.

Can i have some help plz??
Left by Arrad Tarik on Jan 03, 2006 12:32 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
This example work fine but when i use a flat file whith mixed order like folowing :
0120200300 01 ...
0120200400 01 ...
0080300101 JENNIFERMRS ...
0120200400 01 ...
0080300101 JENNIFERMRS ...

Thas not work and i have this error message :
There was a failure executing the receive pipeline: "PPRMTL_Schemas.MTLReceivePipeline" Source: "Flat file disassembler" Receive Location: "C:\AcredBiztalkSolution\PPRFlows\PPR_MTL\MTLIn\*.txt" Reason: Unexpected data found while looking for:
'012'
'008'
The current definition being parsed is MTL. The stream offset where the error occured is 228. The line number where the error occured is 4. The column where the error occured is 0.

Can i have some help plz??
Left by Arrad Tarik on Jan 03, 2006 2:46 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hey Arrad,

The problem is that the AirlineMessage record element has it's Group Order Type set to "Sequence" it therefore expects 012 records before 008 records change the AirlineMessage record elements Group Order Type Property to "Choice", it's Group Max Occurs to Unbounded and Group Min Occurs to 0.

Good question
Cheers
Rob
Left by Rob A on Jan 15, 2006 6:49 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
I have a positional flat file with multiple records but no delimiters, not even end of
line. I get this error

C:\BillInquiryResponseIN.txt Unexpected end of stream while looking for:
Positional data (length is 32)
Positional data (length is 1)
The current definition being parsed is BillPmtStatus. The stream offset where the error occured is 9161.

Does this have any meaning? Can someone help without me sending the files?
(I don't know where to send the files, or do you want me to paste here?)

thanks
awni
Left by awni on Jan 18, 2006 9:54 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Paste the files here.
Left by Rob A on Jan 18, 2006 6:43 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Thanks Rob i will make changes and i'll be back to you.
Left by Arrad Tarik on Jan 21, 2006 9:53 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi Rob, i have change MTL record elements Group Order Type Property to "Choice" and it's Group Max Occurs to Unbounded and Group Min Occurs to 0, but it stil give me an the following error :
There was a failure executing the receive pipeline: "PPRMTL_Schemas.MTLReceivePipeline" Source: "Flat file disassembler" Receive Location: "C:\AcredBiztalkSolution\PPRFlows\PPR_MTL\MTLInBound\*.txt" Reason: Unexpected data found while looking for:
'1'
The current definition being parsed is MTL. The stream offset where the error occured is 228. The line number where the error occured is 3. The column where the error occured is 0.
My inbound flatefile is like that :
1496033010051380MADANI AIDA 200508002939460004899100284147800000000000979820732084
2496026010211596AZZAOUI ABDELKRIM 2005080008160000013600000816000000000000000000I0420235
2496036010669799BOUSSEFIANE MHAMMED 2005080069660000116100005688900000000000000000I0312692
3496026010120279MOUSSAOUI ABDELHAQ 2005080080340000133900003800100000000000133900R0326701
3496030010120732OUAKIB KADDOUR 2005080126590940416499007125695000000000416499R0733158
1496033010051380MADANI AIDA 200508002939460004899100284147800000000000979820732084
2496036010669799BOUSSEFIANE MHAMMED 2005080069660000116100005688900000000000000000I0312692

NB: The Lookahead Depth property is set to : 0 and Allow Early Termination to : Yes

Thanks for help
Left by Arrad Tarik on Jan 23, 2006 9:05 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
The Parser optimisation is set to : Complexity and Suppress Empty Nodes to : Yes

Thanks
Left by Arrad Tarik on Jan 23, 2006 9:07 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
I have a similar flatfile like Arrad Tarik and gives the same error as his. Does anyone have the solution? Would be a great help. Thanks in advance.
Left by Sara on Mar 01, 2006 5:27 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi Arrad\Sara,
I am freeing up a bit now between projects. Arrad your schema looks different, Sara I don't have your schema or test data. I'm interested in having a go at fixing the problem but you will need to paste the schema and the test data (input file) here.
Thanks
Rob
Left by Rob A on Mar 01, 2006 6:55 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Hi Arrad\Sara,

Check weather your tag identifier is contained(other than the tag identifier position you specfied) in the data.

set the pad character for the record elements.

chandra
Left by Chandra on Apr 25, 2006 6:25 AM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Did anyone looked at servingxml.sourceforge.net
Left by hari on Apr 08, 2007 10:34 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Please help...
Child Delimiter property is not appearing in Visual Studio node proeties window.....I am using BTS 2006.
Left by Dinesh on Aug 11, 2007 6:59 PM

# re: Another BizTalk 2004 Positional Flat File Schema Example
Requesting Gravatar...
Are you using the flat file extension?
Left by Rob on Aug 11, 2007 11:43 PM

# XML to excel
Requesting Gravatar...
Hi,

I'm quite new in all this BizTalk stuff and I need some help. I want to transform an xml file in excel file, using BizTalk.

Some ideas?


thank you

oana
Left by oana on Sep 03, 2007 9:43 PM

Your comment:
 (will show your gravatar)


Copyright © Rob Addis | Powered by: GeeksWithBlogs.net