Lance's TextBox

Zipping data from/to memory

Recently, several people have asked me the same question:  How do I zip from a memory stream to a memory stream?

With the Zip component that comes in IPWorks Zip, you can zip from any kind of file or stream to any kind of stream or file.  There is one trick to going from memory stream to memory stream though - and that is keeping the stream open after compressing.

By default, the component will automatically close an input stream after it compresses from it.  But we added a config setting that you can use to turn this off and manually close the stream yourself after you're really done with it (after the extraction or whatever other plans you have for the stream after compression).

The compression:

   1:  MemoryStream file1 = new MemoryStream(GetBytes("This is test 1"));

   2:  MemoryStream file2 = new MemoryStream(GetBytes("This is test 2"));

   3:  MemoryStream myStream = new MemoryStream();

   4:   

   5:  Zip zip = new Zip();

   6:  zip.SetArchiveOutputStream(myStream);

   7:  zip.Files.Add(new ZIPFile("test.txt", file1InputStream));

   8:  zip.Files.Add(new ZIPFile("test2.txt", file2InputStream));

   9:  //config the component to keep the stream open

  10:  zip.Config("CloseStreamAfterCompress=false");

  11:  zip.Compress();

Now I can get onto whatever business I have with the compressed stream.  If I want to extract an already compressed stream, its just as easy:

   1:  //make sure I'm at the beginning of the compressed data:

   2:  myStream.Seek(0, SeekOrigin.Begin);

   3:   

   4:  zip = new Zip();

   5:  zip.ExtractToPath = "..\\myextractedfiles\\";

   6:  zip.SetArchiveInputStream(myStream);

   7:  zip.ExtractAll();

   8:  myStream.Close();

Hope this helps!

posted @ Thursday, May 22, 2008 4:04 PM | Feedback (0) | Filed Under [ Programming IPWorks ]

Stuff White People Like

I don't often link to blogs just for the sake of it, but this one is so funny that I just can't help it.  I hear he just got a book deal too.  And yesterdays post was hilarious.  Stuff White People Like.

Now Playing: O.A.R. - Heard The World (Album Version)

posted @ Thursday, May 22, 2008 10:16 AM | Feedback (0) | Filed Under [ General Just For Fun ]

South Mountains State Park

A couple weeks ago I had the pleasure of going on a camping trip with some friends:  Diana, Korby, and Blake.  Destination:  South Mountains State Park.  It was a blast.  My favorite part actually turned out to be what I was afraid of. 

We left on Friday after work, which meant we would arrive at the campsite late.  After taking our sweet time getting there (even driving around searching for the Russel Stover outlet on the way) we made it about 10 minutes before the gates to the park closed (whoops, that was close)!  Then we discovered that all the car camping places were already taken, and we'd have to hike out to a campsite.  Ouch!  That's exactly what I was afraid of!  We had no backpacks, and it was dark and we only had one headlamp!  We ended up strapping whatever we could fit on our backs and biking in the dark to the campsite about 2 miles away.  Once we got there it was awesome, until we realized that we forgot to bring something to make fire with!  :)  Fortunately there was one other group not too far away and they gave us some fire.

Blake showed us how to make "boy scout burgers" - a ton of hamburger meat and onion and spices smashed down inside aluminum foil and thrown directly in the fire.  Those were great.  The next night we made s'mores as well, which I hadn't done since I was a little kid in the boy scouts.

We went mountain biking and hiking all over.  The biking was amazing - it was incredibly difficult going uphill (I did a lot of walking and pushing my bike uphill!) and it was incredible SCARY going downhill!  A couple of times I thought for sure I was going to go flying over my handle bars and tumbling down a mountain side!  The highlight of the hiking was High Shoals Falls.  It was at the top of this 80 foot waterfall that I realized my brand new phone was designed by a moron.

posted @ Tuesday, May 20, 2008 3:11 AM | Feedback (0) | Filed Under [ General Just For Fun ]

Remixing data from Google Docs & Spreadsheets

Right after Google Spreadsheets came out, I got a ton of downloads from a little class library I wrote for interacting with it over HTTP.  Problem is, Google didn't provide a true API - the solution was to use HTTP to manually perform the gets and posts required to manage each spreadsheet.  Now its much easier thanks to RSSBus!

Here is a sample Google spreadsheet, it looks like this:

Google Docs already allows me to create an RSS feed of this document, but its not very smart.  Its only useful in a feed reader - I can't use this feed in an application without a lot of pain.  But, if I want to create a rich feed of this document, I can do this using the GsheetOps RSSBus Connector (note:  use the RSSBus Excel Operations to work with Excel Spreadsheets).  Here are some examples of things I can do with this Google Spreadsheet and RSSBus:

• Generate a simple RSS feed of the data contained in the spreadsheet
• Sort the data in the spreadsheet how I like it
• Pipe the data into other applications like Excel, Access, a database, etc.
• Pipe the data to Internet services like Amazon S3, FTP/mail/etc servers, Salesforce.com, etc.

Generate a simple RSS feed of the data

The gsheetQuery operation (shown in the image above) allows me to provide my Google email and password (or optional SID that can be generated once with gsheetAuth to avoid repeated authentications) and point to a spreadsheet by specifying its name.  Then, clicking "call operation" shows me a preview of the RSS feed that will be generated by this call, which contains a series of items that look like this:

gsheet:date
2007-07-24T18:56:27.447Z

gsheet:edituri
http://spreadsheets.google.com/feeds/list/o03623.5323249/od6/private/full/

gsheet:hours
10

gsheet:ipm
0.0033

gsheet:items
2

gsheet:name
Bingley

The above example shows only one item, which has 6 elements, all with the gsheet prefix.  Now clicking the "create feed" will generate an .rsb file which will make this feed accessible over HTTP.   Optionally, I can modify this .rsb file to customize my feed to my liking.  Here's what the .rsb looks like by default:

<rsb:info title="gsheetQuery" description="A feed of data from a Google Spreadsheet.">
  <input name="name" description="The spreadsheet name. If left empty the edituri is required" default="myuploadedsheet" /> 
  <input name="sheetname" description="The worksheet name. If left empty the edituri is required" default="sheet1" /> 
</rsb:info>

<rsb:set attr="email" value="youremail@gmail.com"/>
<rsb:set attr="password" value="yourpassword"/>

<rsb:call op="gsheetQuery" output="out">
  <rsb:push title="[rss:title | def('untitled item')]">
    [out.* | tohtml()]
    <hr>(auto-generated by RSSBus Feed Wizard - www.rssbus.com)
  </rsb:push>
</rsb:call>

The .rsb file itself is very simple.  The first part, the rsb:info section, is just simple markup that describes what the feed is and what inputs it takes (in this case, just a spreadsheet name and a worksheet name).  Next, the rsb:set keyword is used to set some constant values - a gmail email and password that the gsheetQuery operation can use to authenticate.  Finally the rsb:call keyword is used to call the gsheetQuery operation, and all of the output item attributes (out.*) are pushed out (rsb:push) as as html.

You can use this script with your own Google Docs account just by modifying the spreadsheet name, worksheet name, email, and password specified in each script.

Sort the Data

Now I can pipe this gsheetQuery call into a feedSort call, and get a feed sorted by whichever column I like.  To do this, I can use the rsb:pipe keyword to pipe the output of gsheetQuery into the feedSort operation, like this:

<rsb:info title="gsheetQuery" description="A feed of data from a Google Spreadsheet.">
  <input name="name" description="The spreadsheet name. If left empty the edituri is required" default="myuploadedsheet" /> 
  <input name="sheetname" description="The worksheet name. If left empty the edituri is required" default="sheet1" /> 
</rsb:info>

<rsb:set attr="email" value="youremail@gmail.com"/>
<rsb:set attr="password" value="yourpassword"/>

<rsb:pipe>
  <rsb:call op="gsheetQuery" output="out" />
  <rsb:call op="feedSort?sort=gsheet:hours&type=numeric" />
</rsb:pipe>

Now I get a feed of items sorted numerically by the number of hours.  The rsb:pipe keyword is very handy, and I can continue the pipe as long as I want.  I could add another call after the feedSort to pipe its output to an operation to send an email, post a blog entry, send a paypal or credit card payment, create a row in an Excel spreadsheet or a database, or lots of other things.  For a full list of available RSSBus operations check out the connector list at http://www.rssbus.com.

More

Note the other operations included in GsheetOps:

gsheetAddRow - Insert/add a new row in a Google Worksheet.

gsheetAuth - Authenticate to Google Spreadsheets service.

gsheetDeleteRow - Delete an existing row from a Google Worksheet.

gsheetEditCell - Edit a particular cell value in a Google Worksheet.

gsheetGetRange - Get a specified range from a Google Worksheet.

gsheetListSpreadsheets - List the Spreadsheets in a Google Spreadsheets account.

gsheetListWorksheets - List the Worksheets in a Google Spreadsheet.

posted @ Thursday, May 15, 2008 12:34 PM | Feedback (0) | Filed Under [ RSSBus ]

FXP - FTP transfers from server to server

Every so often I get a question from someone who wants to transfer files between two FTP servers.  In order to do so, they usually have to connect to server 1, download the files, and then connect to server 2 and upload the files.  The process could go much sooner if they could eliminate the extra transfers and just copy the files directly from server 1 to server 2.  It turns out this is possible, and is in fact mentioned in the FTP spec (RFC 959), but most people don't realize it.  Also most servers don't allow it by default, for security reasons.

FXP, aka site-to-site transfer, aka server-to-server transfer is really just a sneaky trick of the FTP protocol itself.  Here's how FXP works:

1. Create a connection to FTP server 1, and another connection to FTP server 2. 
2. On FTP server 1, enter passive mode (data connections will be incoming), and make a note of the servers response to the PASV command (what ip the server will be listening on, and on what port). 
3. On server 2, send a PORT command (data connections will be outgoing), such that the port command data corresponds to the data in FTP server 1's reply to the PASV command gathered in step 2. 
4. Finally, send the STOR command to ftp server 1, and the RETR command to FTP server 2.  The data connection will be made between those two servers, and when the transfer is complete the client will be notified by

Here's how you can do this with two instances of the IPWorks FTP component:

   1:      public void FXP()

   2:      {

   3:        //FXP, aka site-to-site or server-to-server transfers

   4:        //not commonly supported by FTP servers by default, for

   5:        //security reasons

   6:   

   7:        string[] portdata = null;

   8:        Ftp ftpA = new Ftp();

   9:        ftpA.Timeout = 5;

  10:        ftpA.OnPITrail += new Ftp.OnPITrailHandler(delegate(object sender, FtpPITrailEventArgs e)

  11:        {

  12:          if (e.Message.StartsWith("227")) //response to PASV

  13:          {

  14:            string portparams = e.Message.Substring(e.Message.IndexOf("(") + 1);

  15:            portparams = portparams.Remove(portparams.IndexOf(")"));

  16:            portdata = portparams.Split(new char[] { ',' });

  17:          }

  18:        });

  19:        ftpA.RemoteHost = "serverA";

  20:        ftpA.User = "test";

  21:        ftpA.Password = "password";

  22:        ftpA.Passive = true;

  23:        ftpA.Logon();

  24:        ftpA.Command = "PASV";

  25:        //if this reply was 200 OK, then the server will allow the site to site transfer.

  26:   

  27:        //now we know the port number that server A is listening on (portparams),

  28:        //so start a retriever on server B:

  29:        ThreadPool.QueueUserWorkItem(new WaitCallback(FTPRetriever), portdata);

  30:   

  31:        //start the sender on server A:

  32:        ftpA.Command = "STOR uploaded.txt\r\n"; //the file to create on server 1

  33:   

  34:        //done..disconnect

  35:        ftpA.Logoff();

  36:      }

  37:   

  38:      static void FTPRetriever(object parameter)

  39:      {

  40:        string[] portdata = (string[])parameter;

  41:        Ftp ftpB = new Ftp();

  42:        ftpB.Timeout = 5;

  43:        ftpB.RemoteHost = "serverB";

  44:        ftpB.User = "test";

  45:        ftpB.Password = "mypassword";

  46:        ftpB.Passive = false;

  47:        ftpB.Logon();

  48:        ftpB.RemotePath = "destinationfolder/";

  49:        ftpB.ListDirectory();

  50:   

  51:        ftpB.Command = "PORT " + portdata[0] + "," + portdata[1] + "," + portdata[2] + "," + 

  52:                                     portdata[3] + "," + portdata[4] + "," + portdata[5];

  53:        ftpB.Command = "RETR uploaded.txt"; //the file to get from server A

  54:        ftpB.Logoff();

  55:      }

There you have it, a pretty straight forward process.  ftpA connects to server A and sends the command needed to create an incoming data transfer connection and start sending a file called uploaded.txt.  ftpB connects to server B and sends the commands needed to create an outgoing data transfer connection (to server A) and receive the file uploaded.txt.

posted @ Thursday, May 15, 2008 12:02 PM | Feedback (0) | Filed Under [ Programming IPWorks ]

Software for Rock Star Wanna-be's

posted @ Thursday, May 15, 2008 12:01 PM | Feedback (0) | Filed Under [ Software General ]

RSSBus: Simple Ways to Connect Data

I just uploaded a new YouTube video for  RSSBus.  Some of it is hard to see, but soon it will be published at rssbus.com as a high quality flash video.

RSSBus can be used to securely serve custom feeds over the Internet or on the local host.  This video shows examples of very basic feeds that can be created with RSSBus with just a few mouse clicks, and more complex piped feeds.  Near the end, the video shows some ways in which RSSBus can be integrated into real world solutions.  Obviously, RSSBus can make your data accessible through anything that supports RSS or Atom feeds, but it can also make your data accessible to many other productive environments and applications like Excel, software development languages (.Net, ActiveX, Delphi, C++ Builder, VC++, Python, etc), Windows PowerShell, Windows WorkFlow, etc.

Enough of that:  here's the video, yo:

posted @ Tuesday, May 06, 2008 2:57 PM | Feedback (0) | Filed Under [ Programming Software RSSBus ]