My first thought – who needs another language? Are new languages the new rage?

If I were to design a new language (I am not qualified, trust me), what would it address . . .

• It would be garbage collected (no brainer)
• It would run in a VM (actually, many different VMs )
• It would also be statically typed (I think many would disagree these days)
• Threading would be a focus (with message passing as a language feature). Certainly include some aspects of functional programming languages
• It would have first class functions to throw around
• Some sort of cool, built-in data types – if only to set it apart from other languages.
• Nearly all functionality in libraries, and try to get others to actually write the libraries

Sounds a bit like Google Go - “Expressive, concurrent, garbage collected” they say. Fast, safe, concurrent, fun, open source they continue. Fun and fast? I don’t think “compile time” is an issue and nearly everything these days, interpreted or not, runs “nearly as quickly as C or C++ code”. So that’s nothing new. All programming is fun. (Well, nearly all).

My language would not have a googly-goofy name, like “Go” with all the “Go nuts”, “Go for it”, etc., phrases being thrown around. I do like the co-routines moniker, err, I mean Go-routines, however. At least they did not call it “goo” or “goog”. In fact, without the goroutines I probably would not have given this thing a second look.

So what is new with this language?

A Systems Programming Language

I think that title is a clue as to what this all might be about – I would love to see C/C++ disappear, but I would like to see that done through evolution of those languages. Something like removing all the “bad goo” from C/C++, but leaving the good stuff alone. Maybe that is impossible – we can only pile on, not take away. A colleague of mine once said that C/C++ would always be around because someone has to write the thread scheduler and the device drivers for the OS – I disagree, eventually we will jettison what is bad about C and keep what’s not. Is that GO? I don’t know.

Why does it have pointers? Who in their right mind would add that as a language feature? It has no pointer arithmetic – so maybe this is OK, then? Slices – they seem to allow you to address arbitrary blobs of bytes (defined via a pointer, maybe) or portions of arrays, as the documentation states, in a semi-safe manner without the pointer arithmetic. Sounds reasonable for a systems programming language. I wonder, then, why are strings immutable?

Things I like

• Maps, slices, and channels
• Go-Routines
• No implicit numeric conversions
• Lots of implicit “compatible conversions” (my terminology)
• No side-effects or alternatives syntaxes for things
• No operator overloading

Things Not-so-hot

• No exception handling
• No generics
• Pointers
• No method overloading

No type inheritance? I’ll have to think about that one for a while. Sounds very interesting, though, since a significant amount of refactoring I seem to do involves unwinding complex type hierarchies into composition models. There is also implicitly-inherited (my terminology) types – i.e., when one type just happens to implement a set of methods that matches another type or interface. They are then magically, implicitly compatible. That is cool. Very cool.

I plan to spend some time over the next few months looking at Google Go. There are lots of very smart people over in google-land, so I am sure there is a lot to consider and learn from the ideas around Go. Too bad I’ll have to fire up linux or actually boot the Macbook into OSX instead of Windows 7 to play with it. Stupid decision on google’s part, if you ask me.

One of the things I have always hated was “Windows Search”. Why the heck would I need “instant results” – instant as in donating countless CPU cycles and disk churns to background indexing operations running all day long on my box? YUK! I am never actually “looking” for something – I know where everything is. It’s right where I put it. Who are these people “looking” for things on their computers all the time?

Typical Scenario

My typical use case came up again this morning: I downloaded a source distribution from Google Code, and decided to check the files into my local SCM system (sidebar: this project is not taking any patches). Before doing this, I would like to delete all of the hidden “.svn” folders that came down with the distro. “Search for all folders named .svn”. Now keep in mind that I am not actually looking for the folders, as I know where they are (there is one folder named .svn in each of the folders in the distribution). I simply want to find them so I can control-A (select all) and press the delete key. You see what I mean.

Trouble with Windows Search

I always have trouble with Windows Search. I start typing “.svn” and immediately my file display window changes and I get a bunch of wrong answers – files with the letters “svn” in them (dot being ignored), sometimes files with “svn” inside of them, as well as virtually all of the files inside the folders, since many contain .svn extensions. Great. Hundreds of results when I was expecting around ten. What happened to the good old days when “find files” actually did what I wanted!

Read the Manual

Read the manual. For the first time, I decided to read the manual. I found a page at microsoft.com with the search syntax. For my scenario:

kind:folder .svn

The dot “.” is still ignored, and you get all folders – not just folders with “svn” in the name. Hmm. A little more digging. . .

kind:folder name:.svn

Now I am getting just the folders named “.svn”, but I am still getting folders named “svn” (no dot), as well as any folder with “svn” in the name. Hmm.

kind:folder name: “.svn”

That did it. The dot is still being ignored, despite the quotes, but it is not a wildcard in some sort of pseudo-regular expression. The documentation says that * and ? are the wildcard characters, taking on their typical meanings. There’s plenty of syntax for AND, OR, greater or less than (for things like dates and sizes), and virtually any property you can think of – in or out of the file(s).

Maybe I’ll spend some more time looking over the documentation. Maybe not. At least I can now find a bunch of folders to delete. The next time I’m looking for a MSWord document containing the phrase “place contract”, the word “lease”, and was edited within the last 3 days, created by “mydomain\joe”, and last saved by “otherdomain\mike” – I’ll know how to do it. Thanks, Microsoft.

Visual Studio 2010 beta2 + Framework 4.0

. . . with a few comments on Windows Virtual PC.

Installation

I don’t “work” on my laptop – mostly used for travel, library, watching Hulu and channel 9, etc. Since I will be out of town for a while, however, I decided to install VS20101b2 in a winVPC for the trip on the laptop. The only complaint about the new VPC offering from Microsoft is that when it is running, it takes forever for my notebook to initiate sleep mode. I don’t recall the old product having this issue.

Installation of 2010 was a breeze, especially Team Foundation Server (holy crap is that a different experience!). The new icon needs work – looks very bad in the new Windows 7 taskbar with the default theme.

First Impressions

The first thing I did was fire up the VM and connect to TFS server – all running on the laptop – in a 1GB VM, I might add. I create a new team project (‘TestProject’). This all worked. I seem to remember several days before I was able to accomplish this on tfs2005, and at least a few hours on tfs2008. Today’s elapsed time is a couple of minutes (for the TFS part).

Overall, I found a snappy IDE that has a finished feel to it. Nice. March release should be no problem.

Branching and Merging

I setup the TestProject with a typical branching and merging strategy (dev/main/prod), added a few work items and a bug, and got to work.

[ Note: I like the little ‘Save’ icon – saves a picture of the visualization] There seem to be lots of tools accessible from this UI, such as branch compare tools. Overall, this visualizer is a nice feature (the simple picture shown does not do justice). I’m still looking for the merge visualizations (update: found them!). I’d like to see merge arrows that are labeled per the changeset description and clickable to show history/changeset details for merges. It looks like I’ve got some exploring to do.

Team Web Access

I was not expecting this. Even in the Basic product running on a local client OS. Nice touch. It’s a beautiful source, work item, project browser web application. Who needs SharePoint?

New in Framework

I’ve already blogged a few times about changes in the Framework 4.0 beta. A few more in this release:

• DLR library
• Several more threading primitives
• System.Collections.Concurrent namespace
• MEF improvements
• In-process side-x-side CLR

Static Reflection in .Net

Where have I been?

I recently took a few minutes to look through the latest MSDN magazine and ran across a nice article by Jeremy Miller on “Functional Programming for Everyday .NET Development”. The section titled “Lambdas as Data” was my first introduction to static reflection in .Net. Where the heck have I been? I don’t write database persistence code, so I am not familiar with NHibernate (or fluent NHibernate) (which is where Jeremy’s sample comes from), but I am still surprised at how little I understand about everything Linq brought to the .Net party. A quick search around the internet reveals this topic is well-covered, and several years old.

Example Code

Suppose you have a need to dynamically discover properties on an object. Easy, right? After all, that’s what the System.Reflection namespaces is for! However..

Let’s look at a class named “SomeObject” with a property “SomeProperty”:

class SomeObject

{

    public string SomeProperty { get; set; }

}

You can use so-called “static reflection” like this:

   1: class Program

   2: {

   3:     static void Main(string[] args)

   4:     {

   5:         SomeObject o = new SomeObject()

   6:         {

   7:             SomeProperty = "Test"

   8:         };

   9:         staticReflection(o, e => e.SomeProperty);

  10:         Console.ReadKey();

  11:     }

  12:     static void staticReflection<T>(T o, Expression<Func<T,object>> expression)

  13:     {

  14:         MemberExpression me = expression.Body as MemberExpression;

  15:         if (me != null && me.Member.MemberType == System.Reflection.MemberTypes.Property)

  16:         {

  17:             Console.WriteLine("Property: '{0}' of type '{1}' has value '{2}'.",

  18:                 me.Member.Name,

  19:                 expression.Body.Type,

  20:                 expression.Compile().Invoke(o)

  21:                 );

  22:         }

  23:     }

  24: }

The output is:

Property: 'SomeProperty' of type 'System.String' has value 'Test'.

Very cool. Now I need to understand why the second argument has to be something like Expression<Func<T, object>> rather than simply Expression<T>. If the latter is used, the compiler rewards me with:

Error    1    Cannot convert lambda to an expression tree whose type argument 'StaticReflection.SomeObject' is not a delegate type

It’s the whole “return type” thing on the delegate that throws me as well – where does that come from? In my code sample I have declared the Func return type as type “object” (it is actually a string in my case). Why don’t I have to do something like “e => {return e.SomeProperty;}”, and how come a lambda expression with a statement body like that cannot be converted into an expression tree (compiler’s words, not mine)?

I still have a little more learning to do before really understanding what’s going on here. My suspicion is that Linq brought a heck of a lot more to the party than it might appear at first look.

Designing Software for Scientists

Creating a software design for scientists can be tricky business. Unlike the businessman, who is better off looking at the pretty database report or marveling over the catchy name of the latest Forms over Data application, the scientist can actually contribute to the implementation – provided you use the right architecture. In year’s past (I’m referring to 20+ years ago), the typical Physicist / Chemist / etc. was nearly on par with the typical software engineer / computer scientists with respect to software design skills. Although the scientific community of that time preferred the ‘GOTO’, they nevertheless understood the value of using subroutines to nearly the same degree as a computer scientists.

Not so, today. Software design has come a long way since then.

Software Design has Matured

Software design has matured. We have terms like inheritance and encapsulation. We talk about singletons, abstract factories, and chain of responsibility. We use terms like polymorphism, covariance, and contravariance (as applied to software design). We compose systems with Inversion of Control containers. Mock objects are used for unit testing. Reflection, dynamic and static languages. The discipline of software engineering has matured to a point where the typical scientist cannot possibly hope to understand what makes a well-design system – no more than a software engineer can understand how to model protein folding of polypeptides.

So how do we design software so that other scientists and engineers can participate in the implementation? Here are some ideas.

1. Keep it Simple.

Don’t compose in an IoC container – too abstract. Do use layers, repositories, and multiple program units. Scientists understand process control, state transitions, and other engineering/scientific/mathematics concepts so use them in your software designs. [Object models? Not sure where I stand on this one. There might be a whole blog post in here somewhere.]

2. Work in Pairs

Match a subject matter expert with a software design expert.

3. Limit Participation

Limit the “coding” performed by the scientist to the functional code, not the design. The infrastructure, layout, program flow as well as all meta-design elements (project structure, build environment, coding standards, etc.) – essentially the software design, should be strictly the realm of the software architect. Leave the algorithms for the subject matter expert.

4. Move the problem domain.

Let the scientists program in their own “language” - provide a scripting component, meta-programming, or data-driven model such that the subject matter expert remains closer to their problem domain, even if this means additional software (such as interpreters, data transformation engines) must be developed. In a really large system, you might go so far as to develop a rules engine or other knowledge base approach.

5. Use Abstractions

The nature of software is to provide abstractions. This also happens to be common of all science and engineering. You can use your entire software design Bag ‘o Tricks provided you supply an appropriate level of abstraction over top. Explain the abstraction, and the typical scientist will have no trouble using it, even if they really don’t know what is going on. So in spite of the first idea listed here, you can fire up your favorite IoC container and use it – as long as you provide the right abstraction for your scientific colleagues.

Three’s a Charm

It looks like the third iteration of Team Foundation Server might be a good one. I just finished reading Brian Harry’s post on TFS 2010 and it contains some interesting references.

SourceSafe Replacement?

I never could understand why TFS was often compared to SourceSafe. All the sessions on migrating from VSS to TFS, TFS for VSS users, etc. Is a team really going to migrate from VSS to TFS? Really? You have got to be kidding me! TFS is change management (to borrow a phrase from Rational), and VSS was barely version control.

… but now I think I get it …

All those VSS (or CVS, etc.) users were left stranded – there was no lightweight version control server for individuals (or very small teams). So they all went to SVN. Quick setup, version control, painful (but possible) branching and merging, command line, nice shell client, poor VS client, and all of that.

VSTS 2010

In steps Visual Studio Team System, Team Foundation Server, 2010.

• Can run locally on Windows 7 and Windows Vista client OS’
• Simple installation / Configuration
• Can optional use SQL Server express edition
• Can be installed on a domain controller
• Stand-alone version includes Work Item Tracking, Version Control

Note that this stand alone version also appears to include the build server/services!

I assume with the new project collections feature, you could move your SQL Express database over to a real SQL Server when your team gets a bit bigger. And good pricing is also on the Redmond minds.

Looks to me like they did their research. And did it well.

Nice.

The Process class in .Net makes it easy to run command line tools from within a .Net program. Here is an implementation with a couple of twists:

• Environment variable expansion
• Optional asynchronous execution
• Transformation of output into an enumerable collection of objects

The implementation takes the form of an object named CommandLineTool with the following constructors:

public CommandLineTool(string command)
    : this(command, String.Empty, Environment.CurrentDirectory, _defaultTransform) { }
public CommandLineTool(string command, string arguments)
    : this(command, arguments, Environment.CurrentDirectory, _defaultTransform) { }
public CommandLineTool(string command, string arguments, string workingDirectory)
    : this(command, arguments, workingDirectory, _defaultTransform) { }
public CommandLineTool(string command, string arguments, string workingDirectory, Func<string, object> transform)
{
    this.Command = Environment.ExpandEnvironmentVariables(command);
    this.Arguments = Environment.ExpandEnvironmentVariables(arguments);
    this.WorkingDirectory = Environment.ExpandEnvironmentVariables(workingDirectory);
    this.Transform = transform;
    _isBusy = false;
}

The simplest form provides for an executable filename only, while the last constructor accepts arguments, working directory, and transformation function for the output lines. Note that environment variables, such as "%ComSpec%, may be included in any of the first three arguments.

Output Transformation

Each line of output (strings) can be sent to a transformation function, defined as a Func<string, object>, and returned as an enumerable collection of objects (IEnumerable<Object>) accessed via the OutputObjects property. Output can also be expressed as a single string (Output) or an array of lines of text (OutputLines). These are exposed as properties. The transformation function is free to return null to exclude an item from the OutputObjects collection.

   1: /// <summary>

   2: /// Command output.

   3: /// </summary>

   4: public string Output { get; private set; }

   5: /// <summary>

   6: /// Command output lines.

   7: /// </summary>

   8: public string [] OutputLines

   9: {

  10:     get

  11:     {

  12:         if (Output != null)

  13:         {

  14:             return Output.Split(_seperators, StringSplitOptions.RemoveEmptyEntries);

  15:         }

  16:         return null;

  17:     }

  18: }

  19: /// <summary>

  20: /// Output Objects.

  21: /// </summary>

  22: /// <remarks>

  23: /// Output lines are converted to output objects using the transform

  24: /// function. The default transform functions results in an enumeration

  25: /// of the out lines (strings).

  26: /// </remarks>

  27: public IEnumerable<Object> OutputObjects

  28: {

  29:     get

  30:     {

  31:         if (Transform != null)

  32:         {

  33:             return OutputLines.Select(Transform).Where(o => o != null);

  34:         }

  35:         return null;

  36:     }

  37: }    

Asynchronous and Synchronous Operation

Both asynchronous and synchronous modes of operation are provided. An event (ExecuteCompleted) is raised following the completion of an asynchronous execution. Overloads are provided for both the Execute() and ExecuteAsync() methods to allow for an alternate transformation function to be applied in successive calls. Only one asynchronous operation at a time is allowed. The asynchronous version simply calls on the synchronous version using a background thread from the thread pool. Error output is provided in a similar manner to standard output, except that there is no option for transformation of lines of text on the error output to objects.

   1: /// <summary>

   2: /// Execute the command.

   3: /// </summary>

   4: /// <param name="transform">Transform to apply to the output.</param>

   5: /// <returns>Command result.</returns>

   6: public int Execute(Func<string, object> transform)

   7: {

   8:     this.Transform = transform;

   9:     return Execute();

  10: }

  11: /// <summary>

  12: /// Executes the command.

  13: /// </summary>

  14: /// <returns>Command result.</returns>

  15: public int Execute()

  16: {

  17:     ProcessStartInfo info = new ProcessStartInfo();

  18:     info.FileName = this.Command;

  19:     info.Arguments = this.Arguments;

  20:     info.WorkingDirectory = this.WorkingDirectory;

  21:     info.CreateNoWindow = true;

  22:     info.UseShellExecute = false;

  23:     info.RedirectStandardError = true;

  24:     info.RedirectStandardOutput = true;

  25:     Process p = Process.Start(info);

  26:     Output = p.StandardOutput.ReadToEnd();

  27:     ErrorOutput = p.StandardError.ReadToEnd();

  28:     return p.ExitCode;

  29: }

  30: /// <summary>

  31: /// Execute the command asynchronously.

  32: /// </summary>

  33: /// <param name="transform">Transform to apply to the output.</param>

  34: public void ExecuteAsync(Func<string, object> transform)

  35: {

  36:     this.Transform = transform;

  37:     ExecuteAsync();

  38: }

  39: /// <summary>

  40: /// Execute the command asynchronously.

  41: /// </summary>

  42: public void ExecuteAsync()

  43: {

  44:     lock (this)

  45:     {

  46:         if (!_isBusy)

  47:         {

  48:             _isBusy = true;

  49:             ThreadPool.QueueUserWorkItem((WaitCallback)((state) =>

  50:             {

  51:                 int result = Execute();

  52:                 if (ExecuteCompleted != null)

  53:                 {

  54:                     ExecuteCompleted(this, EventArgs.Empty);

  55:                 }

  56:                 lock (this)

  57:                 {

  58:                     _isBusy = false;

  59:                 }

  60:             }));

  61:         }

  62:         else

  63:         {

  64:             throw new InvalidOperationException("Async operation in progress.");

  65:         }

  66:     }

  67: }

Using the Code

Using the code is fairly easy. Just create a new CommandLineTool object and call either the Execute() or ExecuteAsync() methods. Here is a snippet from one of the unit tests:

   1: string command = "cmd.exe";

   2: string arguments = "/c dir";

   3: CommandLineTool target = new CommandLineTool(command, arguments);

   4: int expected = 0;

   5: int actual;

   6: actual = target.Execute();

   7: Assert.AreEqual(expected, actual);

A more involved example (somewhat contrived) includes transformation of the results and asynchronous operations. [Results are transformed into FileObject objects].

   1: // The FileObject class ...

   2: internal class FileObject

   3: {

   4:     public DateTime ModifiedDate { get; set; }

   5:     public string Name { get; set; }

   6: }

   7:  

   8: // Code ...

   9:  

  10:     string command = "cmd.exe";

  11:     string arguments = "/c dir /s";

  12:     Func<string, object> transform = new Func<string, object>(s =>

  13:         {

  14:             DateTime modified = DateTime.MaxValue;

  15:             if (s.Length > 39 && DateTime.TryParse(s.Substring(0, 20), out modified))

  16:             {

  17:                 return new FileObject()

  18:                 {

  19:                     ModifiedDate = modified,

  20:                     Name = s.Substring(39)

  21:                 };

  22:             }

  23:             return null;

  24:         });

  25:     CommandLineTool target = new CommandLineTool(command, arguments);

  26:     target.ExecuteAsync(transform);

  27:     while (target.IsBusy) Thread.Sleep(0);

  28:  

  29:     List<FileObject> files = target.OutputObjects.Cast<FileObject>().ToList();

  30:  

  31:     ... etc ... 

Hope this comes in handy for someone. Let me know if you see any errors/problems or room for improvement.

Remember when a program was termed a “GUI Application” - a program with a Graphical User Interface? It had features like menus that “dropped-down” (or “popped up”), scroll bars to reveal more of the virtual surface area, a pointing device (still called a cursor), and lots and lots of Drag & Drop. Thankfully, that moniker has mostly died. Now it is time another term to fall by the wayside – the Web Application.

What is a Web Application?

Just what is a web application, anyway. An application delivered over the web? A Java applet, flash, or silverlight program running in a browser? How about HTML+Javascript with lots of Ajaxy asynchronous updating? What about plain old static HTML pages with hyperlinks to navigate between them?

Who are Web Developers?

Just who are these web developers? Are they PHP junkies, ASP.net gurus, Flex monkeys? Are the real web developers HTML+CSS+JavaScriptors? Rails developers? Web site designers? What about a C++ application that runs locally yet stores data in the cloud or communicates with other instances on the internet? Is the twitter client a web application?

The ubiquitous internet makes every application a web application. The term has lost its meaning, and the term must die.

Just like “GUI”. If an application requires a user interface, it will be a graphical one. If an application requires communication between computers, or presence on more than one computer, it will use the “Web”. In one way or another. Server side, client side, I don’t care. These days, every “application” is a “web application”.

Below are a few thoughts on the upcoming framework 4.0. I am not sure if the feature set is finalized, but I’ll comment on what I think are some of the more interesting features being planned for this release, as seen in the beta.

Managed Extensibility Framework

Plug-in architectures are a fairly common pattern for many scenarios, and I have seen quite a few over the years. MEF seems to be a nice combination of plug-in/IoC container/Dependency Injection framework. It’s use in Visual Studio 2010 should prove to give this framework a leg-up. I’m looking forward to using it as a replacement to the typical roll-your-own plug-in architecture. Not sure if it will be used where a complete dynamically composed extensible application is called for, however, that is exactly the scenario that MEF is designed for. I’ll see how it works out for plug-ins, first, before jumping in with both feet.

Language Enhancements

Some of these are obvious improvements: Optional arguments, COM interop/PIA enhancements. Some are welcome, such as named parameters. As for the new dynamic language features in C# ... well, I’m not qualified to comment on that yet. Interoperability with dynamic languages is a good thing. Whether or not I’ll be writing “dynamic c#” in the near future remains to be seen.

Base Class Library

I’ve blogged about one new class, the Memory Mapped File, in the recent past. There are a host of additional improvements, including the inclusion of the Task Parallel Library (TPL) that I believe came out of Microsoft research and has been release in CTP form a few times. Code Contracts look interesting, but I seriously doubt I will be adding pre-conditions, post-conditions and the like to very much code. I suppose there is a use for this – for some types of code; but I don’t see it for typical use. I had some experience with Eiffel in the past and did not care for it.

More Framework, Less P/Invoke

I would like to see more framework and less P/Invoke, particularly for new technologies such as those introduced in Windows Vista. There are managed wrappers provided for many things from Microsoft, but how about adding Windows 7 stuff to framework 4.0? It’s fine that wrapper libraries appear on CodePlex, but I would like to see the kind of treatment you would get from including new APIs in the framework – the way the System.Net.Sockets namespace is not simply a wrapper around the Win32 sockets API. I know, it’s hard – but what else is windows except a platform for development? If I was looking for merely an OS to run standard applications on, I could look elsewhere. Windows is first and foremost an application development platform for me. And for applications, I prefer managed code. Leave the C code (and COM goo) for drivers and low-level applications like virus scanners.

Visual Studio 2010

This looks like a big change. I’ll probably be using this for R&D (and also at home). But will probably wait about a year (or more) before converting all development projects over to vs2010. Although the beta seems pretty solid, my experience is that the first of a major change (vs2002/2003, the initial vs2008) in the Visual Studio line is not the greatest (contrast to vs6, vs2005 and now 2008/SP1). I love two things: multi-targeting and the ability to run different visual studio versions side x side. I expect that experience to be kept in vs2010. This allows developers to install the new tools with “no fear”, committing to them down the road. Goes a long way to increase adoptability. All that being said, the C++ team is saying “10 is the new 6”, and MSBuild can now be used for C++ code, etc., etc. I’ll keep an eye on thing in 2010.

F#?

Don’t know about this one. I’m thinking that the pile of technologies that I am now “expert” in is getting a little too large. However, I believe in the mantra “learn a new programming language every year”. It can be a bit confusing to switch from maintaining classic C code, early C++ code (mostly C with sprinkles of C++, MFC/ATL), later C++ code (STL, etc.), some VB6 and VBscript in classic ASP or office apps; then perform new development in JavaScript, C#, Windows Forms, WPF, Silverlight, and ASP.net. with a sprinkling of objective C for an iPhone app or two (quite a mix of frameworks and languages in that sentence, but you get the idea). I love it all. I’ll have to work F# in somewhere, because a functional language perspective brings new thought patterns, which is always a good thing. I suppose I’m glad it’s in the box, although at first thought I might want IronRuby or IronPython in there instead. I don’t see any production apps developed in F# in the next year or two, but an R&D project is likely.

-   -    -     -

I have to admit that I have not played all that much with the beta; been pretty busy over the past few months. But I’m looking forward to framework 4.0, and expect it will turn out to be a significant release.

There have been a few famous quotes on the subject – how much memory do you really need; how many GHz is enough; how much storage to hold all the books in the world; how many personal computers are enough for planet earth (let alone a single family), etc., etc. - I’ll stay away from predicting the future and just talk about what I need today.

2 THz processor

I need a 2THz processor. It took me 5 hours (300 minutes) to encode a family video to MP4 (H.264 720p) on a dual-core 2.66GHz machine. It is reasonable to want this to take just a few minutes (around the time it takes to burn an audio CD today) so that I could grab a video, throw some clips together, and encode the clips in an afternoon before dinner.

100 GB of RAM

Remember the days when large digital images could not fit in RAM? I would like my 5GB of video to fit in RAM – maybe two or three of them – comfortably – while editing the video. 100GB should be enough (for today). Speed up disk transfer, network, etc. by 100x or 1000x while we’re at it so we can push all this data around easier.

Petabyte Hard Drive

We need a place to store all these bits. A Petabyte would be a good starting point. While it is true that all of the emails, financial data, and other digital records I have can fit on a $20 8Gb flash card, and the music collections of everyone in the house combined add up to less than 100 Gb, we still have a cabinet full of DVD disks and nearly zero video digitally encoded and stored on the computers. Kind of like our audio data was some years ago.

Final Thoughts

Just sticking with what I need today – right now – I came up with these numbers. But I’m stuck with just a few GHz of cycles, a few Gb of ram, and a few Tb of storage. And I’ll be heating up the computer room for 5 hours at a clip with fans on high and CPU’s pegged at 100%.

We still have a long way to go.

As my home router’s “blocked web sites” list slowly began to fill up with open proxies (kproxy.com, getus.in, vtunnel.com, browse007.com, and anonymouse.org, etc.), I realized that it’s time to stop blocking and start monitoring.

DLink DIR-655 Event Logging

My DLink DIR-655 (great little router) contains an event logging feature that can send events, SYSLOG-style, over the network to a collector.

<110>Mon Aug 10 20:32:30 2009 DIR655 System Log: Web site geekswithblogs.net/ accessed from 192.168.1.91

<110>Mon Aug 10 20:32:30 2009 DIR655 System Log: Web site geekswithblogs.net/ScriptResource.axd?d=afn4d1BEk_dsfd45j-LusKDJDF accessed from 192.168.1.91

<110>Mon Aug 10 20:32:30 2009 DIR655 System Log: Web site geekswithblogs.net/Geekswithblogs.css accessed from 192.168.1.91

<110>Mon Aug 10 20:32:30 2009 DIR655 System Log: Web site geekswithblogs.net/WebResource.axd?d=PKDJF7383jdhksj_jd accessed from 192.168.1.91

<110>Mon Aug 10 20:32:30 2009 DIR655 System Log: Web site geekswithblogs.net/ScriptResource.axd?d=kalwo*hsdh%%jdjdgs90-48JSH accessed from 192.168.1.91

<110>Mon Aug 10 20:32:31 2009 DIR655 System Log: Web site geekswithblogs.net/?_TSM_HiddenField_=ctl00__mainContent_ScriptMana accessed from 192.168.1.91

<110>Mon Aug 10 20:32:31 2009 DIR655 System Log: Web site ads.geekswithblogs.net/a.aspx?ZoneID=1&Task=Get&IFR=False&PageID=51102& accessed from 192.168.1.91

<110>Mon Aug 10 20:32:31 2009 DIR655 System Log: Web site docs.geekswithblogs.net/DiscountASPNET/160x600_Champs_Trophy_6Mo_SM.gif accessed from 192.168.1.91

<110>Mon Aug 10 20:32:31 2009 DIR655 System Log: Web site www.gravatar.com/avatar.php?gravatar_id=4837367e8383a8b38376f accessed from 192.168.1.91

All I have to do is run a little server on the SYSLOG port, collecting messages as they come in and storing them in a database that can later be used as a source for reporting. I use the following regular expression to parse these messages:

private Regex _webSiteRegex = new Regex(@"^<\d{1,3}>(?<date>.+) DIR655 System Log: Web site (?<msg>.+) accessed from (?<source>[A-F0-9\.]+)", RegexOptions.Compiled);

This expression provides me with the date string, the partial URL of the site accessed, and the source of the request. The DLink router indicates the source as either an IP address or a MAC address. Wireless devices, such as an iPod Touch, are usually reported by MAC address while wired computers working off the router’s DHCP server are reported via IP. I decided to store MAC address information in the database, since the IP address for the various devices can change (I use DHCP). A simple ARP request can convert any of the IP address to MAC address on my subnet as needed.

To facilitate reporting, I decided to only store the hostname and domain portions of the URL in the database. My daily report, sent via email, displays the top 10 domains per device along with any domains that I have not seen before.

The Results

Well, the kids spend most of their time on facebook; I spend mine on various programming related sites; there are a fair amount of online shopping sites being hit, and of course there are way too many advertising sites.

Not that I didn’t already know this information.

But it will be nice to spot the occasional new domain that I might want to discuss with the kids. Sure beats the cat-and-mouse game of trying to keep the blocked list up to date, which I was beginning to lose.

I recently needed to create a web service to provide coastal tidal predictions, i.e., the low tide and high tide times for a given day at a given location.

The Web Service

I decided to forego a traditional “web service” and provide a calendar feed (using iCalendar data format) instead. This approach allows devices such as an iPod Touch (or iPhone), as well as most internet-aware calendar programs, to grab and display the data on a calendar – perfectly appropriate for this kind of information. Internet-based calendars, such as google calendar, can also grab and display the feed. The service is designed to provide 30 days worth of data beginning with the current day.

The Data Source

The best source for this type of data in the US is the often overlooked National Oceanic and Atmospheric Administration (NOAA). When it comes to weather, the National Weather Service arm of NOAA provides a slew of web service products for your programs to consume. For tidal data, however, not so lucky. I was stuck with HTTP-page-scraping or FTP file transfers – yuk. Ugly, but doable. A quick peek at the format of the HTML served up by NOAA’s tidal prediction web site reveals that they most likely store data in a simple text file (for ftp-based access), and simply pump this out inside of a HTML element for HTTP access. A year’s worth of predictions is stored in the page. I don’t know when the “next year” is made available, but since I am much more interested in tidal data for the summer months, I’m not concerned with how well the service will perform over the Dec-Jan timeframe. The data is easy to scrape with a regular expression.

Using Regular Expressions

The following regular expression was used to scrap the data from NOAA’s web page:

private static Regex _reg = new Regex(@"((?<date>\d{1,2}/\d{1,2}/\d{4})\s+\w+)?\s+(?<time>\d{1,2}:\d{2}(A|P)M)\s+(LST|LDT)\s+(?<height>[-\+\d\.]+)\s+(?<tide>(L|H))");

The date and time of the predictions, as well as the type (Low or High), and the height (which I don’t use) are captured from the expression. NOAA uses local time, which they indicate with the LST or LDT monikers – I will simply assume all times are local for the given date, which was correct for this year’s data, anyway.

Grabbing the Web Page

It’s quite easy to grab a page via HTTP these days:

WebRequest request = null;

WebResponse response = null;

Stream responseStream = null;

try

{

    request = WebRequest.Create(uriString);

    response = request.GetResponse();

    responseStream = response.GetResponseStream();

    if (responseStream.CanRead)

    {

        using (StreamReader sr = new StreamReader(responseStream))

        {

            string data = sr.ReadToEnd();

            return CreateFromData(data, startDate, endDate);

        }

    }

}

finally

{

    if (responseStream != null) responseStream.Close();

    if (response != null) response.Close();

}

The CreateFromData() method simply walks through the regular expression matches to build a list of NOAATidePrediction objects:

public static NOAATidePredictions CreateFromData(string data, DateTime startDate, DateTime endDate)

{

    NOAATidePredictions result = new NOAATidePredictions();

    DateTime date = DateTime.MinValue;

    foreach (Match m in _reg.Matches(data))

    {

        if (m.Groups["date"].Success)

        {

            date = DateTime.Parse(m.Groups["date"].Value);

        }

        if (date != DateTime.MinValue &&

            date >= startDate &&

            date <= endDate

            )

        {

            TidePrediction.TideType t = TidePrediction.TideType.High;

            if (m.Groups["tide"].Value == "L")

            {

                t = TidePrediction.TideType.Low;

            }

            DateTime time = DateTime.Parse(m.Groups["time"].Value);

            result.Tides.Add(new TidePrediction(date.AddMinutes(time.TimeOfDay.TotalMinutes), t, double.Parse(m.Groups["height"].Value)));

        }

    }

    return result;

}

Not all of the code is shown here, but it is pretty easy to figure out what the NOAATidePredictions and TidePrediction classes, for example, look like.

Converting the TidePrediction objects to iCalendar events is pretty easy as well (also not shown) since we are only creating simple events for each tide prediction.

Serving up the Calendar Feed

I already have an HTTPHandler on my web site that gets invoked when “*.ics” files are requested. A request to “tides.ics”, along with the location information, now serves up the data – courtesy of NOAA – created on the fly from the code shown above.

Not bad for a few hours work. With a quick glance at my iPod Touch, I can view the high/low tide times for the next 30 days on it’s calendar.

I finally installed the Apple iPod Touch 3.0 software update and, to my surprise, my iPod has been greatly improved. I chose the word surprise because Apple has disappointed me (and the rest of the world) twice in the past, and I made the assumption they would disappoint again.

You see, the iPod Touch is a nice little computer, disguised as a music player, with WiFi built-in. You would certainly expect it to be able to:

• Sync music with no wires
• Download audio and video podcasts from the internet (again, no wires)
• Subscribe to internet calendars (again, no wires)
• Sync contacts, email, calendars, etc. with the internet (no wires)

Unfortunately, since the day I purchased this device I have been plugging it into my computer and painfully watching my CPU usage pegged at 100% as iTunes eventually manages to start up and draw a ridiculously poor-performing UI on the screen while my processor fan switches to high speed. Ugh.

What good is an IP address if you’re not going to use it!

Original Firmware (1.0)

On the original iPod touch, firmware 1.0, you could sync with Microsoft Outlook’s calendars and contacts. The inconvenience of getting Outlook involved aside, this feature is of no use to me (despite the fact that I use Outlook daily) since my calendar information in Outlook is pulled from various internet “iCalendar” feeds (as well as a local intranet iCalendar feed, i.e., “the family calendar”). The iPod Touch would not sync with most of that information, even when it was present/pulled into Outlook. The iPod could only sync with Outlook’s “main” calendar (which is blank, in my case). Nice try.

Update Firmware (2.0)

Firmware update 2.0 provided some relief for contacts/mail and also vaguely promised improved calendar sync support – multiple Outlook calendars was even mentioned as a new feature. It also came with a charge – yep, Apple actually wanted money for the firmware update (reminds me why I stopped buying Macs some years back – tired of being nickel and dimed). I installed it on my iPod Touch. The new firmware did work with multiple Outlook calendars, however, it only worked with Outlook’s built-in calendars (or Exchange calendars, I assume) – not with Outlook’s calendar subscriptions and not with feeds from internet sources. I was out of luck again of the calendar front. And still physically plugging the darn iPod into the computer every day.

All the while I’m wondering why Apple would have a device with built-in WI-FI sync with a program like Outlook, anyway. We should be able to sync with internet calendars (iCalendar, Caldav, Google Calendar, etc.), and other internet data.

What good is an IP address if you’re not going to use it!

Latest Firmware (3.0)

In comes the latest software (along with the obligatory nickels and dimes), which promised better calendar support. This time Apple was not so vague, promising things like Caldav and iCalendar subscription support. Accordingly (and to my surprise), this little device with built-in Wi-Fi can finally subscribe to calendar data from one or more URLs, and update the calendar data just by accessing the internet.

It’s about time.

Note: Now, if I can just get our school district to publish an iCalendar based feed of school and sports events instead of being so proud of their new PDF-based calendar generator… but that’s another story.

iTunes Software

I still cannot sync music without physically plugging in the unit and painfully watching the iTunes Software Monster take over my machine, but I no longer have to do this every day! In fact, I only have to do this when there’s a change to my music library. Not so often. That’s probably the best part of the firmware upgrade. Maybe that will make up for one of those nickels (or dimes).

The latest computer arrived today – an Asus EeePC 1005HA, the perfect Netbook

• Intel Atom N280 1.66GHz
• 2Gb RAM / 160Gb HDD / 1024x600
• Wired 100BaseT / Wireless N
• Bluetooth 2.0 (stereo audio)
• Camera, SD slot, microphone
• $400

The Perfect Netbook

The Asus EeePC 1005HA is a beautiful machine. For the same (little lower) price than Dell, Acer, HP, etc., you get the best in network screen resolution (1024x600), reasonable camera, great low-power CPU (hyper-threaded N280), super battery life (8+ hours real) quality HDD, maximum memory supported by the chipset (2Gb), wireless N (not G), and a bluetooth radio that supports my stereo headphones. This allows me to watch Hulu streaming video in [what Hulu claims is] HD quality while running on the treadmill for 8 straight hours with no wires attached to me or the netbook (Full Disclosure: I never really did that). It’s also great for the usual netbook stuff, i.e., “web application” tasks (checking email, IM/Skype, and clicking on stuff in a web browser).

First Boot Experience

The first boot experience was not too bad, particularly when compared to Dell or HP machines. However, Windows XP simply will not do (it looks like a tired old joke these days). After verifying that all of the hardware worked, running some disk checks, running some RAM checks, noting battery performance, CPU/motherboards temperatures, etc., it was time to install a modern operating system on this thing.

Asus ships the machine with several partitions on the drive. The installation media on stored on a secondary (not normally mounted) partition of the drive, and provides a method in the BIOS to “recover” from this media. I wish netbook manufacturers would ship “pure” windows installation media on a USB stick “thrown in the box” instead, and simply forget about the add-on crapware. However, I have to give Asus a little credit – they also partition the drive so that OS is on one partition and data is on another; I suspect that their “recovery process” re-creates the OS partition only, but I will never know. Either way, it is now time to wipe the drive along with Asus’ hard work. :-)

Windows 7 Installation

Note: I am using the RC build of Windows 7

The first step is to build a USB boot disk with Windows 7 32-bit media on board. This is quite easy to do (has been since the Windows Vista days) with the following steps:

1. Obtain a command prompt in Vista (or later) with Administrative rights
   • This is most easily accomplished by clicking the Start Menu, type “cmd”, then right-click “Run as Administrator” on the shortcut.
2. Use diskpart program to prepare a blank USB drive [Warning: doing this to the incorrect drive is, well, …]
   • Purchase a $14 4Gb USB drive, and plug it in
   • run “diskpart” in the administrative console command prompt window
   • type “list disk” to identify your 4Gb (about 4000 MB) disk
   • type “select disk X” where ‘X’ is the disk identified in previous step [obviously important to select the correct disk!]
   • type “clean” to wipe the drive [I hope you selected the correct disk]
   • type “create partition primary” to create a primary partition [if you get here, then you likely got the right disk]
   • type “select partition 1” to select the new partition
   • type “active” to mark as active boot partition
   • type “format fs=fat32” to format the partition
   • type “exit” to quit; unplug and re-plug in the USB drive.
3. Copy installation media contents to the drive
   • Insert the Windows 7 DVD and simply copy the contents to the USB drive.
     • Add some extra utilities if you like (I usually add some diagnostic apps)
   • You now have a boot USB stick – much nicer than a slow DVD!

I poked around the BIOS a bit to figure out how to boot from USB on the Asus 1005HA. Although I found it in there, the manual later pointed out that pressing ESC during boot will display a ‘boot menu’ allowing me to select which disk to boot from – USB stick or internal hard drive. Nice. There is a little feature called “Fast Boot”, which I think I had to disable. Not exactly sure what this does, but most likely simply skips some POST steps and goes right to OS loader on same device as “last time”. It did speed up boot time; Asus probably has their act together here, although I wish they would call it something like “skip some power-on tests” rather than “fast boot” so we can make technical decisions rather than marketing decisions about such issues.

Anyway, I booted up from USB and a few minutes (literally) later, I’m up and running with Windows 7. Fourteen minutes, to be exact. Including wiping all disk partitions and formatting a single new one.

Windows 7 Performance

Hardware detection was perfect; driver installation successful for all devices. But are all hardware features available? Here is what I found. Asus does not support Windows 7 at this time, so specific Windows 7 drivers for the equipment cannot be obtained from Asus. I’m sure this will change when Windows 7 is released to the public in October.

The Results:

• All hardware devices were recognized by the boot media and drivers installed except for one – the ACPI Device, i.e., power management. This was corrected automatically after connecting to the internet for the first time. Three other drivers (Atheros wireless, bluetooth, and Intel’s onboard video) also took the opportunity of being on the net to updates themselves via Windows Update.
• Memory performance, i.e., transfer speed from RAM, was measured at 4074MB/s, which is not too good compared to a modern desktop. This is not reflected in the Windows experience score, so there is either something wrong with the measuring utility or I am too accustomed to max-scores of 5.9 on my desktop boxes (I wonder how good those machines really are).
• The “Windows Experience” numbers. Funny how well Aero works with only a 2.0.
• Processor: 2.3
• Memory: 4.5
• Graphics: 2.0
• Gaming Graphics: 3.0
• Primary HD: 5.4
• Network performance was measured with > 50Mb/s throughput under ideal conditions, which is excellent.
• The Synaptics touchpad is running with generic drivers. These leaves me without “special” touchpad features (like scrolling). I suspect this will change as October draws near. [Synaptics is a very mainstream touchpad manufacturer]. I’ll update to the Vista driver from the Synaptics web site when I get a chance.
• Subjective performance is very good: video is smooth from Web, SD card, and Hard Disk; Aero is perfect, including all the new peek and preview stuff in Windows 7; disk access is good; the little computer is quite snappy.
• Battery life came around 8 hours first cycle (fairly-heavy on–and-off usage), charges quickly when sleeping (1 hour or so if I remember correctly?). I have not gone through another cycle. My guess is that with Bluetooth and Wireless Ethernet on and cranking, you will get 8Hr max (less watching video this way), but you could get 10+Hours on an airplane ride with the radios off. Very nice. [Note: Asus markets as ‘Up to 10.5 hours’, which is a reasonable claim.]
• Visual Studio: Sorry, I have not installed and run this. I don’t plan on using the netbook for software development. I’ll drag around my “regular” notebook for that purpose.

What else to say? Not much. Great little netbook. Perfect for travel.

What version of Windows 7 for Developers?

It has become quite cliché to point out that owners of Windows Vista Ultimate Edition got ripped-off. True, the ultimate extras never really materialized (a poker game or two and animated wallpaper do not count), however, most software developers like myself just had to get the Ultimate Edition for seven simple reasons:

1. We wanted Media Center, even though we did not need it
2. We wanted DVD authoring, event though we could live without it
3. We could not live without remote access (server and client)
4. Nor could we survive without Fax Manager, Disk Image Backup and Restore
5. Installing (legally) another instance in a Virtual PC was a bonus
6. Mobility Center and Bitlocker for the laptops
7. And most of us need to join a domain

Or, maybe it was just two of these

1. Join a Domain
2. Media Center

Or these two?

1. Media Center
2. Remote Access Server

Either way, Ultimate was the only answer. But what about windows 7?

Windows 7 Editions

Windows 7 Editions are cumulative! Could purchasing merely the Professional Edition be the way to go …

• Remote Desktop Host (all of us) [Professional and up]
• Backup images to network shares (most of us) [Professional and up]
• Domain Join (some of us) [Professional and up]

However, with Professional Edition, we give up

• Boot from VHD
• Bit Locker to go

How could we possibly give up boot from VHD?

Looks like Ultimate Edition is the only choice for developers in Windows 7, just as it was for Vista.