LINQ

LINQ Overview, part two (Lambda Expressions)

newman — Tue, 29 Jan 2008 15:41:15 GMT

Back Links

LINQ Overview, part one (Extension Methods)

NOTE: This article is dedicated to Keith Elder...even if he never sent me a bologna sandwich.

Apparently, two months is my definition of "very soon". Let's continue.

Since .NET 1.1 we've had the concept of delegates. They are the constructs that allow us to call methods on objects via reference such as:

delegate int AddFunc(int x, int y);

public static class MathOps
{

   public static int Add(int x, int y)
   {

      return x + y;

   }

class Program
{

   static void Main(string[] args)
   {

      AddFunc f = new AddFunc(MathOps.Add);

      Console.WriteLine("Delegate: 2 + 2 = {0}", f(2, 2));

      Console.ReadLine();

   }

There is nothing new and exciting about delegates as calling a function via pointer has been around for a very long time. In fact, delegates are actually somewhat annoying in terms of syntax. They must be declared in a class, you must wrap them in an object, etc. Why can't we have a simpler syntax? After all, most of the time delegates are used to respond to relatively simple events or act as part of a strategy pattern (e.g. in a sort).

Anonymous Methods

In honor of Bill Gates, .NET 2.0 decided to give us a kindler and gentler delegate syntax. The main method above could easily be rewritten as:

class Program
{

   static void Main(string[] args)
   {

      AddFunc f = delegate(int x, int y) { return x + y; };

      Console.WriteLine("Anonymous Method: 2 + 2 = {0}", f(2, 2));

      Console.ReadLine();

   }

As is common on the .NET platform the delegate keyword was overloaded to give it additional meaning. Now one could assign to a delegate variable directly, in the current scope. The new anonymous method syntax was similar to a method declaration. The differences are pretty obvious, but I'll list the major ones. Firstly, anonymous methods don't require identifiers, hence the terms anonymous methods. Secondly, anonymous methods do not need to specify a return type. This is due to some rudimentary type inference built into the compiler. In essence, if we already know that we are assigning to a delegate of type "AddFunc" whose return type is "int", it should be obvious to the compiler that as long as the return statements in the delegate's body return an "int" then our anonymous delegate matches the signature of "AddFunc". The counterintuitive aspect of this is that we still have to specify the types of the anonymous method's arguments. After all, shouldn't the compiler be smart enough to also assume the types of our "x" and "y" based on the delegate type we are assigning to? It should be, but unfortunately it is not.

There is something else I want to say about anonymous methods before moving on. This is something I come across all the time and some developers just don't get: anonymous methods allow for lexical closures.

Lexical Closures

There is a lot of bickering on the net about "does .NET support 'true' closures?" Well, based on my understanding and in my opinion, they support lexical closures or at least something close enough that for most practical purposes it doesn't matter. I'll leave the 100% correct definition to the language lawyers and just give a quick example and some reasons why a lot of developers get caught in the lexical closure trap.

delegate int Increment();

static void Main(string[] args)
{

   Increment AddOne = AnonInc(0, 1);
   Increment SubOne = AnonInc(10, -1);

   for (int i = 0; i < 10; ++i)
   {

      Console.WriteLine("{0},{1}", AddOne(), SubOne());

   }

   Console.ReadLine();

static Increment AnonInc(int start, int by)
{

   return delegate { return start = start + by; };

}

The output of the about code should be:

1,9
2,8
3,7
4,6
5,5,
6,4,
7,3
8,2
9,1
10,0

First, take a look at our delegate "Increment". It takes no arguments and returns an "int". The idea is that delegates will somehow increment "a value" and return the next value in the sequence.

Next, look at the method "AnonInc". Does it return a delegate? That's crazy! Further, it returns a delegate that makes use of something commonly referred to as "up values" or "outer variables" depending on the person/system/said person's mood. An outer variable is simply a variable that exists in the scope that contains the delegate. In this case, our delegate's scope is the "AnonInc" method in which the "start" and "by" arguments are implicitly defined local variables.

Now, based on the definition of the delegate returned by "AnonInc" and the output of the program we can tell something interesting is going on here. The question you should be asking right now is, "How is it that we are modifying the value of a local variable inside a delegate and it is keeping track of the change?"

If you recall delegates, and therefore anonymous methods, are represented by objects. These objects are instances of classes that are automatically generated for you at compile time. They have funny, mangled names and you can not really do too much with them. The thing that one needs to know is that any outer variables used by an anonymous delegate become attributes of this auto-generated class. So, in our case if we look at the assembly generated by the above program using a tool like Reflector we should find a class like:

[CompilerGenerated]
private sealed class <>c__DisplayClass7
{
    // Fields
    public int by;
    public int start;

    // Methods
    public int <AnonInc>b__6()
    {
        return (this.start += this.by);
    }
}

As you can see, the above class has two attributes with the same names as our outer variables and a method that accesses them. Looking at the code this way kind of takes the magic out of anonymous methods and we being to realize that it is sort of like what I said about extension methods, it is just syntactic sugar. Handy, but not magical.

So, what is this trap I was talking about? Well, it has to do with the garbage collector. As we all know, in .NET an object lives in memory until it is explicitly disposed of or goes out of scope. In general perhaps "goes out of scope" is best thought of as "until no other object holds a reference to it". With lexical closures happening more or less behind the scenes it is very easy to create a memory leak such as the following:

public class ResourceWrapper
{

    public void OpenOnClick(Button btnOpen, string resourcePath)
    {

        SomeResource res = new SomeResource(resourcePath);

        btnOpen.Click += delegate(object sender, EventArgs e) { res.Access(); };

    }
    
}

public class SomeResource
{

    public SomeResource(string path) { }

    public void Access() { }

}

Granted, this example is contrived, but you see similar things all the time. So, what's going on here? Basically if we look at "OpenOnClick" we can see that an anonymous method is being registered as the Click event for a button. Further, the anonymous method is using an outer variable "res". This means that the following class gets generated for us:

[CompilerGenerated]
private sealed class <>c__DisplayClass1
{
    // Fields
    public SomeResource res;

    // Methods
    public void <OpenOnClick>b__0(object sender, EventArgs e)
    {
        this.res.Access();
    }
}

Normally, we'd just assume that since "res" is a local variable in the "OpenOnClick" method that it'd die as soon as it ran out of scope, i.e. at the end of the method. However, since our anonymous delegate is holding a reference to it, the object "res" is referencing will live until the anonymous delegate itself goes out of scope. One can easily see how this kind of situation can go bad quickly. To avoid this situation, be careful to unregister your anonymous methods when you use them as event handlers!

Alright, so why did I get into all of this anonymous method stuff if the post is supposed to be about Lambda Expressions? Well, because Lambda Expressions in C# are just an evolutionary step beyond anonymous methods. Let's chip away at some of the sugar...

Our first Lambda

It is difficult to describe the syntax of a lambda expression since it is very ambiguous and depends on multiple factors. With that in mind let's look at a quick example:

AddFunc f = (x, y) => x + y;

The above snippet declares a new AddFunc delegate and assigns a lambda expression to it. Everything to the right of the = operator is the lambda definition.

Some questions:

Where is the return type?
Where is the identifier?
Does (x, y) denote the parameter list?
What does the => do?
Why isn't there a return statement?

Some answers:

Lambda expressions do not need an explicit return type. Just like with anonymous methods the compiler is smart enough to infer the return type based on the type of delegate it is being assigned to. In this case AddFunc returns an int, and so the lambda implicitly returns and int. Obviously it is a compiler error if the lambda does not.
Lambda expressions are by definition anonymous. They do not have identifiers.
Yes. Further, you should note that lambda parameters do not need to explicitly state their type. This, like the return type, is inferred by the compiler based on their order compared to the delegate's parameters list. You can, however, state the types explicitly. (int x, int y) is a valid lambda expression parameter list.
The new => operator is the start of the expression's body. Everything after => defines what the lambda expression does.
Lambda expression don't require an explicit return statement. When a return isn't provided the return value is assumed to be whatever the lambda expression evaluates to.

So, let's take a look at a few other valid ways to write lambda expressions:

(int x, int y) => { return x + y; };

The above is the most explicit way. We've specified types for the parameters and a real return statement. Notice how when we use an actual return expression we have to use the { } brackets? This same syntax allows us to create multi-line lambdas and lambdas that declare local variables.

(x, y) => { return x + y; };

This one keeps the return statement and just drops the optional types in the parameter list.

() => x + y;

In the above, we've specified a lambda with an empty parameter list. In this case we are assuming the existence of x and y as outer variables (yes, lambda expressions support lexical closures just like anonymous methods).

A Lambda is what you assign it to

So far we've seen that lambda expressions are compatible with delegates in the sense that you can assign a lambda directly to a delegate, but there are other interesting uses. Take a second and think about writing a program in a text editor. To the text editor, or for that matter to the compiler, the lines of code your write are just data. The compiler doesn't execute your program, it simply translates data from one format to another. It is natural then to ask, "If I can store a program as data, can I load a program as data at run time and then execute it?" With lambda expressions the answer is yes.

If we assign a lambda expression to a delegate it becomes a delegate of that type.

If we assign a lambda expression to an appropriately typed Expression Tree it gets converted at compile time to equivalent Expression objects.

For example:

Expression<Func<int, int, int>> exp = (x, y) => x + y;

This statement simply says, "Convert this lambda expression into an expression tree equivalent to a method that takes two integer parameters and returns the sum as an integer".

There is no resulting compilation of this tree and no execution of code as a result of this statement. If at runtime we need to execute the function the tree represents, we must say:

Expression<Func<int, int, int>> exp = (x, y) => x + y;
var func = exp.Compile();
Console.WriteLine("{0}", func(1, 1));

Now, it isn't inherently obvious why this is cool so I'll spell it out: If the compiler can represent executable code using Expression objects, so can we. In fact, we will do exactly that by the end of this series.

As funny as it may sound, this is all you really need to know about lambda expressions. You can use them in place of anonymous delegates (and you should), they forced the .NET team to provide C# with something approaching real type inference, and they allow us to represent code as data in a statically type checked way.

LINQ Tie In

Awesome. How are Lambda Expressions useful in LINQ? Well, by now you've read the basic LINQ syntax somewhere else as I asked so I'll just show a couple of quick examples:

static void UseLINQ()
{

    var names = new List<GenderedName> { 
        new GenderedName { Name="Bob", Gender=Gender.Boy }
        , new GenderedName { Name="Sally", Gender=Gender.Girl }
        , new GenderedName { Name="Jack", Gender=Gender.Boy }
        , new GenderedName { Name="Sarah", Gender=Gender.Girl }
        , new GenderedName { Name="Philbert", Gender=Gender.Boy }            
    };

    var boyNames = names.Where((n) => n.Gender == Gender.Boy).Select((n) => new { n.Name });

    foreach (var name in boyNames)
        Console.WriteLine("{0}", name.Name);

}

This above function queries a list of names for those that are traditionally used for boys. In order to make use of the actual lambda expression syntax I used the method based approach to querying with LINQ. In fact, there are two lambdas in our code:

(n) => n.Gender == Gender.Boy

This lambda is for our selection criteria and simply compares the given name, n, to see if it is used for boys.

(n) => new { n.Name }

In this expression we are returning a new anonymous type that just contains the Name property of the GenderedName that has passed our selection criteria.

We can simplify, or rather pretty up, this method by using the new LINQ keywords as so:

static void UseLINQ()
{

    var names = new List<GenderedName> { 
        new GenderedName { Name="Bob", Gender=Gender.Boy }
        , new GenderedName { Name="Sally", Gender=Gender.Girl }
        , new GenderedName { Name="Jack", Gender=Gender.Boy }
        , new GenderedName { Name="Sarah", Gender=Gender.Girl }
        , new GenderedName { Name="Philbert", Gender=Gender.Boy }            
    };

    var boyNames = from n in names
                   where n.Gender == Gender.Boy
                   select new { n.Name };

    foreach (var name in boyNames)
        Console.WriteLine("{0}", name.Name);

}

It doesn't look like we are using lambda expressions here, but we really are. It is just that the compiler needs to turn our pretty code into the same method calls that we just used, and therefore ultimately into an Expression Tree for later execution.

I just want to be very explicit here and point out something. When we are using LINQ we use lambda expressions as delegates. We know this because the parameters of the Where method accept arguments of the Func<T> variety. The Func series of generic types are actually generic delegates. For example, MSDN has the following definition for Func<T, TResult>:

public delegate TResult Func<T, TResult>(
    T arg
)

This usage of delegates and expression trees is what allows LINQ to support Lazy Evaluation.

LINQ Overview, part one (Extension Methods)

newman — Mon, 12 Nov 2007 15:08:27 GMT

Back Links

LINQ Overview, part zero

Forward Links

LINQ Overview, part two (Lambda Expressions)

In part zero I stated my intentions, now it is time to act.

If you've ever programmed in C (no, I didn't forget the #) you may have had a function prototype laying around similar to:

int Deposit(struct account *acct, double amnt);

If one were to rewrite this today in C# you'd probably have a class to represent accounts and your method definition would just be:

public void Deposit(double amount)

You would have dropped the int because you can throw an exception if there is an issue and you no longer need to pass in a pointer (i.e. reference) to the account because our account object will contain all the state for us. In reality what is the difference here? Well, it more or less comes down to syntax. To execute this code in C I would have to say:

Deposit(&acct, 350.75);

In C# I'd write:

acct.Deposit(350.75);

The great thing about the C# version in terms of syntax is that there are no funky operators to deal with and it is more English like in terms of reading left to right. In terms of flexibility, however, I have to give the edge to the C version. Why? Well, because if I need to add a new operation on the account data type in C I can do it anywhere that I want. All I need to have access to is the prototype of the account struct and I can introduce the following:

int Withdraw(struct account *acct, double amnt);

In C#, I simply can't add a "third party" method to a class that I don't have the source code to. Sure, partial classes allow me to add methods, but they must be in the same namespace and assembly in order to work since they are a compile time construct. I could also just take the C approach and say something like:

static void Withdraw(ref Account acct, double amnt)

That however, doesn't clean up the syntax issue from the caller's perspective as they'd now have to pass in the reference to an Account object like so:

Helper.Withdraw(ref acct, 350.75);

Crap! It is actually more to type than the C version.

Enter extension methods. An extension method is a static method whose first parameter is decorated with the overloaded 'this' keyword. For example:

public static void Withdraw(this Account acct, double amnt)

Alright, still not much shorter for me as the extension method's author, but how does it look when a piece of client code calls it?

acct.Withdraw(350.75);

As you can see, the call syntax is exactly the same as with the C# version of Deposit. Under the hood all that is going on is the compiler is seeing our use of 'this' in front of the first parameter and saying, "OK, I know now that I can allow this to be called on Account objects". Further, if Account was a base class of another type, e.g. BusinessBankAccount, then the extension method would also work on BusinessBankAccount objects. Do I need to mention that the same holds true for extensions defined to operate on interfaces? Heck, you can even make an extension method that makes use of Generics!

Another cool aspect of extension methods is that in the Visual Studio 2008 (and even Visual Studio 2005 if you have the .NET 3.0 CTP) environment they are fully supported by intellisense.

There are a couple of restrictions that should be obvious, but I'll list them here anyway. First, since the extension method is still technically a member of a different class it will only have access to the public members exposed by the class being extended. In other words, our Withdraw extension method can not access the private and protected members of the Account class. This restriction places a definite limitation on what can be achieved through extension methods, but is necessary to avoid violating encapsulation. Second, a consumer of the extension method needs to reference the assembly in which the extension lives or the compiler won't see it.

LINQ Tie In

Now that we know what extension methods are let's take a look at how they are utilized by LINQ. LINQ is designed to extend query capability to .NET types using extension methods. The standard query operators of LINQ operate on any type that implements IEnumerable<T>. There are other technologies in the LINQ family that provide sets of extensions methods, for example LINQ To Dataset provides the same extensions, but for types derived from Dataset.

If we focus for now on vanilla LINQ, i.e. the one that operates on in memory collections, we can think of at least three ways to construct it.

methods that accept IEnumerable<T> parameters using existing, i.e. pre 3.0, syntax
add methods to the IEnumerable<T> interface for things like Select, Join, etc
provide extension methods for the IEnumerable<T> interface

Item one will probably work, but requires the clunky syntax we saw before. It could possibly be hidden behind new C# keywords, but it may have required more work at the compiler level and would have made doing any type of dynamic LINQ an undue burden on the developer.

Item two is obviously out the window. For starters, changing such a core interface like IEnumerable<T> would require so many rewrites not only in the framework, but third party code as well, that Microsoft would have had a developer mutiny on their hands.

Item three is what they ultimately went with and is basically the same as item one now that we know how extension methods actually work. The advantage is the cleaner syntax offered to developers.

Naive, LINQ-like Extension

There are probably a million blogs out there that have the information you've already seen here so far. So, I am not going to go over the actual LINQ syntax right now. Instead I am going to show the method by which LINQ is constructed in a very limited, naive case.

Let's say LINQ doesn't exist and your team was on a project where you were constantly searching through collections of objects using lots of basic criteria. We could introduce an extension to the IEnumerable<T> type that allows us to specify our criteria and get back a new collection of items that match it. Our method might look something like:

public static IEnumerable<TResult> SelectWhere<TResult>(     
     this IEnumerable<TResult> source,     
     Func<TResult, bool> filter)
{

     var results = new List<TResult>();

     foreach(var s in source)
          if(filter(s))
               results.Add(s);

     return results;

}

The above code defines an extension method named SelectWhere that accepts a generic argument called TResult. TResult is used to flesh out the method's remaining arguments as well as its return type.

The return type and first parameter (the one decorated with 'this') are obvious to us at this point. The second parameter, Func<TResult, bool>, is simply a generic delegate type. For a method to match the delegate's signature it must accept a single TResult parameter and return a bool.

If we examine the implementation it is pretty straightforward. We simply iterate over the entire collection and call the delegate for each item to determine if we need to save it.

Now, how can we call this from client code? There are a few ways, each of which is legal so let's start with the most explicit and work our way towards the sugar.

First we need a method that matches the delegate to act as our filter criteria:

public static bool PersonIsRich(Account acct) 
{

     return acct.Balance >= 10000D;

}

Your definition of a rich person may be different than mine, but I am sure you get the idea. We return true if the account has 10,000 or more.

Next we call the method on a collection of Accounts:

var richPeople = SelectExtension.SelectWhere<Account>(allPeople, PersonIsRich);

As we can see the above is our more verbose usage and takes us back to the C function from before. We need to pass in both the collection we are operating on as well as our filter.

var richPeople = allPeople.SelectWhere<Account>(PersonIsRich);

In this call we've used the extension method syntax and are just specifying the generic argument and the filter.

var richPeople = allPeople.SelectWhere(PersonIsRich);

At this point, we are beyond extension methods. What just happened above is that the C# compiler is smart enough to figure out what our generic argument has to be in order to satisfy the extension method and doesn't bother making us put it there ourselves. Logically speaking, if I am calling SelectWhere on an IEnumerable<Account>, then there is only one type that TResult can be, i.e. Account. This is called 'type inference' and is the same reason that I can use the 'var' keyword instead of specifying the type of richPeople statically. The compiler determines the type for me at compile time.

Summary

We have now seen what an extension method is and how it relates to LINQ. We also now know that they are not magic and that we could get the same behavior from a normal, static method. The new syntax helps add clarity though, and in development, clarity is a good thing.

As a teaser for the next part, take a look at this:

var richPeople = allPeople.SelectWhere(acct => acct.Balance >= 10000D);

This is the final magic. Where did our delegate function go? What is that crazy syntax? Well, that is a lambda function and they will be the topic of my next post.

Download Solution - LinqOverview.zip

LINQ Overview, part zero

newman — Fri, 09 Nov 2007 15:31:59 GMT

Forward links

LINQ Overview, part one (Extension Methods)

LINQ Overview, part two (Lambda Expressions)

I realize that these next few posts may be late to the game. The LINQ CTPs for .NET 3.0 have been out for quite a while and everyone already knows about the massive amount of improvements Microsoft made in the Beta1 & Beta2 releases of .NET 3.5. Further, we are supposedly less than a month away from seeing the official Visual Studio 2008 release. All that being said I am going to spend a little time introducing the foundational pieces of LINQ so that I can lay the ground work for a more ambitious series.

First, it is important to understand that LINQ is not magic. It is purely syntactic sugar and is built from the ground up using some new primitives introduced in .NET 3.0/3.5. The three most important are:

Extension Methods
Lambda Expressions
Anonymous Types

To tie it all together the LINQ team added several new keywords to the .NET language pool, but don't be mistaken, these are just conveniences like property getters and setters or the event keyword. Everything done using the keywords can be done using raw code.

My plan is to go from the ground up and demonstrate each new language feature briefly. Doing so will allow us to examine compile time LINQ usage with all the cards on the table and nothing hidden behind syntax.

While I take a day to put together part one, you might find the following links interesting. Some of them are a bit heavy, but they are all valuable. You'll also want to have access to some variant of Visual Studio 2008. The Express versions are available free from Microsoft.

Visual Studio 2008 Beta2 - The Express editions are always going to be free and the downloads are a lot smaller, I tend to stick with the Professional versions as I get them via MSDN.

Official LINQ Project website - Contains basic documentation and downloadable samples.

Matt Warren's Blog - High upper on the LINQ Project team, invaluable.