Alois Kraus

Posts: 69
Comments: 233
Trackbacks: 162

Saturday, December 10, 2011

This is another x64 issue where I was really surprised that it does exist. When you compile in VS2010/MsBuild a managed target for which the corresponding serialization assembly is generated you will find that all works until you try to compile for 64 bit. There you will get:

"SGEN : error : An attempt was made to load an assembly with an incorrect format: xxxx.dll."

Here is the 32 bit SGen is called for the 64 bit target which cannot load a 64 bit assembly. I have no idea why MS did not provide a SGen compiled as Any CPU but the Windows SDK does bring in a 32 and 64 bit version of SGen. After a little digging around I quickly found that I can manually set the SGenToolPath MsBuild property. I could have patched

C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Common.targets

directly but then I would have changed the checksum of the file and future .NET Framework patches would not be able to patch this file anymore. Since the VS 2010 project format builds completely upon MsBuild I can set the path also in my Studio project for each build mode correctly.

By the way it is not enough to simply set “Generate serialization assembly” to On to get an xxx.XmlSerializers assembly for all public types.

The SGen task of the .NET Framework does by default only create an serialization assembly if it does contain types derived from some web proxy classes used for Web Services. To create an xml serializer type for all public types of my assembly I needed therefore to set for all build modes

    <GenerateSerializationAssemblies>On</GenerateSerializationAssemblies>
    <SGenUseProxyTypes>false</SGenUseProxyTypes>

MsBuild does automatically add all direct assembly references to SGen as well to allow SGen to create the serialization code for types which have base classes in other assemblies. This service is good but it is not perfect. If you have a deeper inheritance tree you will need more than all direct references of your assembly. Since SGen is located in the Windows SDK there is no way to specify a probing path relative to my build output directory since .NET allows only directories below the executable location as additional probing paths. I decided to make life easy and simply copy SGen to my build output directory to cover all possible cases.

At first I do need the path to the right sgen.exe in the current build mode. For this I did define the property __SdkSgenTool. I do need two tries to resolve the right SGen since the SDK can be in Program Files (32 bit Windows) or Program Files (x86) (64 bit Windows). Then I do copy sgen.exe to the build output directory $(TargetDir).

The only missing thing is that I do need to set SGenToolPath to my build output directory. This was harder as expected since as a normal property it was overwritten by other MsBuild tasks. The solution that finally did work was to create the already existing property and set the value to its final value when no other tasks could interfere.

Below is the “code” to make Sgen work in 64 bit. You need to define the __SdkSgenTool variable in all build modes since the post build steps like copy are executed regardless of the build mode.

  <PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Release|x64'">
   ….
    <GenerateSerializationAssemblies>On</GenerateSerializationAssemblies>
    <SGenUseProxyTypes>false</SGenUseProxyTypes>
    <__SdkSgenTool Condition="exists('C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools\x64\sgen.exe')">C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools\x64\sgen.exe</__SdkSgenTool>
    <__SdkSgenTool Condition="exists('C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools\x64\sgen.exe')">C:\Program Files\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools\x64\sgen.exe</__SdkSgenTool>
  </PropertyGroup>

…

  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
  <Target Name="BeforeBuild">
    <Copy SourceFiles="$(__SdkSgenTool)" DestinationFiles="$(TargetDir)\sgen.exe" SkipUnchangedFiles="true" />
    <CreateProperty Value="$(TargetDir)">
      <Output TaskParameter="Value" PropertyName="SGenToolPath" />
    </CreateProperty>

I have heard that this issue will be fixed with VS2012 which is a good thing. MsBuild is very powerful but I do prefer a working build in all build modes without any surprises. The path to SGen is hard coded since I do not expect that somebody does install Visual Studio to another location as Program Files. The SDK directories below Microsoft SDKs\Windows\ are normal downloadable Windows SDKs if they contain only a version number. If there is an A at the end these SDKs are the ones which Visual Studio does bring with.

Share This Post:

Posted On Saturday, December 10, 2011 7:43 PM | Feedback (0)

Friday, December 09, 2011

Microsoft == x86 && Sometimes !x64

Whenever I do use Visual Studio and try to compile something under 64 bit I run into problems. It seems that most MS Devs for Visual Studio and the relevant tool chain are still mainly writing 32 bit applications.

Here are some of the latest issues I did run into.

COM applications targeting x64 are still using x32 as target platform for the MIDL compiler by default

Resolution: You have to select in the UI MIDL – Target Environment X64 by yourself. Alternatively you can edit the vcxproj file directly and set the flag there:
  <Midl>
  <TargetEnvironment>X64</TargetEnvironment>

The othere issue was that on my dev machine Visual Studio 2010 SP1 did freeze quite often. The dump showed that it did hang while loading the assembly Microsoft.VSDesigner. I have filed a Connect issue for this but I got not any helpful feedback yet. In the meantime I did find out by myself why VS was hanging. Once the hang did occur it was freezing quite often which is good for a repro but very annoying if there is nothing you can do about it. I did send my error reports to MS every time hoping that they resolve the issue before VS2012 is released. The Connect support person requested another dump from me this time taken with VS directly. I have to remember that for my own trouble shooting that VS can now also take dumps which really helps. The interesting thing was that the VS call stack was more helpful since it seemed to resolve the symbols better. The deadlock has frozen the UI thread while some stack frames from MSI were on it.

    ntdll.dll!_ZwWaitForSingleObject@12() + 0x15 bytes
    ntdll.dll!_ZwWaitForSingleObject@12() + 0x15 bytes
    kernel32.dll!_WaitForSingleObjectExImplementation@12() + 0x43 bytes
    msenv.dll!_VsCoCreateAggregatedManagedObject() + 0xe2 bytes
    msenv.dll!_VsLoaderCoCreateInstanceUnknown() + 0x8e bytes
    msenv.dll!CVsLocalRegistry4::CreateInstance() + 0x4a bytes
    msenv.dll!CXMLMemberIndexService::GetCulture() + 0x17f5 bytes
    msenv.dll!CXMLMemberIndexService::LocateAndOpenXMLFile() + 0x14 bytes
    msenv.dll!CXMLMemberIndexService::CreateXMLMemberIndex() + 0x78 bytes
    cslangsvc.dll!CMetaDataLoader::EnqueueMemberIndexRequest() + 0xe80cc bytes
    cslangsvc.dll!CMetaDataTypeData::GetDocumentationComment() + 0x1b bytes
    cslangsvc.dll!CSymbolDescription::CPartCollector::ConditionallyAddDocCommentParts<CTypeData>() + 0x32 bytes
    cslangsvc.dll!CSymbolDescription::CTypeProviderHelper::TryExecute() + 0x278 bytes
    cslangsvc.dll!CSymbolDescription::CNameProviderVisitor::Visit() + 0x56 bytes
    cslangsvc.dll!CTypeProvider::Accept() + 0x13 bytes
    cslangsvc.dll!CSymbolDescription::CNameProviderVisitor::TryExecute() + 0x2a bytes
    cslangsvc.dll!CSymbolDescription::TryGetDescription() + 0x37 bytes
    cslangsvc.dll!CSymbolDescription::TryAppendDescription() + 0x61 bytes
    cslangsvc.dll!CSpanBinder::TryExecuteAndGetDescription() + 0x88 bytes
    cslangsvc.dll!CQuickInfo::TryGetRawIntelliSenseQuickInfo() + 0x81 bytes
    cslangsvc.dll!CQuickInfo::TryGetFullIntelliSenseQuickInfo() + 0x4d bytes
    cslangsvc.dll!CQuickInfo::TryExecute() + 0x3e bytes
    cslangsvc.dll!CEditFilter::GetDataTipText() + 0xdd bytes
    cslangsvc.dll!CVsEditFilter::GetDataTipText() + 0x52 bytes
    user32.dll!_InternalCallWinProc@20() + 0x23 bytes
    user32.dll!_UserCallWinProcCheckWow@32() + 0xb7 bytes
    user32.dll!_DispatchMessageWorker@8() + 0xed bytes
    user32.dll!_DispatchMessageW@4() + 0xf bytes
    msi.dll!MsiUIMessageContext::RunInstall() + 0x21231 bytes
    msi.dll!RunEngine() + 0xb3 bytes
    msi.dll!ConfigureOrReinstallFeatureOrProduct() + 0xfa bytes
    msi.dll!_MsiReinstallFeatureW@12() + 0x66 bytes
    msi.dll!ProvideComponent() + 0x10957 bytes
    msi.dll!ProvideComponentFromDescriptor() + 0x154 bytes
    msi.dll!_MsiProvideAssemblyW@24() + 0x437 bytes
    msenv.dll!_VsCoCreateAggregatedManagedObject() + 0xe2 bytes
    msenv.dll!_VsLoaderCoCreateInstanceUnknown() + 0x8e bytes
    msenv.dll!CVsLocalRegistry4::CreateInstance() + 0x4a bytes
    msenv.dll!CXMLMemberIndexService::GetCulture() + 0x17f5 bytes
    msenv.dll!CXMLMemberIndexService::LocateAndOpenXMLFile() + 0x14 bytes
    msenv.dll!CXMLMemberIndexService::CreateXMLMemberIndex() + 0x78 bytes
    cslangsvc.dll!CMetaDataLoader::EnqueueMemberIndexRequest() + 0xe80cc bytes
    cslangsvc.dll!CMDMemberData::GetDocumentationComment() + 0x51 bytes
    cslangsvc.dll!CSymbolDescription::CPartCollector::ConditionallyAddDocCommentParts<CMemberData>() + 0x2f bytes
    cslangsvc.dll!CSymbolDescription::CMemberProviderHelper::TryExecute() + 0xdc bytes
    cslangsvc.dll!CSymbolDescription::CNameProviderVisitor::Visit() + 0x53 bytes
    cslangsvc.dll!CAbstractNameProviderBoolDefaultVisitor::Visit() + 0x2b bytes
    cslangsvc.dll!CAggregateMemberProvider::Accept() + 0x16 bytes
    cslangsvc.dll!CSymbolDescription::CNameProviderVisitor::TryExecute() + 0x2a bytes
    cslangsvc.dll!CSymbolDescription::TryGetDescription() + 0x37 bytes
    cslangsvc.dll!CSymbolDescription::TryAppendDescription() + 0x61 bytes
    cslangsvc.dll!CSpanBinder::TryExecuteAndGetDescription() + 0x88 bytes
    cslangsvc.dll!CQuickInfo::TryGetRawIntelliSenseQuickInfo() + 0x81 bytes
    cslangsvc.dll!CQuickInfo::TryGetFullIntelliSenseQuickInfo() + 0x4d bytes
    cslangsvc.dll!CQuickInfo::TryExecute() + 0x3e bytes
    cslangsvc.dll!CEditFilter::GetDataTipText() + 0xdd bytes
    cslangsvc.dll!CVsEditFilter::GetDataTipText() + 0x52 bytes

The language service tries to get some tool tip (GetDataTipText) text and tries to load an assembly. For reasons unknown to me MSI is used to get an assembly and install it if it is not already present. Since the requested assembly was not found an installation was performed. While the installation is running MSI does pump again window messages which does let VS again to call GetDataTipText since my mouse was still hovering over some code element in the editor. MSI would be called again to resolve the same assembly but since the installation is already running VS does block. There is another helper thread running at cslangsvc.dll!CBackgroundQueue::ExecuteRequests() which seems wait for the msi installation to finish on the UI thread but since MSI cannot report any progress or failure back via the Window message loop we have a classic hang. You could ask how I do know about the other thread holding the same lock?

Starting with Windows Vista WCT (Wait Chain Traversal) was added to the Kernel which is accessible via some Windbg extensions or much easier via the Windows Resource Monitor (at the comamnd line enter: perfmon.exe /res). Hung processes are marked as red and are displayed at first in the process list (very nice). From there it is a simple right click on the hung process to analyze the locks. There you can see directly which thread is waiting for which other thread/s. No more kernel debugging!

The question that remains is why the MSI installation did never complete anyway which would have caused only one VS hang. To see what is going on on the MSI side you need to enable MSI logging. With a little practice you can quickly find the interesting lines:

=== Verbose logging started: 05.12.2011 15:57:51 Build type: SHIP UNICODE 5.00.7600.00 Calling process: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe ===
Command Line: REINSTALL=SysClrTypFeature REINSTALLMODE=pocmus CURRENTDIRECTORY=C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE CLIENTUILEVEL=3 CLIENTPROCESSID=11280
MSI (s) (5C:38) [06:29:54:573]: SOURCEMGMT: Failed to resolve source
MSI (s) (5C:38) [06:29:54:573]: Product: Microsoft SQL Server System CLR Types (x64) -- Error 1706. An installation package for the product Microsoft SQL Server System CLR Types (x64) cannot be found. Try the installation again using a valid copy of the installation package 'SQLSysClrTypes_amd64_enu.msi'.

The MSI which was tried to patch was Microsoft SQL Server System CLR Types (x64). For some reason the original MSI was no longer cached in the \Windows\Installer directory which caused the installation silently to fail. From there it is easy to resolve the issue: Install the original Microsoft SQL Server System CLR Types (x64) MSI and you should see no more failed patching while VS is running.

This issue was really annoying but since then VS 2010 is running happily on my machine without any hangs. I am really happy that I was able to resolve this problem so easily. The only issue I still do have that VS2010 is a memory hog and I do pity the poor people with slow hard discs. On these machines VS 2010 is no fun to use.

Share This Post:

Posted On Friday, December 09, 2011 10:53 AM | Feedback (0)

Friday, December 02, 2011

Simple Producer Consumer With Tasks And .NET 4

Threading was never so easy since .NET 4 with the TPL has been released. I know I am a bit late but there are so many nice things which might still be new to many of us. The IEnumerable interface has become famous with the introduction of LINQ but many of us have not yet realized that IEnumerable<T> and T[] or List<T> can be exchanged in many cases but there are cases where it is important to fall back to a pure IEnumerable<T> if you want to support lazy evaluation. .NET 4 has for example taken advantage of the lazy nature of IEnumerable<T> with the introduction of Directory.EnumerateFiles which returns immediately until the first file is found. Previously you had only the option to call Directory.GetFiles which does potentially search for a long time and will only return when all matching files have been found. This can make a big difference if you search recursively in a big file tree or a directory with many files. I had up to 40s delays in some applications which did process a large directory. 40s waiting time until you can process the first file is certainly not something you want. I did solve this issue in .NET 3.5 with DirectorySearcherAsync which did work quite well.

One additional optimization you do is to continue searching for your files on another thread when the first file has been found and returned by the enumerator. My DirectorySearcherAsync does this as well. This is working but the code is quite complex. We can get the same functionality much easier with .NET 4. The pattern applied here is a simple producer consumer pattern where the producer and the consumer live on different threads. It is easy to transform an IEnumerable<T> to an IEnumerable<T> which does the enumeration on another thread and gives you access to the returned items via a blocking queue in a thread safe manner. The new concurrent collections in .NET 4 are named a little bit different and we can find it under the name BlockingCollection<T> which was made for this exact purpose. Since I did not want to return a BlockingCollection<T> directly but I wanto to give you the possibility to

Block with a timeout to get all elements fetched so far
Block until the next item was found
Cancel a running operation

I have created an extension method not with the signature

static BlockingCollection<T> EnumerateAsync(this IEnumerable<T> source)

but these two

public static IEnumerable<T> EnumerateAsync<T>(this IEnumerable<T> source)

public static IEnumerable<T[]> EnumerateAsync<T>(this IEnumerable<T> source, CancellationToken token, int millisecondsTimeout=-1, int maxCount=0)

The first one is simply a convenience method over the second one which does basically cover the use case you expect from a responsive UI. I do need the ability to cancel the current operation which can be done nicely with a CancellationToken if the user has changed his mind and pressed impatiently the stop button. At the same time he does want to see the results as fast as possible when they are ready. This is done with the millisecondsTimeout flag which does unblock the enumerator e.g. every 500ms to give you the e.g. found files so far. If none have been found yet you are not woken up since there is no work to do. At the same it is also disturbing to update the UI with 10 000 entries at once even if it did take only 500ms. There is need to update the UI more often than every 500ms if many items are pending. For this case the maxCount flag is there to give you at most e.g. 100 items at once. For your exact use case the numbers may turn out different but for file based operations they turned out to be quite good. You did notice that the return value of the second method is not IEnumerable<T> but IEnumerable<T[]>?

The reason is simple: Chunking. What would happen if you get 10000 items in 500ms? You have most likely some code like this in the beginning.

void WorkerThread()
{
    foreach(var x in Directory.EnumerateFiles(“C:\\”,”*.*”, SearchOption.AllDirectories)
                              .EnumerateAsync(source, token,500, 100)) 
         UpdateUI(x);
}

You will find that 10K updates in 500ms will render your UI unusable and you do need a way to update your UI less often with a chunk of gathered data. Every UpdateUI call will need to post a message into the message loop of your to reach the UI thread which is a costly undertaking. If x is not of type T but of type T[] you get a coarse grained UI update. In the previous example we did reduce the pressure on the window message pump loop by a factor 100 which feels much better.

using System;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Threading.Tasks;
using System.Threading;

namespace AsyncEnumeration
{
    public static class Extensions
    {
        /// <summary>
        /// Start enumerating on another thread. The returned enumerator blocks until the next element
        /// is available or the source enumeration has no more elements.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="source">Source enumerable</param>
        /// <returns>Async enumerable</returns>
        public static IEnumerable<T> EnumerateAsync<T>(this IEnumerable<T> source)
        {
            foreach (var t in EnumerateAsync(source, CancellationToken.None,-1,1))
            {
                yield return t[0];
            }
        }

        /// <summary>
        /// Start enumerating on another thread. The returned enumerator blocks until the next element
        /// is available or the source enumeration has no more elements.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="source">source enumerable</param>
        /// <param name="token">Cancellation token if cancellation support is desired or CancellationToken.None</param>
        /// <param name="millisecondsTimeout">Maximum number of milliseconds to wait until the so far fetched elements are returned</param>
        /// <param name="maxCount">Maximum number of elements to fetch before they are returned</param>
        /// <returns>An enumerable with an array of fetched elements to support chunking in asynchronous operations.</returns>
        public static IEnumerable<T[]> EnumerateAsync<T>(this IEnumerable<T> source, CancellationToken token, int millisecondsTimeout=-1, int maxCount=0)
        {
            BlockingCollection<T> coll = new BlockingCollection<T>();

            Task t = Task.Factory.StartNew(() =>
                {
                    try
                    {
                        // enumerator on other thread and add element to thread safe collection
                        foreach (var item in source)
                        {
                            token.ThrowIfCancellationRequested();
                            coll.Add(item);
                        }
                    }
                    finally
                    { 
                        // signal collection that no more elements will be added.
                        coll.CompleteAdding();
                    }
                },
                token);

            DateTime last = DateTime.Now;
            TimeSpan timeout = millisecondsTimeout == -1 ? TimeSpan.MaxValue : new TimeSpan(0, 0, 0, 0, millisecondsTimeout);

            List<T> chunk = new List<T>();
            T got = default(T);

            while (!coll.IsCompleted)
            {
                // Get next element or block until timeout
                if (coll.TryTake(out got, millisecondsTimeout, token))
                {
                    chunk.Add(got);
                }

                // if timeout has elapsed or the maximum chunk size has been 
                // reached yield fetched elements
                if( (chunk.Count > 0 && DateTime.Now - last > timeout ) ||
                    (maxCount > 0 && chunk.Count > maxCount))
                {
                    yield return chunk.ToArray();
                    chunk.Clear();
                    last = DateTime.Now;
                }
            }

            // if there are pending elements yield them as well
            if (chunk.Count > 0)
            {
                yield return chunk.ToArray();
            }
            
            // not really necessary but it is a good idea to wait here 
            // to allow exceptions to be thrown by the task here if there were any.
            t.Wait();
        }

        /// <summary>
        /// Wrap an enumerable to cache any yielded elements so far for faster retrieval.
        /// </summary>
        /// <typeparam name="T"></typeparam>
        /// <param name="source">Source enumerable</param>
        /// <returns>Caching enumerable</returns>
        public static IEnumerable<T> Cache<T>(this IEnumerable<T> source)
        {
            return new MemoizingEnumerator<T>(source);
        }
    }
}

That are some quite simple but powerful methods. I have not yet talked about the last method which does cache the returned elements of the source enumerator. It is sometimes useful to check if there are elements at all before you start doing something. To stay with the directory example I want to validate that the directory query does match any files at all. I can do this by

        IEnumerable<string[]> CheckQuery()
        {
            var cachedQuery = Directory.EnumerateFiles(@"C:\", "*.*", SearchOption.AllDirectories)
                                       .EnumerateAsync(Cancel.Token, 500, 100)
                                       .Cache();

            if (cachedQuery.FirstOrDefault() == null)
            {
                throw new ArgumentException("No files found");
            }

            return cachedQuery;
        }

I do start the async operation and check if there are any matches. If there are none I do throw an exception and if yes I do not want to start over again with the file search but continue. This something you would do during parameter validation in a console application to verify that the entered file queries do yield at least one file.

The MemoizingIterator behind the Cache method is very simple

using System.Collections;
using System.Collections.Generic;

namespace AsyncEnumeration
{
    class MemoizingEnumerator<T> : IEnumerable<T>, IEnumerable
    {
        public MemoizingEnumerator(IEnumerable<T> input)
        {
            _Input = input;
        }

        #region IEnumerable Members

        #endregion

        #region IEnumerable<T> Members

        public IEnumerator<T> GetEnumerator()
        {
            if (_Cache == null)
            {
                _enumerator = _Input.GetEnumerator();
                _Cache = new List<T>();
            }
            else
            { 
                foreach (var cachedValue in _Cache)
                {
                    yield return cachedValue;
                }
            }

            if (_enumerator != null)
            {
                while (_enumerator.MoveNext())
                {
                    _Cache.Add(_enumerator.Current);
                    yield return _enumerator.Current;
                }
                _enumerator = null;
            }
        }

        #endregion

        #region IEnumerable Members

        IEnumerator IEnumerable.GetEnumerator()
        {
            return GetEnumerator();
        }

        #endregion

        IEnumerable<T> _Input;
        IEnumerator<T> _enumerator;
        List<T> _Cache;
    }
}

Now we can create a simple UI but responsive UI with only a few lines of Code to display the file names of our hard drive.

The code to make it is not much but you have to be careful where you need locking and where not. When you press the Start/Stop button we toggle the text between Start and Stop. If the operation has finished on the worker thread we do switch back to start. But in the code below you will not find any locks. Why? Since we do know that our UI does run (in fact can only run) on one thread with the UI message pump there is no need to protect our self from other threads since the UI is always single threaded.

If you want to dive deeper into threading with .NET I can recommend

The best overview about threading for .NET on the net: http://www.albahari.com/threading/
Free EBook from MS (Stephen Toub is one of the authors!) http://msdn.microsoft.com/en-us/library/ff963550.aspx
Patterns for Parallel Programming as PDF http://www.microsoft.com/download/en/details.aspx?id=19222
News and updates from the pfx team http://blogs.msdn.com/b/pfxteam/

If you want to know more about the changes in the TPL with the upcoming .NET 4.5 I can recommend TPL Performance Improvements in .NET 4.5 which does contain a detailed analysis what was changed to make continuations as fast as possible. Continuations are supported with .NET 4.5 via the await keyword which would make the code below even simpler.

using System;
using System.Linq;
using System.Windows.Forms;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncEnumeration
{
    public partial class AsyncEnumeration : Form
    {
        CancellationTokenSource Cancel = new CancellationTokenSource();
        Task RunningTask = null;

        public AsyncEnumeration()
        {
            InitializeComponent();
            TaskScheduler.UnobservedTaskException += new EventHandler<UnobservedTaskExceptionEventArgs>(TaskScheduler_UnobservedTaskException);
        }

        private void StartStopButton_Click(object sender, EventArgs e)
        {
            if (!TryCancelCurrentOperation())
            {
                StartSearch();
            }
        }

        private void StartSearch()
        {
            StartStopButton.Text = "Stop"; // switch start button to stop button
            ListViewDisplay.Clear();     // delete old content
            string dirName = cDirectory.Text; // get from UI start directory

            // Do the file search not on the UI thread
            RunningTask = Task.Factory.StartNew(() =>
            {
                // EnumerateAsync starts another task which lets the enumeration running while you
                // can fetch data from the chunked results.
                foreach (var dirArray in Directory.EnumerateFiles(dirName, "*.*", SearchOption.AllDirectories)
                                                  .EnumerateAsync(Cancel.Token, 500, 100))
                {
                    // EnumerateAsync returns either when 500ms have elapsed and some data was found
                    // OR 100 items were found. 
                    var listViewItems = (from x in dirArray
                                         select new ListViewItem(x)).ToArray();

                    // block the UI thread as short as possible by adding a bunch of results
                    this.Invoke(new MethodInvoker(() => ListViewDisplay.Items.AddRange(listViewItems)));
                }
            },
            Cancel.Token);

            // When the enumeration task has finished (with or without error)
            // display the Start button again
            var uiScheduler = TaskScheduler.FromCurrentSynchronizationContext();
            RunningTask.ContinueWith((t) =>
            {
                StartStopButton.Text = "Start";
                if (t.IsFaulted)
                {
                    MessageBox.Show(t.Exception.ToString());
                }
                RunningTask = null;
            }, uiScheduler);
        }

        bool TryCancelCurrentOperation()
        {
            if (RunningTask != null)
            {
                Cancel.Cancel();
                Cancel = new CancellationTokenSource();
                return true;
            }

            return false;
        }

        void TaskScheduler_UnobservedTaskException(object sender, UnobservedTaskExceptionEventArgs e)
        {
            MessageBox.Show(e.Exception.ToString());
            e.SetObserved();
        }
    }
}

Share This Post:

Posted On Friday, December 02, 2011 12:24 PM | Feedback (4)

Friday, September 23, 2011

Unintended Api Changes - .NET 4.0 Breaking Changes

The nice thing about unintended changes is that you never think it could happen until you get bitten by a nasty change. Microsoft did publish a complete list of breaking changes here which is complete to my knowledge. But although the intentional changes are listed there are side effects which can cause you to search for hours your (non) fault.

One change was to create a new GAC for .NET 4 assemblies to prevent breaking applications which rely on the structure of the GAC introduced with .NET 2.0.

Feature: Global assembly cache location change

Differences from 3.5 SP1: For .NET Framework 4 assemblies, the global assembly cache has been moved from the Windows directory (%WINDIR%) to the Microsoft.Net subdirectory (%WINDIR%\Microsoft.Net). Assemblies from earlier versions remain in the older directory.

The unmanaged ASM_CACHE_FLAGS enumeration contains the new ASM_CACHE_ROOT_EX flag. This flag gets the cache location for .NET Framework 4 assemblies, which can be obtained by the GetCachePath function.

Recommended changes: None, assuming that applications do not use explicit paths to assemblies, which is not a recommended practice.

Ok MS thinks that you do not need to make any changes to your application until you rely on the structure of the GAC. Well all our applications rely on the structure of the GAC. Not directly on the file structure as the writer of this change did think but we are all bound to the limitations of the file system.

If you create an installer for your managed assemblies and you do install into the GAC then it can happen that you see this:

When you turn on MSI logging (e.g. msiexec /i <your msi> /lvx* c:\temp\log.txt) for the faulting msi you will get this

Error 1935. An error occurred during the installation of assembly

'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab,version="1.0.0.0",culture="neutral",publicKeyToken="F9DE1EF17FB257B6",processorArchitecture="MSIL"'. Please refer to Help and Support for more information. HRESULT: 0x8007006F. assembly interface: IAssemblyCacheItem, function: Commit, component: {BE634139-4AD2-5A1B-9C96-B0D01F8F7AEB}
MSI (s) (1C:14) [22:32:24:210]: Note: 1: 2205 2: 3: Error
MSI (s) (1C:14) [22:32:24:211]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (s) (1C:14) [22:32:24:211]: Product: AA_GacInstaller -- Error 1935. An error occurred during the installation of assembly 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab,version="1.0.0.0",culture="neutral",publicKeyToken="F9DE1EF17FB257B6",processorArchitecture="MSIL"'. Please refer to Help and Support for more information. HRESULT: 0x8007006F. assembly interface: IAssemblyCacheItem, function: Commit, component: {BE634139-4AD2-5A1B-9C96-B0D01F8F7AEB}

As you see the HRESULT from IAssemblyCacheItem you get the Win32 error code 0x6F. What could that mean?

0x0000006F
ERROR_BUFFER_OVERFLOW

The file name is too long.

Do you remember the old MAX_PATH variable which has value of 256 characters. No full path on NTFS can be any longer than that. If you have a perfectly working application with .NET 3.5 and switch over to 4.0 everything will continue to work fine until you try to deploy the new version to the GAC. Installation does fail and you search for hours the fault on your side because a switch to a new .NET version requires many non trivial changes.

The innocent looking change of the GAC location did cause our application to break because we had a quite long assembly name (if this is a good idea is another topic) which did just work with .NET 3.5 but stopped working when the same assembly is compiled for .NET 4 and then deployed via MSI into the new GAC.

.NET Version	GAC Location
3.5	C:\Windows\assembly\GAC_MSIL\<Name>\<Version>__<hash>\<Name>.dll
4.0	C:\Windows\Microsoft.NET\assembly\GAC_MSIL\<Name>\v4.0_<Version>__<hash>\<Name>.dll

As you can see the .NET 4.0 GAC directory has an added “Microsoft.NET\” and “v4.0_“ for every .NET 4 assembly. This results in the loss of 19 characters for the complete path. Since the assembly name is part of the directory name as well we get a 9 character shorter maximal assembly name length with .NET 4.

.NET Version	Maximum Assembly Name Length in characters
2.0,3.5,3.5 SP1	99
4.0	90

Oh and while I am at it. What about this change?

Feature: Assembly loading

Differences from 3.5 SP1: To prevent redundant loading of assemblies and to save virtual address space, the CLR now loads assemblies by using only the Win32 MapViewOfFile function. It no longer also calls the LoadLibrary function.

This change affects diagnostic applications in the following ways:

A ProcessModuleCollection will no longer contain any modules from a class library (.dll file), as obtained from a call to Process.GetCurrentProcess().Modules.

Win32 applications that use the EnumProcessModules function will not see all managed modules listed.

Recommended changes: None

No changes necessary? Really? What about a diagnostic application that can switch on tracing while the app is running? The list of processes is filtered by only these processes which have loaded the diagnostic assembly in their process space. After a switch to .NET 4 I am quite certain that the process list will be empty. Since there is no documented way from MS how to list for all processes their file mappings there is no easy workaround possible. Of course I can simply create a named mutex or some other named kernel object but this suffers from security issues due to session 0 isolation. Only processes with the privilege SE_CREATE_GLOBAL_NAME are allowed to create cross session objects. This makes it e.g. very hard to switch on tracing for a non privileged system service from an interactive user session.

As a fun fact I do know the true story behind this change. A long time ago MS did develop Visual Studio 2010 and .NET 4.0. Everybody was adding features and all was running smoothly. Smoothly? One minor issue was that Visual Studio was crashing and running out of memory very quickly.

Hypothetical PM discussion with Dev:

Dev: Visual Studio runs out of memory.

PM: No problem. We will address this in a future version.

Dev: But we have got plenty of customer complaints about performance and footprint.

PM: Hm. Do we know what is causing this?

Dev: Nope.

PM: Hm.

…..

Nobody did know what was going on until someone realized that the CLR did load every assembly twice. One time via LoadLibrary and a second time as memory mapped file. That is no problem perf wise since the data is always read from the OS cache. But for a 32 bit application such a Visual Studio it meant that every dll consumed twice the virtual memory it needed to. The fix was to load every assembly only once as it is the case with .NET 4.

You can see this still today in every .NET 3.5 SP1 application with VMMap. There you can see in seconds that the assembly UnitTest.dll is loaded twice into different address spaces. With other tools such as a debugger it is nearly impossible to find such issues except if you do kernel debugging as your day job.

If you look deeper with VMMap you can find the following differences between .NET 3.5 SP 1 and .NET 4

.NET Framework Version	Behaviour
.NET 3.5 SP1	Every assembly is loaded twice. If a native image is present it is also loaded. Code is loaded up to 3 times in every process!
.NET 4.0	Every assembly is loaded only once. If a native image is present nothing from the GAC is loaded. Code is loaded only once in every process

What does this mean for my UI? If the application is NGened .NET 4 will load the native image via LoadLibrary as usual and it will show up as native image in the ProcessModuleCollection. The breaking change is luckily not so breaking since native images are still loaded the “old” way. One more reason to NGen every assembly to speed up warm startup.

With .NET 4.5 there is Managed Profile Guided Optimization possible that allows us to reduce cold startup times as well by a factor two and reduce the footprint also significantly. The trick is to profile the application during startup in a real product scenario and allow the optimizer to reorder the native image pages with the instrumented assembly which does contain profiling information.

The code is reordered so that the “hot path” used during startup is stored in subsequent pages which results in a linear read for the hard disc which is good. Exceptional code such as exception handlers are put far away into the cold pages which are hopefully not needed during startup. By reordering the native code (I am sure the are other tricks also applied) this way the application will use less physical memory since all loaded pages contain only code that will be executed during the startup. This was done already for the CLR assemblies of .NET 3.5 SP1 but with .NET 4.5 we get this cool tool also.

Share This Post:

Posted On Friday, September 23, 2011 10:13 AM | Feedback (0)

Friday, August 19, 2011

Memory Consumption: List<T> vs CompressedIndex

In my last post I did show how you can consume much less memory if you read values into a list which have often the same value. The IndexedList class uses internally a reference cache similar to the String.Intern method. But what if we do read value types such as int, enum, int64, … into a list which do repeat themself also very often? Can we store an enum which can have four values in less memory than an integer? Sure we can. Lets have a look at the following data

Row	Values
0	1
1	2
2	3
3	1
4	2
….

The enum in our sample data can take only 3 distinct values so we can use bit arrays to represent each value by its own bit array

Row	BitArray 1	BitArray 2	BitArray 3
0	1	0	0
1	0	1	0
2	0	0	1
3	1	0	0
4	0	1	0
….

This way we consume for each distinct value only one bit for each row. In the BCL there is a BitArray class implemented but I did find a Word Aligned BitArray implementation which claims to be much faster than the one of the BCL so I did use this one. If we try to compress a list of integers which has more than 32 distinct values in the data we do read then we would consume more memory as the original data. It is therefore a good idea to limit the number of bit arrays in use and switch back to a conventional list if more distinct values are found.

The best thing about this CompressedIndex is that although there may be many different values possible. Only for the actually occurring values we do allocate a bitvector. If you read e.g. a 500 MB data file which do contain the logs for a process with the id 5100 then we will allocate only one bit array for the value 5100. In the end we use only one bit for each row instead of 32 bits if we would have stored the process id as integer.

I do think you cannot get much better than this. But we have still plenty of room to optimize…

During my journey into the BCL I have found BitVector32 which was unkown to me until now. It does wrap an uint value which allows you to store up to 32 boolean flags in one BitVector32 instance. On the other hand it is no wonder why have never found this class. I did not care so deeply about memory until recently. The old C/C++ tricks are still the best and can be used in C# as well where necessary.

Here is the code for the CompressedIndex

    /// <summary>
    /// Create an index which uses for each distinct value a bitarray. More different values are stored than he has bits
    /// we switch to a plain list.
    /// </summary>
    /// <typeparam name="TKey">Storage type. Must be implementing GetHashCode and equals properly!</typeparam>
    class CompressedIndex<TKey> where TKey : struct
    {
        public CompressedIndex()
        {
            Type t = typeof(TKey);
            if (t.IsEnum) // Marshal can only get the size of primitive types. 
            {
                t = Enum.GetUnderlyingType(t);
            }

            _MaxBits = Marshal.SizeOf(t) * 8;
        }

        public void Add(int row, TKey key)
        {
            // we have more distinct values than the value has bits we need to swicht to a plain list
            if (_Cache != null && _Cache.Count > _MaxBits) 
            {
                ConvertCache();
            }

            if (_Cache != null)
            {
                // get bitarray for this value
                WAHBitArray bitArray;
                if (!_Cache.TryGetValue(key, out bitArray))
                {
                    bitArray = new WAHBitArray();
                    _Cache[key] = bitArray;
                }
                bitArray.Set(row, true);  // set at row index for this value the bitarray value to 1
                _MaxIndex++;
            }
            else
            {
                TKey value = key;
                _ListCache.Add(value);  // store for the same value the same reference to the list
            }
        }

        public TKey[] UniqueValues()
        {
            if (_Cache != null)
            {
                return _Cache.Keys.ToArray();
            }
            else
            {
                HashSet<TKey> valueTypeValues = new HashSet<TKey>();
                List<TKey> uniqueValues = new List<TKey>();
                foreach (var val in _ListCache)
                {
                    if( valueTypeValues.Add(val) )
                    {
                        uniqueValues.Add(val);
                    }
                }

                return uniqueValues.ToArray();
            }
        }

        public TKey this[int index]
        {
            get
            {
                if (_Cache != null)
                {
                    foreach (var kvp in _Cache) // get all bitarrays
                    {
                        if (kvp.Value.Get(index) == true) // The bitarray which has 1 marks the actual value
                        {
                            return kvp.Key;
                        }
                    }

                    return default(TKey);
                }
                else
                {
                    return _ListCache[index];
                }
            }
        }

        /// <summary>
        /// We have reached the limit. Switch back to List
        /// </summary>
        void ConvertCache()
        {
            var newCache = new List<TKey>(_MaxIndex + 1);
            for (int i = 0; i < _MaxIndex; i++)
            {
                newCache.Add(this[i]);
            }
            _ListCache = newCache;
            _Cache = null;
            _MaxIndex = 0;
        }

        /// <summary>
        /// Trim internal buffers to the exact count. After this
        /// operation it can be accessed read only by multiple threads.
        /// </summary>
        public void TrimExcess()
        {
            if (_ListCache != null)
            {
                _ListCache.TrimExcess();
            }

            if (_Cache != null)
            {
                foreach(var kvp in _Cache )
                {
                    // get last value which can cause the bitvector to resize itself
                    // to its final size. Only then the bitvector is thread safe for read only 
                    // access.
                    kvp.Value.Get(_MaxIndex); 
                    kvp.Value.TrimExcess();
                }
            }
        }

        Dictionary<TKey, WAHBitArray> _Cache = new Dictionary<TKey, WAHBitArray>();
        int _MaxIndex = 0;

        List<TKey> _ListCache;
        static int _MaxBits;  // threshold value 
    }

Share This Post:

Posted On Friday, August 19, 2011 2:36 AM | Feedback (0)

Thursday, August 18, 2011

Memory Consumption: List<T> vs IndexedList<T>

When you want to store millions of strings in a list you will notice in your memory profiler that the biggest memory consumers strings and lists are. If you e.g read many lines from a file with repetitive data inside it you can use a reference cache to consume the actual memory for the same string only once. The CLR offers String.Intern to do exactly that but the string reference cache is cross AppDomain static which is mostly never a good choice. Where could you use this? If you parse log files where many rows of data are often the same like type name, method name, … it can save you a lot of memory.

If you use a List<string> and intern all of your read strings you still store references in your list which will make the object graph more complicated and GC cycles slower. Besides this a reference does cost you 4/8 bytes in 32/64 bit processes. A generic read only index based list to which you simply add your data can make a huge difference. I could read a 200 MB log file in 17s by using a List<string> and it did drop to 10s after switching to IndexedList. Not bad for such a simple thing. But be warned it does only help in your specific case you you now that you are adding redundant data to it. IndexedList will consume twice as much memory compared to List<string> if all strings are different!

As added bonus you just have made searching inside your list a lot faster (at least for strings) by taking advantage of the fact that the same value is the same object. It is therefore enough to check for reference equality. String.Equals does this already so there is no need to use a custom comparer. But for custom objects you should check out if they do implement Equals properly.

As usual her comes the code:

    /// <summary>
    /// Memory conserving list which helps to spare memory if many identical values are added.
    /// Internally it uses integers to lookup its values from a hash table. The base structure is an integer list which 
    /// performs much better if millions of entries with only a few thousand distinct values are added. 
    /// </summary>
    /// <typeparam name="T">List Type</typeparam>
    class IndexedList<T> : IList<T> where T : class
    {
        Dictionary<T, T> _ReferenceCache = new Dictionary<T, T>();            // Use the same reference if the value is the same
        Dictionary<object, int> _Value2Index = new Dictionary<object, int>(); // use reference equality for dictionary
        Dictionary<int, T> _Index2Value = new Dictionary<int, T>();
        List<int> _List = new List<int>();                                    // The list holds at i the index which is the key to _Index2Value 

        public IndexedList()
        {
        }

        #region IList<T> Members

        public int IndexOf(T item)
        {
            throw new NotImplementedException();
        }

        public void Insert(int index, T item)
        {
            throw new NotImplementedException();
        }

        public void RemoveAt(int index)
        {
            throw new NotImplementedException();
        }

        public T this[int index]
        {
            get
            {
                return _Index2Value[_List[index]];
            }
            set
            {
                throw new NotImplementedException();
            }
        }

        #endregion

        #region ICollection<T> Members

        public void Add(T item)
        {
            T cachedReference = null;
            int idx = -1;
            if (!_ReferenceCache.TryGetValue(item, out cachedReference))
            {
                _ReferenceCache[item] = item;
                idx = _ReferenceCache.Count;
                _Index2Value[idx] = item;
                _Value2Index[item] = idx;
            }
            else
            {
                idx = _Value2Index[cachedReference];
            }
            _List.Add(idx);
        }

        public void Clear()
        {
            _Index2Value.Clear();
            _ReferenceCache.Clear();
            _Value2Index.Clear();
        }

        public bool Contains(T item)
        {
            return _ReferenceCache.ContainsKey(item);
        }

        public void CopyTo(T[] array, int arrayIndex)
        {
            throw new NotImplementedException();
        }

        public int Count
        {
            get { return _List.Count;  }
        }

        public bool IsReadOnly
        {
            get { return true; }
        }

        public bool Remove(T item)
        {
            throw new NotImplementedException();
        }

        #endregion

        #region IEnumerable<T> Members

        public IEnumerator<T> GetEnumerator()
        {
            throw new NotImplementedException();
        }

        #endregion

        #region IEnumerable Members

        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
        {
            throw new NotImplementedException();
        }

        #endregion

        internal IEnumerable<T> UniqueValues()
        {
            foreach (var key in _ReferenceCache.Keys)
            {
                yield return key;
            }
        }

        internal void TrimExcess()
        {
            _List.TrimExcess();
        }
    }

Share This Post:

Posted On Thursday, August 18, 2011 8:02 PM | Feedback (0)

How To Get Faster – Forwarding Empty Values

Last time we did have a look into the issues you get if you create many class instances. The more class instances you have the more complex is your object graph which means more work for the GC. If the GC has more work there is less CPU time for your application left which does make it slower. An easy way to fix this issue is to create a struct which is a value type which is copied by value and does not add complexity to your object graph. Common wisdom suggests that your structs should not get bigger much than the actual reference type size (16 bytes on 32 bit, 24 bytes for 64 bit). This quite big limitation renders structs useless as general purpose data access object containers. Really?

The key insight is that we do not need to store the actual data inside the value type. We must only be able to access it from our struct. This way we can employ the struct as forwarder which does not contain the actual data but uses indexes to retrieve the actual instance data from a DataProvider object.

    public class DataProviderSimple : IList<ForwardingEmptyValue>
    {
        internal List<double> Value1 = new List<double>();
        internal List<int> Value2 = new List<int>();

        #region IList<ForwardingEmptyValue> Members
        public ForwardingEmptyValue this[int index]
        {
            get
            {
                return new ForwardingEmptyValue(this, index); // return forwarding struct to save memory
            }
            set
            {
                throw new NotImplementedException();
            }
        }
    }

    public struct ForwardingEmptyValue
    {
        public ForwardingEmptyValue(DataProviderSimple provider, int row)
        {
            _Provider = provider;
            _Row = row;
        }

        public double V1
        {
            get { return _Provider.Value1[_Row]; }
        }

        public int V2
        {
            get { return _Provider.Value2[_Row]; }
        }

        DataProviderSimple _Provider;
        int _Row;
    }

Ok this looks better but we have not gained much since we still have a reference to our provider object. The best solution I came up with so far was to replace the reference to the data provider with a byte which is an index to a static list of data provider objects. We still need to know from which row to read our data from which will result in a 5 byte value type object which can give you an arbitrary amount of data.

Here is my current version of a forwarding struct:

    [StructLayout(LayoutKind.Sequential, Pack=1)]
    public struct ForwardingEmptyValueStruct
    {
        int _Row;
        byte _ProviderIdx;
        

        public ForwardingEmptyValueStruct(byte providerIdx, int row)
        {
            _ProviderIdx = providerIdx;
            _Row = row;
        }

        public double V1
        {
            get { return DataProvider._DataProviders[_ProviderIdx].Value1[_Row];  }
        }

        public int V2
        {
            get { return DataProvider._DataProviders[_ProviderIdx].Value2[_Row];  }
        }
    }

I had to tweak the layout of the struct a bit since .NET has the habit to fill any gaps between natural alignment with zeros which would have resulted in an 8 byte struct instead of 5 bytes. As you can see there are no more object references left which will come in handy if we have millions of instances hanging around. As a general rule are index based data structures more more GC friendly than reference (read pointer) based structures. The Microsoft guys do know this of course which explains why highly efficient data structures like Dictionary<T> stores its values internally in index based structs.

[StructLayout(LayoutKind.Sequential)]
private struct Entry
{
    public int hashCode;
    public int next;
    public TKey key;
    public TValue value;
}

The differences can be substantial if your application is in a high throughput scenarios where you read large amounts of data to calculate some intermediate value (statistics, filter, …) and discard the data immediately after it had been processed. To illustrate this fact I have written a DataProvider which uses a forwarding struct as its DAO object and another one which uses a forwarding class object.

***** UnitTests.DataProviderTests.Read_Via_Class
Did read 10000000 lines in 2,96s Managed Memory: 320.017.224

***** UnitTests.DataProviderTests.Read_Via_Struct
Did read 10000000 lines in 0,84s, Managed Memory: 170.004.248

There is over a factor 3 difference between these two if you are in a high GC scenario. The actual benefit may range from 10% up to factors.

The test I did was

        [Test]
        public void Read_Via_Struct()
        {
            var sw = Stopwatch.StartNew();
            long start = GC.GetTotalMemory(true);
            using (var prov = new DataProvider(NumericData))
            { 
                prov.ReadData();
                Console.WriteLine("Reading did take: {0:F2}", sw.Elapsed.TotalSeconds);
                sw = Stopwatch.StartNew();
                var array = CopyToArray(prov);
                DoGCHeavyThings();
                sw.Stop();
                Console.WriteLine("Did read {0} lines in {1:F2}s, Managed Memory: {2:N0}", prov.Count, sw.Elapsed.TotalSeconds, GC.GetTotalMemory(true)-start);
            }
        }

        T[] CopyToArray<T>(IList<T> data)
        {
            T[] ret = new T[data.Count];
            data.CopyTo(ret, 0);
            return ret;
        }

        private void DoGCHeavyThings()
        {
            List<List<string>> lists = new List<List<string>>();
            for (int i = 0; i < 200 * 1000; i++)
            {
                var lst =  new List<string>(new string [] { new string("Test".ToCharArray()) } );
                lists.Add(lst);
            }
        }

Besides the much better speed we need with the forwarding struct solution needed nearly half of the memory since for values type the 12 (20 on x64) byte object header for class instances does not exist. This may be a major factor if you read millions of rows from a large data file.

Below is the source code for a real world DataProvider which reads data from a file which has one double and int per row.

1,1 0
2,1 1
3,1 2
4,1 3
5,1 4

….

I have found it useful to implement IList so I can read the n-th object via the index operator in an intuitive manner. Besides this I can change the underlying data structures much easier if I use the .Count and [] as prime data access method. So far we have only dealt with value types like int or double inside our test files. If we do know more about the actual structure of the data we do read we can save much more memory than already. I have written a simple (okok but it is fast as hell) trace viewer which uses this method and I could beat an already optimized viewer in terms of reading speed by a factor 6 (thanks to the simplified object model and other tricks) and memory consumption by a factor 3. Next time we have a look at the data we read and how to reduce the memory consumption with bit vectors, reference caches and indexed lists.

Below is the implementation of a simple DataProvider which does use the defined ForwardingEmptyValueStruct as data access object. You can store the structs inside an array, sort them and do everything you like like with usual classes. The forwarding struct has reference type semantics which means if you mutate the data through the struct all others will see the change as well since the data is not part of the struct.

    /// <summary>
    /// Read a file which has a double and integer value in each line separated by a space.
    /// The read data can be accessed via the index operator. The returned values are forwarding
    /// structs to use as little memory as possible. After you are done you should dispose the dataprovider as usual.
    /// </summary>
    public class DataProvider : IList<ForwardingEmptyValueStruct>, IDisposable
    {
        internal List<double> Value1 = new List<double>();
        internal List<int> Value2 = new List<int>();

        static object _Lock = new object();
        internal static DataProvider[] _DataProviders = new DataProvider[0]; // static list of data providers
        byte _ThisIdx;                                                       // index to _Dataproviders array of current instance
        FileStream _dataSource;                                              // Opened file

        public DataProvider(string file)
        {
            _dataSource = new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
            AddThis();
        }

        unsafe public void ReadData()
        {
            string line;
            double v1;
            int v2;
            using (StreamReader reader = new StreamReader(_dataSource))
            {
                while ((line = reader.ReadLine()) != null)
                {
                    int space = 0;
                    fixed (char* pStr = line)
                    {
                        for (int i = 0; i < line.Length; i++)
                        {
                            if (pStr[i] == ' ')
                            {
                                space = i;
                                break;
                            }
                            else if (pStr[i] == ',')
                            {
                                pStr[i] = '.';
                            }
                        }

                        if (space != line.Length)
                        {
                            v1 = double.Parse(new string(pStr,0, space));
                            v2 = IntParseFast(pStr+space, line.Length-(space+1));
                            Value1.Add(v1);
                            Value2.Add(v2);
                        }
                    }

         
                }
            }
            Value1.TrimExcess();
            Value2.TrimExcess();
        }


        unsafe static int IntParseFast(char *pStr, int len)
        {
            int result = 0;
            for (int i = 0; i < len; i++)
            {
                char letter = pStr[i];
                result = 10 * result + (letter - 48);
            }
            return result;
        }

        private void AddThis()
        {
            lock (_Lock)
            {
                var prov = new List<DataProvider>(_DataProviders);

                if (prov.Count != byte.MaxValue)
                {
                    _ThisIdx = (byte)prov.Count;
                    prov.Add(this);
                }
                else // more than 256 files open 
                {
                    int tmpIdx = prov.FindIndex(x => x == null);
                    if (tmpIdx == -1)
                    {
                        throw new NotSupportedException("The dataprovider does support only 256 simultanously open files. If you did not expect this error search for missing dispose calls for unused provider objects.");
                    }
                    _ThisIdx = (byte)tmpIdx;
                    prov[_ThisIdx] = this;

                }
                _DataProviders = prov.ToArray();
            }
        }

        #region IDisposable Members

        public void Dispose()
        {
            lock (_Lock)
            {
                _dataSource.Close();
                _DataProviders[_ThisIdx] = null;
                this.Value1 = null;
                this.Value2 = null;
                if (_DataProviders.All(x => x == null))  // no more files open, reset byte array
                {
                    _DataProviders = new DataProvider[0];
                }
            }
        }

        #endregion



        #region IList<ForwardingEmptyValueStruct> Members

        public int IndexOf(ForwardingEmptyValueStruct item)
        {
            throw new NotImplementedException();
        }

        public void Insert(int index, ForwardingEmptyValueStruct item)
        {
            throw new NotImplementedException();
        }

        public void RemoveAt(int index)
        {
            throw new NotImplementedException();
        }

        public ForwardingEmptyValueStruct this[int index]
        {
            get
            {
                return new ForwardingEmptyValueStruct(_ThisIdx, index); // return forwarding struct to save memory
            }
            set
            {
                throw new NotImplementedException();
            }
        }

        #endregion

        #region ICollection<ForwardingEmptyValueStruct> Members

        public void Add(ForwardingEmptyValueStruct item)
        {
            throw new NotImplementedException();
        }

        public void Clear()
        {
            Value1.Clear();
            Value2.Clear();
        }

        public bool Contains(ForwardingEmptyValueStruct item)
        {
            throw new NotImplementedException();
        }

        public void CopyTo(ForwardingEmptyValueStruct[] array, int arrayIndex)
        {
            for (int i = arrayIndex; i < this.Count; i++)
            {
                array[i] = this[i];
            }
        }

        public int Count
        {
            get { return Value1.Count; }
        }

        public bool IsReadOnly
        {
            get { return true; }
        }

        public bool Remove(ForwardingEmptyValueStruct item)
        {
            throw new NotImplementedException();
        }

        #endregion

        #region IEnumerable<ForwardingEmptyValueStruct> Members

        public IEnumerator<ForwardingEmptyValueStruct> GetEnumerator()
        {
            for (int i = 0; i < this.Count; i++)
            {
                yield return this[i];
            }
        }

        #endregion

        #region IEnumerable Members

        System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
        {
            throw new NotImplementedException();
        }

        #endregion
    }

Share This Post:

Posted On Thursday, August 18, 2011 11:24 AM | Feedback (0)

Saturday, August 13, 2011

Do You Know The Costs Of Garbage?

If you want to write scalable applications with a high throughput you need to take care of good data structure design to make your application scale. If you want to read for example 200 MB data from a file and process it you can finish it in 5 or 30s. To be on the 5s side it is educational to see how fast you actually can get. To measure the raw disc performance I do read the file into a pre allocated byte array.

[Test]
public void ReadFile_Store_In_ByteArray()
{
    var sw = Stopwatch.StartNew();
    int read = 0;
    int numRead = 0;
    using (FileStream rStream = new FileStream(TestData, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 4096, FileOptions.SequentialScan))
    {
        byte[] bytes = new byte[210 * 1024 * 1024];

        while ((numRead = rStream.Read(bytes, read, 4096)) > 0)
        {
            read += numRead;
        }
    }
    Console.WriteLine("Did read {0:N0} bytes in {1:F2}s", read, sw.Elapsed.TotalSeconds);
}

***** UnitTests.PushingLimits.ReadFile_Store_In_ByteArray
Did read 216.000.003 bytes in 0,60s

Not bad. The Windows cache does deliver 360 MB/s to my application. If I read the file the first time I see the real disc performance which is around 160 MB which is still quite good for my Intel PostVille X25M. One common myth many believe in is that if you are reading from disc you can process the data much faster than you can read. Lets try it out. This time we do simply read the file line by line and store the lines in a List<string>.

[Test]
public void ReadFile_Store_InList()
{
    var sw = Stopwatch.StartNew();
    FileStream rStream = new FileStream(TestData, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 4096, FileOptions.SequentialScan);
    int lineNo = 0;
    List<string> lines = new List<string>(2*1000*1000);
    using (StreamReader reader = new StreamReader(rStream))
    {
        string line;
        while ((line = reader.ReadLine()) != null)
        {
            lineNo++;
            lines.Add(line);
        }
    }
    Console.WriteLine("Did read {0:N0} lines in {1:F2}s", lineNo, sw.Elapsed.TotalSeconds);
}

***** UnitTests.PushingLimits.ReadFile_Store_In_List
Did read 2.000.000 lines in 4,92s

Why the heck ware we over 8 times slower? I did not do anything with them! Ok to read form the files the strings we need a StreamReader which needs to convert the byte array into a char array and then into a string. The input file was an ASCII file which means that while we convert a 200 MB file into strings we consume 400 MB of memory because .NET strings are UTF-16. These things do certainly cost us some performance but not 8 times. Lets do the same but this time we do only count the lines of the file.

[Test]
public void ReadFile()
{
    var sw = Stopwatch.StartNew();
    FileStream rStream = new FileStream(TestData, FileMode.Open, FileAccess.Read, FileShare.ReadWrite, 4096, FileOptions.SequentialScan);
    int lineNo = 0;
    using (StreamReader reader = new StreamReader(rStream))
    {
        string line;
        while ((line = reader.ReadLine()) != null)
        {
            lineNo++;
        }
    }
    Console.WriteLine("Did read {0:N0} lines in {1:F2}s", lineNo, sw.Elapsed.TotalSeconds);
}

***** UnitTests.PushingLimits.ReadFile
Did read 2.000.000 lines in 1,57s

Thats more like it. From 0,6s to 1,6s is still slower but here we do see what decoding the byte array into .NET strings actually cost. It is not as bad as it looks since we did not yet process (e.g. parse, convert,…) the lines yet which does also add a significant amount of time. But why are we getting over 3 times faster when we throw away our read data instead of keeping it? The Garbage Collector should not have any work to do if we keep all our data in a list. As it turns out this is the point.

The GC has a lot more to do if we store the strings in a list since we add 2 million string references to an array. The GC kicks in from time to time to clean up any garbage and needs to traverse the complete object graph which just becomes more and more complex if we add more managed references to our list.

If your managed application consumes over a few hundred MB of memory it will become much slower if the data is stored in the object graph which needs to be traversed by the GC. It is as simple as that. The more managed references you add the more work the GC has to do for each full GC. To prove this point lets look what happens if we switch from classes to structs which are value types and not under GC control.

First we define a class which has 10 long values stored

class ClassContainer
{
    public ClassContainer(long value)
    {
        v1 = value++;
        v2 = value++;
        v3 = value++;
        v4 = value++;
        v5 = value++;
        v6 = value++;
        v7 = value++;
        v8 = value++;
        v9 = value++;
        v10 = value++;
    }
    long v1,v2,v3,v4,v5,v6,v7,v8,v9,v10;
}

A long value consumes 8 bytes which attributes to 80 bytes of payload of our class.

[Test]
public void Create_Class_List()
{
    var start = GC.GetTotalMemory(true);
    var sw = Stopwatch.StartNew();
    var arr = new ClassContainer[N];
    for (int i = 0; i < arr.Length; i++)
    {
        arr[i] = new ClassContainer(i);
    }
    sw.Stop();
    Console.WriteLine("Did insert {0:N0} classes into list. {1:F2}s, GC Memory: {2:N0}",
                        arr.Length, sw.Elapsed.TotalSeconds, GC.GetTotalMemory(true)-start);
}

***** UnitTests.StructVsClass.Create_Class_List
Did insert 5.000.000 classes into list. 3,74s, GC Memory: 459.996.312

Now lets to the same thing for a struct with the same content.

[Test]
public void Create_Struct_List()
{
    var start = GC.GetTotalMemory(true);
    var sw = Stopwatch.StartNew();
    var arr = new StructContainer[N];
    for (int i = 0; i < arr.Length; i++)
    {
        arr[i] = new StructContainer(i);
    }
    sw.Stop();
    Console.WriteLine("Did insert {0:N0} structs into list. {1:F2}s, GC Memory: {2:N0}",
        arr.Length, sw.Elapsed.TotalSeconds, GC.GetTotalMemory(true)-start);

Console.WriteLine("Class Memory = N*(10*8+Reference+9) = {0:N0}, Struct Memory = N*10*8 = {1:N0}",
N * (10 * 8 + IntPtr.Size+9), N * 10 * 8);
}

***** UnitTests.StructVsClass.Create_Struct_List
Did insert 5.000.000 structs into list. 0,58s, GC Memory: 400.00.096
Class Memory = N*(10*8+Reference+9) = 465.000.000, Struct Memory = N*10*8 = 400.000.000

Although we do insert the same amount of data the performance does differ over a factor 6! This is the price you pay if the GC needs to do more work. Things will become worse if you are a 64 bit application with several GB of memory consumption. Did you notice that (especially managed) applications with a high memory consumption react slowly on user input? It could be that the application does not do much but the GC brings it down. If you want to write an application with a high data throughput and high memory consumption you will hit a wall if you do not take care to design your application more GC friendly.

To declare all classes as structs is not the solution since you are then copying the data by value with every function call you pass the data around which is for my 80 byte structure a quite big overhead. Luckily you can create data structures which do play well with the GC. I will give you some unique data structures in my next posts. It is possible to use structures as general purpose data container with very little overhead …

Share This Post:

Posted On Saturday, August 13, 2011 12:03 AM | Feedback (1)

Saturday, July 02, 2011

About MSI Product/Upgrade/Package Codes

When you generate an MSI you need to set at least

Upgrade code is not mandatory but in reality it is also a must or you will not be able to create upgrade msi packages. One interesting thing about ProductVersion is that you can choose to set a 4 digit version like 1.1.1.0 but MSI will only use the first three digits for version checking. Its format is major.minor.build. This is important if you decide to create a update or upgrade package. Have a look for the rules when to change version and product code. The codes are all in form of a guid like component ids of the form

{12345678-ABCD-EF12-1234-123456789012}

Allowed characters are 0-9 and A-F. The codes must contain only upper case letters. The behavior is undefined if you do not stick to this convention. To generate such a guid you can use Guidgen from Visual Studio. Beware of Guidgen versions previous to VS2010. They had the issue that from time to time it smuggled lower case letters into your guid.

Alternatively you can use Powershell:

"{"+[System.Guid]::NewGuid().ToString().ToUpper()+"}"

Package Code

The easiest code is Package Code. It does change every time you create a new package. When you install a package and generate a new one with <insert your favorite packager here> and then you try to uninstall it by right clicking on the new package you get an error box that another version of the product is already installed. This is the only reason I know of why the package code does exist.

Product Code

The product code identifies your msi with a unique guid. It must be the same for all patch versions (the first two digits of the version are the same). When you create a new product version e.g. a 2.0 and want to install it side by side with the old version you will not succeed. MSI will tell you that another version of your product is already installed.

Upgrade Code

Since you know know that the product code needs to be changed between major versions we need another code to identify a product line. Imagine e.g. Office 2003 and Office 2007. Both major versions need to be serviced (patched and upgraded) independently. Here you assign Office 2003 one upgrade code and Office 2007 another one. Now you can create patch and upgrade versions for both product lines independently although each office upgrade will get a new product code. All Office 2003 msi files regardless if first release, patch or upgrade version will receive the same upgrade code to enable upgrading of any previously installed Office 2003 version.

32/64 Bit Issues

What could possibly go wrong? Well it turns out a lot if you do not pay close attention to your product and upgrade codes.

Product Code

If you release a 32 and 64 bit version of your msi but you do not change the product code between versions you will find out that you can only install either the 32 or the 64 bit version but not both.

Upgrade Code

If you create two unrelated msi files with the same upgrade code (copy and paste is soo easy) and try to install both you will find that the second installation will uninstall the first msi. Why? If your msi files do contain an upgrade table, a FindRelatedProducts and RemoveExistingProducts action then your second msi will find a previous product with the same upgrade code installed and will simply replace it.

If you have the same upgrade code for 32/64 bit versions of your msi then you will run into the scenario above. This is especially nice if a 32 bit msi tries to uninstall a 64 bit msi which are installed into Program Files and Program Files(x86). Windows file system redirection will redirect changes from a 32 bit msi process from Program Files to Program Files(x86) which causes totally undefined behavior. After the “upgrade” from a 64 bit version to a 32 bit version of your application both versions will be unusable and the installer database is almost certainly corrupt.

Component Id

Yes component ids must also be different between 32/64 bit versions of your msi. This can also cause funny things like left over files after uninstallation of your 32 bit msi because the 64 bit msi does still reference the component although the component does point to different files residing under Program Files and Program Files(x86).

Conclusion

It is a very good idea to change all guids (product code, upgrade code, component ids) between a 32 and 64 bit build of your msi or you will experience funny errors. One way to get around these issues is to generate these guids automatically by your packaging tool so you do not forget. A very easy “algorithm” is to increase the first digit of any guid for a 64 bit msi build to prevent such issues.

Share This Post:

Posted On Saturday, July 02, 2011 9:24 AM | Feedback (3)

Thursday, June 23, 2011

Future Of F# At Jazoon 2011

I was at the Jazoon 2011 in Zurich (Switzerland). It was a really cool event and it had many top notch speaker not only from the Microsoft universe. One of the most interesting talks was from Don Syme with the title: F# Today/F# Tomorrow. He did show how to use F# scripting to browse through open databases/, OData Web Services, Sharepoint, …interactively. It looked really easy with the help of F# Type Providers which is the next big language feature in a future F# version.

The object returned by a Type Provider is used to access the data like in usual strongly typed object model. No guessing how the property of an object is called. Intellisense will show it just as you expect. There exists a range of Type Providers for various data sources where the schema of the stored data can somehow be dynamically extracted. Lets use e.g. a free database it would be then

let data = DbProvider(http://.....);

data the object which contains all data from e.g. a chemical database. It has an elements collection which contains an element which has the properties: Name, AtomicMass, Picture, …. You can browse the object returned by the Type Provider with full Intellisense because the returned object is strongly typed which makes this happen.

The same can be achieved of course with code generators that use an input the schema of the input data (OData Web Service, database, Sharepoint, JSON serialized data, …) and spit out the necessary strongly typed objects as an assembly.

This does work but has the downside that if the schema of your data source is huge you will quickly run against a wall with traditional code generators since the generated “deserialization” assembly could easily become several hundred MB.

*** The following part contains guessing how this exactly work by asking Don two questions ****

Q: Can I use Type Providers within C#?

D: No.

Q: F# is after all a library. I can reference the F# assemblies and use the contained Type Providers?

D: F# does annotate the generated types in a special way at runtime which is not a static type that C# could use.

The F# type providers seem to use a hybrid approach. At compilation time the Type Provider is instantiated with the url of your input data. The obtained schema information is used by the compiler to generate static types as usual but only for a small subset (the top level classes up to certain nesting level would make sense to me). To make this work you need to access the actual data source at compile time which could be a problem if you want to keep the actual url in a config file.

Ok so this explains why it does work at all. But in the demo we did see full intellisense support down to the deepest object level. It looks like if you navigate deeper into the object hierarchy the type provider is instantiated in the background and attach to a true static type the properties determined at run time while you were typing. So this type is not really static at all. It is static if you define as a static type that its properties shows up in intellisense. But since this type information is determined while you are typing and it is not used to generate a true static type and you cannot use these “intellistatic” types from C#.

Nonetheless this is a very cool language feature. With the plotting libraries you can generate expressive charts from any datasource within seconds to get quickly an overview of any structured data storage.

My favorite programming language C# will not get such features in the near future there is hope. If you restrict yourself to OData sources you can use LINQPad to query any OData enabled data source with LINQ with ease.

There you can query Stackoverflow with

The output is also nicely rendered which makes it a very good tool to explore OData sources today.

Share This Post:

Posted On Thursday, June 23, 2011 9:29 AM | Feedback (1)

Friday, June 17, 2011

What Every Developer Should Know About MSI Components

Hopefully nothing. But if you have to do more than simple XCopy deployment and you need to support updates, upgrades and perhaps side by side scenarios there is no way around MSI. You can create Msi files with a Visual Studio Setup project which is severely limited or you can use the Windows Installer Toolset. I cannot talk about WIX with my German colleagues because WIX has a very special meaning. It is funny to always use the long name when I talk about deployment possibilities. Alternatively you can buy commercial tools which help you to author Msi files but I am not sure how good they are. Given enough pain with existing solutions you can also learn the MSI Apis and create your own packaging solution. If I were you I would use either a commercial visual tool when you do easy deployments or use the free Windows Installer Toolset. Once you know the WIX schema you can create well formed wix xml files easily with any editor. Then you can “compile” from the wxs files your Msi package.

Recently I had the “pleasure” to get my hands dirty with C++ (again) and the MSI technology. Installation is a complex topic but after several month of digging into arcane MSI issues I can safely say that there should exist an easier way to install and update files as today. I am not alone with this statement as John Robbins (creator of the cool tool Paraffin) states: “.. It's a brittle and scary API in Windows …”. To help other people struggling with installation issues I present you the advice I (and others) found useful and what will happen if you ignore this advice.

What is a MSI file?

A MSI file is basically a database with tables which reference each other to control how your un/installation should work. The basic idea is that you declare via these tables what you want to install and MSI controls the how to get your stuff onto or off your machine.

Your “stuff” consists usually of files, registry keys, shortcuts and environment variables. Therefore the most important tables are File, Registry, Environment and Shortcut table which define what will be un/installed. The key to master MSI is that every resource (file, registry key ,…) is associated with a MSI component.

The actual payload consists of compressed files in the CAB format which can either be embedded into the MSI file or reside beside the MSI file or in a subdirectory below it. To examine MSI files you need Orca a free MSI editor provided by MS. There is also another free editor called Super Orca which does support diffs between MSI and it does not lock the MSI files. But since Orca comes with a shell extension I tend to use only Orca because it is so easy to right click on a MSI file and open it with this tool.

How Do I Install It?

Double click it. This does work for fresh installations as well as major upgrades. Updates need to be installed via the command line via

msiexec /i <msi> REINSTALL=ALL REINSTALLMODE=vomus

This tells the installer to reinstall all already installed features (new features will NOT be installed). The reinstallmode letters do force an overwrite of the old cached package in the %WINDIR%\Installer folder. All files, shortcuts and registry keys are redeployed if they are missing or need to be replaced with a newer version. When things did go really wrong and you want to overwrite everything unconditionally use REINSTALLMODE=vamus.

How To Enable MSI Logs?

You can download a MSI from Microsoft which installs some registry keys to enable full MSI logging. The log files can be found in your %TEMP% folder and are called MSIxxxx.log. Alternatively you can add to your msiexec command line the option

msiexec …. /l*vx <LogFileName>

Personally I find it rather strange that * does not mean full logging. To really get all logs I need to add v and x which is documented in the msiexec help but I still find this behavior unintuitive.

What are MSI components?

The whole MSI logic is bound to the concept of MSI components. Nearly every msi table has a Component column which binds an installable resource to a component. Below are the screenshots of the FeatureComponents and Component table of an example MSI.

The Feature table defines basically the feature hierarchy. To find out what belongs to a feature you need to look at the FeatureComponents table where for each feature the components are listed which will be installed when a feature is installed. The MSI components are defined in the Component table. This table has as first column the component name and as second column the component id which is a GUID. All resources you want to install belong to a MSI component. Therefore nearly all MSI tables have a Component_ column which contains the component name.

If you look e.g. a the File table you see that every file belongs to a component which is true for all other tables which install resources.

The component table is the glue between all other tables which contain the resources you want to install. So far so easy. Why is MSI then so complex? Most MSI problems arise from the fact that you did violate a MSI component rule in one or the other way.

When you install a feature the reference count for all components belonging to this feature will increase by one. If your component is installed by more than one feature it will get a higher refcount. When you uninstall a feature its refcount will drop by one.

Interesting things happen if the component reference count reaches zero: Then all associated resources will be deleted. That looks like a reasonable thing and it is. What it makes complex are the strange component rules you have to follow. Below are some important component rules from the Tao of the Windows Installer

…

Rule 16: Follow Component Rules

Components are a very important part of the Installer technology. They are the means whereby the Installer manages the resources that make up your application. The SDK provides the following guidelines for creating components in your package:

Never create two components that install a resource under the same name and target location. If a resource must be duplicated in multiple components, change its name or target location in each component. This rule should be applied across applications, products, product versions, and companies.

Two components must not have the same key path file. This is a consequence of the previous rule. The key path value points to a particular file or folder belonging to the component that the installer uses to detect the component. If two components had the same key path file, the installer would be unable to distinguish which component is installed. Two components however may share a key path folder.

Do not create a version of a component that is incompatible with all previous versions of the component. This rule should be applied across applications, products, product versions, and companies.

Do not create components containing resources that will need to be installed into more than one directory on the user’s system. The installer installs all of the resources in a component into the same directory. It is not possible to install some resources into subdirectories.

Do not include more than one COM server per component. If a component contains a COM server, this must be the key path for the component.

Do not specify more than one file per component as a target for the Start menu or a Desktop shortcut.

…

And these rules do not even talk about component ids, update packages and upgrades which you need to understand as well. Lets suppose you install two MSIs (MSI1 and MSI2) which have the same ComponentId but different component names. Both do install the same file. What will happen when you uninstall MSI2?

Hm the file should stay there. But the component names are different. Yes and yes. But MSI uses not use the component name as key for the refcount. Instead the ComponentId column of the Component table which contains a GUID is used as identifier under which the refcount is stored. The components Comp1 and Comp2 are identical from the MSI perspective. After the installation of both MSIs the Component with the Id {100000….} has a refcount of two. After uninstallation of one MSI there is still a refcount of one which drops to zero just as expected when we uninstall the last msi. Then the file which was the same for both MSIs is deleted.

You should remember that MSI keeps a refcount across MSIs for components with the same component id.

MSI does manage components not the resources you did install. The resources associated with a component are then and only then deleted when the refcount of the component reaches zero.

The dependencies between features, components and resources can be described as relations. m,k are numbers >= 1, n can be 0. Inside a MSI the following relations are valid

Feature	1 –> n Components
Component	1 –> m Features
Component	1 –> k Resources

These relations express that one feature can install several components and features can share components between them. Every (meaningful) component will install at least one resource which means that its name (primary key to stay in database speak) does occur in some other table in the Component column as value which installs some resource.

Lets make it clear with an example. We want to install with the feature MainFeature some files a registry key and a shortcut. We can then create components Comp1..3 which are referenced by the resources defined in the corresponding tables.

Feature	Component	Registry	File	Shortcuts
MainFeature	Comp1	RegistryKey1
MainFeature	Comp2		File.txt
MainFeature	Comp3		File2.txt	Shortcut to File2.txt

It is illegal that the same resource is part of more than one component since this would break the refcount mechanism. Lets illustrate this:

Feature	ComponentId	Resource	Reference Count
Feature1	{1000-…}	File1.txt	1
Feature2	{2000-….}	File1.txt	1

The installation part works well but what happens when you uninstall Feature2? Component {20000…} gets a refcount of zero where MSI deletes all resources belonging to this component.

In this case File1.txt will be deleted. But Feature1 still has another component {10000…} with a refcount of one which means that the file was deleted too early. You just have ruined your installation. To fix it you then need to click on the Repair button under Add/Remove Programs to let MSI reinstall any missing registry keys, files or shortcuts.

The vigilant reader might has noticed that there is more in the Component table. Beside its name and GUID it has also an installation directory, attributes and a KeyPath. The KeyPath is a reference to a file or registry key which is used to detect if the component is already installed. This becomes important when you repair or uninstall a component. To find out if the component is already installed MSI checks if the registry key or file referenced by the KeyPath property does exist. When it does not exist it assumes that it was either already uninstalled (can lead to problems during uninstall) or that it is already installed and all is fine.

Why is this detail so important? Lets put all files into one component. The KeyPath should be then one of the files of your component to check if it was installed or not. When your installation becomes corrupt

because a file was deleted you cannot repair it with the Repair button under Add/Remove Programs because MSI checks the component integrity via the Resource referenced by its KeyPath. As long as you did not delete the KeyPath file MSI thinks all resources with your component are installed and never executes any repair action.

You get even more trouble when you try to remove files during an upgrade (you cannot remove files during an update) from your super component which contains all files. The only way out and therefore

best practice is to assign for every resource you want to install an extra component. This ensures painless updatability and repairs and you have much less effort to remove specific files during an upgrade.

In effect you get this best practice relation

Feature	1 –> n Components
Component	1 –> 1 Resources

MSI Component Rules

Rule 1 – One component per resource

Every resource you want to install (file, registry key, value, environment value, shortcut, directory, …) must get its own component which does never change between versions as long as the install location is the same.

Penalty

If you add more than one resources to a component you will break the repair capability of MSI because the KeyPath is used to check if the component needs repair.

MSI	ComponentId	Files
MSI 1.0	{1000}	File1-5
MSI 2.0	{2000}	File2-5

You want to remove File1 in version 2.0 of your MSI. Since you want to keep the other files you create a new component and add them there. MSI will delete all files if the component refcount of {1000} drops to zero. The files you want to keep are added to the new component {2000}. Ok that does work if your upgrade does uninstall the old MSI first. This will cause the refcount of all previously installed components to reach zero which means that all files present in version 1.0 are deleted.

But there is a faster way to perform your upgrade by first installing your new MSI and then remove the old one. If you choose this upgrade path then you will loose File1-5 after your upgrade and not only File1 as intended by your new component design.

Rule 2 – Only add, never remove resources from a component

If you did follow rule 1 you will not need Rule 2. You can add in a patch more resources to one component. That is ok. But you can never remove anything from it. There are tricky ways around that but I do not want to encourage bad component design.

Penalty

Lets assume you have 2 MSI files which install under the same component one file

MSI1	MSI2
{1000} - ComponentId	{1000} – ComponentId
File1.txt	File2.txt

When you install and uninstall both MSIs you will end up with an installation where either File1 or File2 will be left. Why? It seems that MSI does not store the resources associated with each component in its internal database. Instead Windows will simply query the MSI that is currently uninstalled for all resources belonging to this component. Since it will find only one file and not two it will only uninstall one file. That is the main reason why you never can remove resources from a component!

Rule 3 Never Remove A Component From an Update MSI.

This is the same as if you change the GUID of a component by accident for your new update package. The resulting update package will not contain all components from the previously installed package.

Penalty

When you remove a component from a feature MSI will set the feature state during update to Advertised and log a warning message into its log file when you did enable MSI logging.

SELMGR: ComponentId '{2DCEA1BA-3E27-E222-484C-D0D66AEA4F62}' is registered to feature 'xxxxxxx, but is not present in the Component table. Removal of components from a feature is not supported!

MSI (c) (24:44) [07:53:13:436]: SELMGR: Removal of a component from a feature is not supported

Advertised means that MSI treats all components of this feature as not installed. As a consequence during uninstall nothing will be removed since it is not installed! This is not only bad because uninstall does no longer work but this feature will also not get the required patches. All other features which have followed component versioning rules for update packages will be updated but the one faulty feature will not. This results in very hard to find bugs why an update was only partially successful.

Things got better with Windows Installer 4.5 but you cannot rely on that nobody will use an older installer. It is a good idea to add to your update msiexec call MSIENFORCEUPGRADECOMPONENTRULES=1 which will abort the installation if you did violate this rule.

Share This Post:

Posted On Friday, June 17, 2011 11:07 AM | Feedback (4)

Sunday, March 06, 2011

Find The Bug

What does this code print and why?

            HashSet<int> set = new HashSet<int>();

            int[] data = new int[] { 1, 2, 1, 2 };

            var unique = from i in data

                         where set.Add(i)

                         select i;

  // Compiles to: var unique = Enumerable.Where(data, (i) => set.Add(i));

            foreach (var i in unique)

                Console.WriteLine("First: {0}", i);

            foreach (var i in unique)

                Console.WriteLine("Second: {0}", i);

The output is:

First: 1
First: 2

Why is there no output of the second loop? The reason is that LINQ does not cache the results of the collection but it does recalculate the contents for every new enumeration again. Since I have used state (the Hashset does decide which entries are part of the output) I do arrive with an empty sequence since Add of the Hashset will return false for all values I have already passed in leaving nothing to return a second time.

The solution is quite simple: Use the Distinct extension method or cache the results by calling .ToList() or ToArray() for the result of the LINQ query.

Lession Learned: Do never forget to think about state in Where clauses!

Share This Post:

Posted On Sunday, March 06, 2011 9:07 AM | Feedback (0)

Thursday, November 11, 2010

The Only Console Application Template You Will Ever Need

I create quite often some small console application to do this or that. But the standard Console Application project is rather … empty. To prevent me from reinventing the wheel over and over again I sat down and created a more usable Console Application skeleton. The features are

Functional style command line parser
Console Coloring
Error Handler set up (try/catch around Main and AppDomain.UnhandledException Handler)
Help
Clean Code
No additional dependencies required. Only 20 KB of additional source code with documentation is added. That is all.

With VS210 there is the new VSIX format to install templates but my working horse is still VS2008. I did create therefore an MSI which takes care of deploying the Color Console template for VS2008 and VS2010 when present.

Your main method looks much more declarative now where you can set in the delegates the values of the command line switches as you like.

       /// <summary>

         /// Main entry point which is directly called from main where nothing happens except exception catching.

         /// </summary>

         /// <param name="args"></param>

         public  Program(string [] args)

             // define parameterless command line switches.

             // Please note: Upper case characters define the shortcut name for each switch

             var  switches = new  Dictionary <string , Action >

                 {"OptionaL" , () => Optional=true  }, // shortcut -ol

};

             // define command line switches which take one parameter

             var  switchWithArg = new  Dictionary <string , Action <string >>

                 {"ArgSwitch" , (arg) => ArgSwitch = arg },  // shortcut -AS

};

             // Handler for <other arguments> if present

             Action <List <string >> rest = (parameters) => OtherArgs = parameters;

             ArgParser  parser = new  ArgParser (switches, switchWithArg, rest);

             // check if command line is well formed

             if  (!parser.ParseArgs(args))

                 parser.PrintHelpWithMessages(HelpStr);

                 return ;

             if  (!ValidateArgs(parser))

                 // Display errors but not the help screen.

                 parser.PrintHelpWithMessages(null );

             else

                 // Ok we can start

                 Run();

Where can you get it? You can download it from the Visual Studio Gallery where I put it as Colored Console Application Template. The command line parser is case insensitive and supports shortcuts for commands.

The main challenge was creating the installer with the Windows Installer Extensions toolkit. MSI is an arcane but still relevant technology. After reading a bit what can go wrong during installation/update/upgrade I wonder if any MSI package out there is 100% correct. No wonder that MS wanted to make deployment easier with the VSIX packages but these do work only with VS2010.

Share This Post:

Posted On Thursday, November 11, 2010 10:22 AM | Feedback (1)

Monday, July 19, 2010

Fault Injection Done Right

Patrick Smacchia has shared his thoughts why 100% test coverage is a desirable thing to have. You can test 80% of your code quite easy but the edge cases are the hard ones which take 80% of the time to write good tests for them. Patrick claims that it is worth the effort since in the last 20% you will find most of your bugs. Why is it so hard to get to full coverage? To make the corner cases testable you need some extra ~~ugly~~ constructs in your product (normally via ifdefs where you throw an exception). Many developers do not do it because they feel that this makes their code obfuscated and not maintainable. I am a firm believer in elegant code so lets see how we can keep our code nice and maintainable at the same time.

This made up example is straightforward and it has no way to simulate from outside failures:

void DoSomeLogic()

            using (Tracer t = new Tracer(myType, "DoSomeLogic"))

                using (var stream = File.Open(tmpFile, FileMode.Open, FileAccess.Read))

                    using (StreamReader reader = new StreamReader(stream))

                        while (true)

                            string line = reader.ReadLine();

                            if (line == null)

                                break;

                            t.Info(Level.L3, "Got line from file {0}", line);

Ok it has some tracing to find out how the code is doing but what if we want to test what does happen if there was an error after the file was opened and when an IOError occurs during reading the second line of the file? If you think I will refactor the code now to make it more testable. Nope not this time. But we need to instrument the code somehow. In a non obvious way we did this already!

Here is the instrumented code:

        void DoSomeLogic()

            using (Tracer t = new Tracer(myType, "DoSomeLogic"))

                using (var stream = File.Open(tmpFile, FileMode.Open, FileAccess.Read))

                    t.Instrument("1");

                    using (StreamReader reader = new StreamReader(stream))

                        while (true)

                            string line = reader.ReadLine();

                            if (line == null)

                                break;

                            t.Info(Level.L3, "Got line from file {0}", line);

Hm the code looks the same except that there is an Instrument trace added. That is strange. Let´s look at the test to simulate an exception after the file was opened:

        static TypeHashes myType = new TypeHashes(typeof(TracerUseCases));

        [Test]

        public void Inject_Fault_After_File_Open()

            DoSomeLogic();

            TracerConfig.Reset("null");

            Tracer.TraceEvent += (severity, typemethod, time, message) =>

                    if (severity == Tracer.MsgType.Instrument)

                        throw new IOException("Hi this is an injected fault");

};

            Assert.Throws<IOException>(() => DoSomeLogic());

And here is the test to force an exception during reading the second line of a file:

        [Test]

        public void Inject_Fault_During_Stream_Read()

            TracerConfig.Reset("null");

            int lineCount = 0;

            Tracer.TraceEvent += (severity, typemethod, time, message) =>

                if (severity == Tracer.MsgType.Information && message.Contains("Got line from"))

                    lineCount++;

                    if (lineCount == 2)

                        throw new IOException("Exception during read of second line");

};

            Assert.Throws<IOException>(() => DoSomeLogic());

The trick is simple but powerful. We can delegate the aspect fault injection to the already existing tracing calls which we do intercept in our tests and throw an exception when needed. The best thing about this is that you need no more changes to your product code to inject faults at critical points in your application. You only need to add some trace statements at strategic points which is a good idea since you want to know what is going on in these cases anyway. You can download the ApiChange Tool Api here. To try it out you can simply reference the ApiChange.Api.dll and copy some of the above code into your solution and you have full tracing enabled. To configure tracing you can look at some documentation here.

When something unexpected happens you can simply change the Tracing configuration in your unit test from null to console to see the tracing output in your test runner.

***** UnitTests.Infrastructure.Diagnostics.TracerUseCases.Inject_Fault_During_Stream_Read
23:34:11.712 05876/04208 <Instrument > UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic 1
23:34:11.713 05876/04208 <Information> UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic Got line from file Line 1
23:34:11.714 05876/04208 <Exception > UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic Exception thrown: System.IO.IOException: Exception during read of second line
   at UnitTests.Infrastructure.Diagnostics.TracerUseCases.<>c__DisplayClass5.<Inject_Fault_During_Stream_Read>b__3(MsgType severity, String typemethod, DateTime time, String message)    at ApiChange.Infrastructure.Tracer.TraceMsg(String msgTypeString, String typeMethodName, DateTime time, String fmt, Object[] args)
   at ApiChange.Infrastructure.Tracer.Info(Level level, String fmt, Object[] args)
   at UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic()
   at UnitTests.Infrastructure.Diagnostics.TracerUseCases.<Inject_Fault_During_Stream_Read>b__4()
   at NUnit.Framework.Assert.Throws(IResolveConstraint expression, TestDelegate code, String message, Object[] args)
23:34:11.724 05876/04208 <         }}< UnitTests.Infrastructure.Diagnostics.TracerUseCases.DoSomeLogic Duration 13ms

If you need to add traces for the sole purpose to inject faults into your product code you can use the Tracer.Instrument overloads which are conditionally compiled into your code when you add INSTRUMENT to your conditional compilation constants to your project. This way you get the best of both worlds. No code in your release build but full instrumentation where necessary.

Have fun getting your code to 100% coverage.

Share This Post:

Posted On Monday, July 19, 2010 9:42 AM | Feedback (2)

Tuesday, July 06, 2010

Parsing A Text File with F#

F# is a powerful language. When you start with functional programming you find it hard to write truly functional code. After all you have thought in imperative terms like for, foreach, while, do while, …. Mark Pearl has provided a little sample how to parse a text file with F# to count the error and warnings inside it. I think it can be more functional. Here is my try:

#light
open System
open System.IO

let lineSequence(file) =
        let reader = File.OpenText(file)
        Seq.unfold(fun line ->
            if line = null then
                reader.Close()
                None
            else
                Some(line,reader.ReadLine())) (reader.ReadLine())

let CalculateStats(lines:seq<string>) =
    lines |> Seq.fold(
        fun (lineCount,info,warn,error) line -> match line with
                                                | _ when line.Contains("info")      -> (lineCount+1, info+1, warn,   error)
                                                | _ when line.Contains("_warning_") -> (lineCount+1, info , warn+1, error)
                                                | _ when line.Contains("*ERROR*")   -> (lineCount+1, info , warn , error+1)
                                                | _ ->                                 (lineCount+1, info , warn , error )) (0,0,0,0)

let lineCount,info,warn,errors = @"C:\Source\fsharpparser\data.txt" |> lineSequence |> CalculateStats
printf "LineCount: %d, Got %d infos, %d warnings, %d errors" lineCount info warn errors
Console.ReadLine() |> ignore

In contrast to Marks version I do read the file line by line instead of reading the whole file at once (lazy evaluation is key) and I do check not only for errors and warnings. I thought that info's might also be part of the log file so I do count them as well. I must admit that it takes some time to get it right but I am like Mark still learning how to employ F# for practical purposes.

Share This Post:

Posted On Tuesday, July 06, 2010 11:10 AM | Feedback (1)

Article Categories

Archives

Post Categories

Image Galleries

Programming

Rico Mariani: CLR Performance Guru