Bruce Ge

We all know the usage of the anonymous type:

What if we want to add a anonymous delegate or Lamba to it?

Like:

or

The above two will cause compile time error, because anonymous type need anonymous type property, not the anonymous method or lambda expression.

There is some way to do it:

1. define a Func:

then we can do it:

var obj = new {Name = "John", Age = 24, behaviour = behav };

2. declare a delegate:

delegate string behav(string name, int age);

then we can do:

var obj = new { Name = "John", Age = 24, behaviour = (behav)delegate(string name, int age) { return "Customer Name: " + name + ", Age: " + age; } };

or

var obj = new { Name = "John", Age = 24, behaviour = (behav)((name, age) => string.Format("Hello, {0}! You are {1} years old.", name, age)) };

This is the demonstration only, if we want to use IEnumerable<AnonymousType>, as anonymous type can only be cast as object, there is article here to describe how to use it. Here is another solution:

for the ado database sync, the method

public virtual SyncContext ApplyChanges(SyncGroupMetadata groupMetadata, DataSet dataSet,SyncSession syncSession)

on server side is not efficient, as it receive the changed data from client side, but it again sent it back to client within the SyncContext.

In the returned SyncContext object, I found DataSet and GroupProgress.Changes is almost the same as input dataset. by verifying the code inside sync framework, I found:

internal SyncGroupMetadata ResetProivderState(SyncGroupMetadata groupMetadata, DataSet dataSet)
{
    this._rawOldAnchor = null;
    this._rawNewAnchor = null;
    this._rawMaxAnchor = null;
    this._syncContext = new SyncContext();
    this._syncContext.DataSet = dataSet;
    SyncGroupMetadata metadata = this.GenerateOrderedGroupMetadata(groupMetadata);
    this._syncContext.GroupProgress = new SyncGroupProgress(metadata, dataSet);
    return metadata;
}

so I use the code below:

SyncContext context = _serverSyncProvider.ApplyChanges(groupMetadata, dataSet, syncSession);
context.DataSet = null;
context.GroupProgress.Changes.Clear();
return context; 

to make the size smaller.

I was using Gzip Encoder to compress wcf message, it surprised me that sometimes the compression message size is even bigger than the original size, so I looked the code, I found within GZipMessageEncoderFactory.cs, the method "CompressBuffer" in the GZipMessageEncoderFactory class is not quite right. it was like this originally:

private static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
{
   ....
  var byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset,
                                                       bufferedBytes.Length - messageOffset);

  return byteArray;
}

if we use the method above, the count of the new ArraySegment<T> will be the bufferedBytes, as BufferManager using TakeBuffer method to grab a buffer which is the nearest bigger 2^N byte, e.g. if input param is 522 bytes for TakeBuffer method, it will grab a buffer which has 1024 bytes. And the constructor of ArraySegment will take the third parameter(bold above) to reflects the actual size of the ArraySegment, if size of the bufferedBytes(first params) is less than third parameter, it will fill byte0 to the rest. that is why sometimes the size is even bigger than the original size. 

it should be like this:

private static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
{
   ....
  var byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset,
                       compressedBytes.Length);

  return byteArray;
}

Also I found the Gzip compression doesn't have high compression ratio, I decided to use 7zip and gzip combination then, I found a good article about 7zip: 7Zip (LZMA) In-Memory Compression with C#, after adding it to my project, I can use it straight forward:

private static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
{
 using (var memoryStream = new MemoryStream())
 {
  var zipperStream = new GZipStream(memoryStream, CompressionMode.Compress, true);

  using (zipperStream)
   zipperStream.Write(buffer.Array,
      buffer.Offset, buffer.Count);

  byte[] compressedBytes1 = memoryStream.ToArray();
  byte[] compressedBytes = SevenZip.Compression.LZMA.SevenZipHelper.Compress(compressedBytes1);
  byte[] bufferedBytes = bufferManager.TakeBuffer(
     compressedBytes.Length + messageOffset);

  Array.Copy(compressedBytes, 0, bufferedBytes, messageOffset,
     compressedBytes.Length);

  bufferManager.ReturnBuffer(buffer.Array);
  var byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset,
     compressedBytes.Length);

  return byteArray;
 }
}

//Helper method to decompress an array of bytes
private static ArraySegment<byte> DecompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager)
{
 var memoryStream1 = new MemoryStream(buffer.Array, buffer.Offset, buffer.Count - buffer.Offset);
 var memoryStream=new MemoryStream(SevenZip.Compression.LZMA.SevenZipHelper.Decompress(memoryStream1.ToArray()));

 var decompressedStream = new MemoryStream();
 int totalRead = 0;
 int blockSize = 1024;
 byte[] tempBuffer = bufferManager.TakeBuffer(blockSize);
 using (var gzStream = new GZipStream(memoryStream, CompressionMode.Decompress))
 {
  while (true)
  {
   int bytesRead = gzStream.Read(tempBuffer, 0, blockSize);
   if (bytesRead == 0)
    break;
   decompressedStream.Write(tempBuffer, 0, bytesRead);
   totalRead += bytesRead;
  }
 }
 bufferManager.ReturnBuffer(tempBuffer);
  
 byte[] decompressedBytes = decompressedStream.ToArray();
 byte[] bufferManagerBuffer = bufferManager.TakeBuffer(decompressedBytes.Length + buffer.Offset);
 Array.Copy(buffer.Array, 0, bufferManagerBuffer, 0, buffer.Offset);
 Array.Copy(decompressedBytes, 0, bufferManagerBuffer, buffer.Offset, decompressedBytes.Length);

 var byteArray = new ArraySegment<byte>(bufferManagerBuffer, buffer.Offset, decompressedBytes.Length);
 bufferManager.ReturnBuffer(buffer.Array);

 return byteArray;
}

Been using ReliableSessionBindingElement within WCF, I found we can not set the retry interval, as we only can set MaxRetryCount. WCF uses an internal algorithm to determine when to retransmit, based on a computed average round-trip time. The initial retry time is computed based on the measured roundtrip time of establishing the session. The retransmission algorithm doubles the delay with every attempt, for example, if we set the MaxRetryCount to 8(default), if WCF determines the initial retry time to be 2 seconds, then 8th retry will set the retry ackonwledgement time to be 2^8 which is 512 seconds. We can not change the acknowledgement interval manually. so if you see from Fiddler, WCF Visualizer or Trace Viewer, it is not surprise to see some messages been sent several times if you use ReliableSession. There is no other way to avoid this except making server stateless (disable ReliableSession)

I have been working on database synchronization for a couple of weeks, and seems it at the final testing stage, I am using Microsoft Sync Framework 2.0, the DB server is SQL server 2008 Standard, and clients are SQL Server 2008 Express as I use Change Tracking to track DB changes instead of TombStone Tables and Guid tracking columns, I use WCF+IIS+SSL to host server side service, the binding uses Gzip binary encoding.

The reason we are not using SQL Server Replication is because our tables need dynamic filters and some of them logic is quite complicated (we need to download the certain data to client according to the client name and setting), so when Replication try to generate the snapshot, it takes quite long time and CPU usage is quite high.

Coding didn't take me so long to finish (main part is write customized filter queries) but I realize some of the sync tables using bigint as primary key instead of Guid, so I have to assign a new seed to each client to avoid confliction. This new seed number will download and apply when clients reinitialize after schema download and applied.

Also, I found after "Applychanges" has been called, some tables identities will be changed, because in the "InsertCommand" of the Adaptor, using "SET IDENTITY_INSERT [tableName] ON", this will cause "If the value inserted is larger than the current identity value for the table, SQL Server automatically uses the new inserted value as the current identity value.", so we use a new table to hold the current identity for tables before applychanges and set it back after applychanges.

I have found when uploading changes to server side, the dataset contains all changed data, and after ApplyChanges called, the return object SyncContext still contains the dataset which sent to server, this is very inefficient, also when some changes apply failed, the return size grows very big because the GroupProgress inside SyncContext contains all dataset info plus failed table and row info, this makes memory growing very fast on server side as well as transferring.

Server side need to maintain a session as the the clients call is each client based, so the WCF interface is like this:

[ServiceContract(SessionMode = SessionMode.Required)]
    public interface IServiceForSync
    {
        [OperationContract(IsInitiating = true)]
        bool InitialiseSetting();

        [OperationContract(IsInitiating = false)]
        SyncContext ApplyChanges(SyncGroupMetadata groupMetadata, DataSet dataSet, SyncSession syncSession);

        [OperationContract(IsInitiating = false)]
        SyncContext GetChanges(SyncGroupMetadata groupMetadata, SyncSession syncSession);

        [OperationContract(IsInitiating = false)]
        SyncSchema GetSchema(Collection<string> tableNames, SyncSession syncSession);

        [OperationContract(IsInitiating = false)]
        SyncServerInfo GetServerInfo(SyncSession syncSession);

        [OperationContract(IsInitiating = false)]
        SyncInfo GetSyncInfo(string SyncHostName, long rpuid);

        [OperationContract(IsInitiating = false)]
        List<String> GetInitSchema();

        [OperationContract(IsInitiating = false)]
        List<String> GetUpdateSchema(string SyncHostName);

        [OperationContract(IsInitiating = false)]
        bool ResetStatus(string SyncHostName);

        [OperationContract(IsTerminating = true)]
        void EndSession(); 
    }

The Service side binding in the web.config is like this:

   <customBinding>
    <binding name="ISyncServer" receiveTimeout="00:30:00" sendTimeout="00:30:00">
     <gzipMessageEncoding innerMessageEncoding="binaryMessageEncoding"/>
     <reliableSession ordered="true" inactivityTimeout="00:10:00" maxPendingChannels="128"/>
     <httpsTransport hostNameComparisonMode="StrongWildcard" manualAddressing="False" maxReceivedMessageSize="2000000000" authenticationScheme="Anonymous" bypassProxyOnLocal="False" realm="" useDefaultWebProxy="True"/>
    </binding>
   </customBinding>

On client side the remoteServer of the syncAgent will be the proxy, and I use one SyncGroup as any exception happens, it will reverse back to original.

public class SqlExpressClientSyncProvider : ClientSyncProvider

and in the ApplyChanges of the ClientSyncProvider, we need to Map SyncDirection from client point of view to our internal server point of view, so part of code is like:

 foreach (var tableMetadata in groupMetadata.TablesMetadata)
            {
                switch (tableMetadata.SyncDirection)
                {
                    case SyncDirection.Snapshot:
                    case SyncDirection.DownloadOnly:
                        tableMetadata.SyncDirection = SyncDirection.UploadOnly;
                        break;
                    case SyncDirection.UploadOnly:
                        tableMetadata.SyncDirection = SyncDirection.DownloadOnly;
                        break;
                }
            }

in GetChanges, we need to swap anchors:

public override SyncContext GetChanges(SyncGroupMetadata groupMetadata, SyncSession syncSession)
        {
            // neet to set the LastReceivedAnchor as the LastSentAnchor since
            // DbServerSyncProvider operates from the server's perspective, so
            // we swap the two fields temporarily.
            foreach (var metaTable in groupMetadata.TablesMetadata)
            {
                var temp = metaTable.LastReceivedAnchor;
                metaTable.LastReceivedAnchor = metaTable.LastSentAnchor;
                metaTable.LastSentAnchor = temp;
            }
          
            var context = _dbSyncProvider.GetChanges(groupMetadata, syncSession);

            if (_bEnableTimeAdjusting.HasValue && _bEnableTimeAdjusting.Value)
                AdjustDSTimeZone(context.DataSet, "CLIENT"); //-TN - adjust all datetime columns before sync.
            //swap them back for consistency
            foreach (var metaTable in groupMetadata.TablesMetadata)
            {
                var temp = metaTable.LastReceivedAnchor;
                metaTable.LastReceivedAnchor = metaTable.LastSentAnchor;
                metaTable.LastSentAnchor = temp;
            }
            return context;
        }