Converting PHP’s “openssl_seal” and “openssl_open” into .NET.

I've been doing a bit of external API integration work lately. Sounds mundane (and it is for the most part) - but there was one notable "gem" I had to tackle. To make long story short, there's this external source written in PHP and exposed as a web service. The operations are simple and there isn't much to it, except one little detail - this service uses PHP's encryption to pass data securely between peers using SSL certificates: the operations are "openssl_seal" and "openssl_open".

Now, I'm coding in .NET and I figure: surely, there must be some sort of equivalent to support for all this and after quickly reading up on what PHP does behind the scenes, off to MSDN I go.

First alarming sign: .NET does not have native (Cryptography namespace) support for RC4 ciphers (this is used in one of the steps - and there are multiple steps, more on this later) nor is there such thing as a "pem" file reader ("pem" file is where the certificate key data is stored). This is going to be more challenging than I expected.

Alright, let's apply some google-fu and see what pops up. Searches like ".NET equivalent of PHP 'openssl_seal'" and such turned up absolutely nothing (hopefully, this will change once I'm done typing this up) but I do find an OpenSSL library implementation in .NET (well, it's more of a managed wrapper around C/C++: http://sourceforge.net/projects/openssl-net/) - it doesn't seem to have been updated for about a year but it's worth a try.

The documentation wasn't the most detailed (argh!):


openssl

..and few hours of looking, the site turned up some useful bits but not a quick-'n-easy equivalent for "openssl_seal" command. This whole thing appears to be PHP only feature (as far as I can tell) which uses some of the OpenSSL functionality.

So, what does this command do exactly? The official PHP site (http://php.net/manual/en/function.openssl-seal.php) doesn't share too much in terms of reverse-engineering, this is about all it states:

open_seal_php

But there are user comments I found more useful, one gave some good pseudo flow:

open_seal_php2

And there's a link to a Java example (aha!) - http://blog.local.ch/archive/2007/10/29/openssl-php-to-java.html. Though, this mostly deals with decrypting data - helpful, but still no luck with the 'encryptor' "openssl_seal" implementation. Another hint - it does use Bouncy Castle libraries for Java (the .NET C# version can be found at http://www.bouncycastle.org/csharp/) so I go ahead in that direction.

As expected, Bouncy Castle does not provide straightforward "openssl_seal"/"openssl_open" implementation. But it seems to have everything I'd need (or at least, I think I need) - oh, and by the way there's no easy documentation there neither. You'll need to search through trunks on Google and such to figure out finer details (for example, http://code.google.com/p/ecc-micropayment/source/browse/trunk/csharp/crypto/src/security/PbeUtilities.cs?r=2).

After whole bunch of trial and error, I got very close. I was able to generate a RC4 random 128-bit key, encrypt data using RC4 cipher and bring it back (decrypt it). However, there was one last step still missing - encrypting the above-mentioned 128-bit key using RSA cipher. It looked like I was doing it as prescribed, what could it be?!

It wasn't until I found PHP security library interop documentation for OpenSSL (http://phpseclib.sourceforge.net/interop.html#  and click on PHP Bindings: openssl_seal() / openssl_open() -> phpseclib) and one detail caught my eye - PHP line "$rsa->setEncryptionMode(CRYPT_RSA_ENCRYPTION_PKCS1); " line.

The key must be encrypted with RSA (I knew that) AND padded using PKCS1!!! Well, of course, isn't that obvious to everyone (*sarcastic argh!). 

So here's my take on all this - go ahead and grab a Visual Studio NuGet Package for "Bouncy Castle". And use following code:

For encryption, the "OPENSSL_SEAL" equivalent:

.NET "open_seal"
  1.  
  2. var randomPassword = Guid.NewGuid().ToString("n").ToCharArray();
  3.  
  4. var salt = new byte[16];
  5. for (int i = 0; i != salt.Length; i++)
  6.     salt[i] = (byte)i;
  7.  
  8. var pGen = new OpenSslPbeParametersGenerator();
  9. pGen.Init(PbeParametersGenerator.Pkcs5PasswordToBytes(randomPassword), salt);
  10. var rc4Parameters = (KeyParameter)pGen.GenerateDerivedParameters("RC4", 128); /* generate random 128 bit key */
  11.  
  12. var rc4Cipher = CipherUtilities.GetCipher("RC4");
  13. rc4Cipher.Init(true, rc4Parameters);
  14. var rc4DataBytes = rc4Cipher.DoFinal(Encoding.ASCII.GetBytes("YOUR_DATA_STRING_TO_ENCYPT")); /* encrypt data symmetrically with RC4 using the random key */
  15.  
  16. var rsaCipher = CipherUtilities.GetCipher("RSA//PKCS1PADDING"); /* that pesky padding flag */
  17. var remReader = new PemReader(new StringReader("YOUR_STRING_OF_PUBLIC_KEY_CONTENTS_FROM_PEM_FILE"));
  18. var rsaParameters = (RsaKeyParameters)remReader.ReadObject();
  19. rsaCipher.Init(true, rsaParameters);
  20. var rsaDataBytes = rsaCipher.DoFinal(rc4Parameters.GetKey()); /* encrypt the random key itself using RSA */
  21.         
  22. .

 

As for decrypting: first, you'll need a way to parse the private key from the PEM file and generate RsaPrivateCrtKeyParameters:

PEM Private Key Reader
  1. public static RsaPrivateCrtKeyParameters GetPrivateKey(String pemFile)
  2. {
  3.     if (string.IsNullOrEmpty(pemFile))
  4.         throw new ArgumentNullException("pemFile");
  5.  
  6.     string privateKey = File.Exists(pemFile) ? File.ReadAllText(pemFile) : pemFile;
  7.  
  8.     var reader = new PemReader(new StringReader(privateKey));
  9.     RsaPrivateCrtKeyParameters privkey = null;
  10.     Object obj = reader.ReadObject();
  11.     if (obj is AsymmetricCipherKeyPair)
  12.     {
  13.         privkey = (RsaPrivateCrtKeyParameters)((AsymmetricCipherKeyPair)obj).Private;
  14.     }
  15.     return privkey;
  16. }

 

..and basically reverse the process but use the private key:

.NET "openssl_open"
  1.  
  2. var privateKeyParameters = GetPrivateKey("YOUR_STRING_OF_PRIVATE_KEY_CONTENTS_FROM_PEM_FILE");
  3.  
  4. var rsaCipher = CipherUtilities.GetCipher("RSA//PKCS1PADDING");
  5. rsaCipher.Init(false, privateKeyParameters);
  6. var keyBytes = rsaCipher.DoFinal(Convert.FromBase64String("ENCRYPTED_KEY_STRING"));                    /* decrypt key using RSA */
  7.  
  8. var rc4CipherDecrypt = CipherUtilities.GetCipher("RC4");
  9. var decryptParameter = new KeyParameter(keyBytes);
  10. rc4CipherDecrypt.Init(false, decryptParameter);
  11. var rc4DataBytesDecrypt = rc4CipherDecrypt.DoFinal(Convert.FromBase64String("ENCRYPTED_DATA_STRING")); /* decrypt data using RC4 */
  12. var dataString = Encoding.ASCII.GetString(rc4DataBytesDecrypt);
  13.  
  14.             .

 

I hope you find this useful and save yourself much headache. Cheers!

Auto-Suggest via ‘Trie’ (Pre-fix Tree)

Auto-Suggest (Auto-Complete) “thing” has been around for a few years. Here’s my little snippet on the subject.

For one of my projects, I had to deal with a non-trivial set of items to be pulled via auto-suggest used by multiple concurrent users. Simple, dumb iteration through a list in local cache or back-end access didn’t quite cut it.

Enter a nifty little structure, perfectly suited for storing and matching verbal data: “Trie” (http://tinyurl.com/db56g) also known as a Pre-fix Tree:

“Unlike a binary search tree, no node in the tree stores the key associated with that node; instead, its position in the tree defines the key with which it is associated. All the descendants of a node have a common prefix of the string associated with that node, and the root is associated with the empty string. Values are normally not associated with every node, only with leaves and some inner nodes that correspond to keys of interest.”

image

This is a very scalable, performing structure. Though, as usual, something ‘fast’ comes at a cost of ‘size’; fortunately RAM is more plentiful today so I can live with that.

I won’t bore you with the detailed algorithmic performance here - Google can do a better job of such.

So, here’s C# implementation of all this. Let’s start with individual node:

Trie Node
  1. /// <summary>
  2. /// Contains datum of a single trie node.
  3. /// </summary>
  4. public class AutoSuggestTrieNode
  5. {
  6.     public char Value { get; set; }
  7.  
  8.     /// <summary>
  9.     /// Gets a value indicating whether this instance is leaf node.
  10.     /// </summary>
  11.     /// <value>
  12.     ///     <c>true</c> if this instance is leaf node; otherwise, a prefix node <c>false</c>.
  13.     /// </value>
  14.     public bool IsLeafNode { get; private set; }
  15.  
  16.     public List<AutoSuggestTrieNode> DescendantNodes { get; private set; }
  17.  
  18.  
  19.     /// <summary>
  20.     /// Initializes a new instance of the <see cref="AutoSuggestTrieNode"/> class.
  21.     /// </summary>
  22.     /// <param name="value">The phonetic value.</param>
  23.     /// <param name="isLeafNode">if set to <c>true</c> [is leaf node].</param>
  24.     public AutoSuggestTrieNode(char value = ' ', bool isLeafNode = false)
  25.     {
  26.         Value = value;
  27.         IsLeafNode = isLeafNode;
  28.  
  29.         DescendantNodes = new List<AutoSuggestTrieNode>();
  30.     }
  31.  
  32.     /// <summary>
  33.     /// Gets the descendants of the pre-fix node, if any.
  34.     /// </summary>
  35.     /// <param name="descendantValue">The descendant value.</param>
  36.     /// <returns></returns>
  37.     public AutoSuggestTrieNode GetDescendant(char descendantValue)
  38.     {
  39.         return DescendantNodes.FirstOrDefault(descendant => descendant.Value == descendantValue);
  40.     }
  41. }

 

Quite self-explanatory, imho. A node is either a “Pre-fix” or a “Leaf” node. “Leaf” contains the full “word”, while the “Pre-fix” nodes act as indices used for matching the results.

 

Ok, now the Trie:

Trie Structure
  1. /// <summary>
  2. /// Contains structure and functionality of an AutoSuggest Trie (Pre-fix Tree)
  3. /// </summary>
  4. public class AutoSuggestTrie
  5. {
  6.     private readonly AutoSuggestTrieNode _root = new AutoSuggestTrieNode();
  7.  
  8.     /// <summary>
  9.     /// Adds the word to the trie by breaking it up to pre-fix nodes + leaf node.
  10.     /// </summary>
  11.     /// <param name="word">Phonetic value.</param>
  12.     public void AddWord(string word)
  13.     {
  14.         var currentNode = _root;
  15.         word = word.Trim().ToLower();
  16.  
  17.         for (int i = 0; i < word.Length; i++)
  18.         {
  19.             var child = currentNode.GetDescendant(word[i]);
  20.  
  21.             if (child == null) /* this character hasn't yet been indexed in the trie */
  22.             {
  23.                 var newNode = new AutoSuggestTrieNode(word[i], word.Count() - 1 == i);
  24.  
  25.                 currentNode.DescendantNodes.Add(newNode);
  26.                 currentNode = newNode;
  27.             }
  28.             else
  29.                 currentNode = child; /* this character is already indexed, move down the trie */
  30.         }
  31.     }
  32.  
  33.  
  34.     /// <summary>
  35.     /// Gets the suggested matches.
  36.     /// </summary>
  37.     /// <param name="word">The phonetic search value.</param>
  38.     /// <returns></returns>
  39.     public List<string> GetSuggestedMatches(string word)
  40.     {
  41.         var currentNode = _root;
  42.         word = word.Trim().ToLower();
  43.  
  44.         var indexedNodesValues = new StringBuilder();
  45.         var resultBag = new ConcurrentBag<string>();
  46.  
  47.         for (int i = 0; i < word.Trim().Length; i++)  /* traverse the trie collecting closest indexed parent (parent can't be leaf, obviously) */
  48.         {
  49.             var child = currentNode.GetDescendant(word[i]);
  50.  
  51.             if (child == null || word.Count() - 1 == i)
  52.                 break; /* done looking, the rest of the characters aren't indexed in the trie */
  53.  
  54.             indexedNodesValues.Append(word[i]);
  55.             currentNode = child;
  56.         }
  57.  
  58.         Action<AutoSuggestTrieNode, string> collectAllMatches = null;
  59.         collectAllMatches = (node, aggregatedValue) => /* traverse the trie collecting matching leafNodes (i.e. "full words") */
  60.             {
  61.                 if (node.IsLeafNode) /* full word */
  62.                     resultBag.Add(aggregatedValue); /* thread-safe write */
  63.  
  64.                 Parallel.ForEach(node.DescendantNodes, descendandNode => /* asynchronous recursive traversal */
  65.                 {
  66.                     collectAllMatches(descendandNode, String.Format("{0}{1}", aggregatedValue, descendandNode.Value));
  67.                 });
  68.             };
  69.  
  70.         collectAllMatches(currentNode, indexedNodesValues.ToString());
  71.  
  72.         return resultBag.OrderBy(o => o).ToList();
  73.     }
  74.  
  75.  
  76.     /// <summary>
  77.     /// Gets the total words (leafs) in the trie. Recursive traversal.
  78.     /// </summary>
  79.     public int TotalWords
  80.     {
  81.         get
  82.         {
  83.             int runningCount = 0;
  84.  
  85.             Action<AutoSuggestTrieNode> traverseAllDecendants = null;
  86.             traverseAllDecendants = n => { runningCount += n.DescendantNodes.Count(o => o.IsLeafNode); n.DescendantNodes.ForEach(traverseAllDecendants); };
  87.             traverseAllDecendants(this._root);
  88.  
  89.             return runningCount;
  90.         }
  91.     }
  92. }

 

Matching operations and Inserts involve traversing the nodes before the right “spot” is found. Inserts need be synchronous since ordering of data matters here.

However, matching can be done in parallel traversal using recursion (line 64).

Here’s sample usage:

  1.  
  2. [TestMethod]
  3. public void AutoSuggestTest()
  4. {
  5.     var autoSuggestCache = new AutoSuggestTrie();
  6.  
  7.     var testInput = @"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer nec odio. Praesent libero.
  8.                 Sed cursus ante dapibus diam. Sed nisi. Nulla quis sem at nibh elementum imperdiet. Duis sagittis ipsum. Praesent mauris.
  9.                 Fusce nec tellus sed augue semper porta. Mauris massa. Vestibulum lacinia arcu eget nulla. Class aptent taciti sociosqu ad
  10.                 litora torquent per conubia nostra, per inceptos himenaeos. Curabitur sodales ligula in libero. Sed dignissim lacinia nunc.
  11.                 Curabitur tortor. Pellentesque nibh. Aenean quam. In scelerisque sem at dolor. Maecenas mattis. Sed convallis tristique sem.
  12.                 Proin ut ligula vel nunc egestas porttitor. Morbi lectus risus, iaculis vel, suscipit quis, luctus non, massa. Fusce ac
  13.                 turpis quis ligula lacinia aliquet. Mauris ipsum. Nulla metus metus, ullamcorper vel, tincidunt sed, euismod in, nibh. Quisque
  14.                 volutpat condimentum velit. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Nam
  15.                 nec ante. Sed lacinia, urna non tincidunt mattis, tortor neque adipiscing diam, a cursus ipsum ante quis turpis. Nulla
  16.                 facilisi. Ut fringilla. Suspendisse potenti. Nunc feugiat mi a tellus consequat imperdiet. Vestibulum sapien. Proin quam. Etiam
  17.                 ultrices. Suspendisse in justo eu magna luctus suscipit. Sed lectus. Integer euismod lacus luctus magna. Quisque cursus, metus
  18.                 vitae pharetra auctor, sem massa mattis sem, at interdum magna augue eget diam. Vestibulum ante ipsum primis in faucibus orci
  19.                 luctus et ultrices posuere cubilia Curae; Morbi lacinia molestie dui. Praesent blandit dolor. Sed non quam. In vel mi sit amet
  20.                 augue congue elementum. Morbi in ipsum sit amet pede facilisis laoreet. Donec lacus nunc, viverra nec.";
  21.  
  22.     testInput.Split(' ').ToList().ForEach(word => autoSuggestCache.AddWord(word));
  23.  
  24.     var testMatches = autoSuggestCache.GetSuggestedMatches("le");
  25. }

 

..and the result:

image

That’s it!

Asynchronous timer threading, TimerCallback.

There’re frequent times when I need to perform some sort of periodic tasks within the application: refresh in-memory shared data from the backend, status checks, emitting a health beat (pulse) with some sort of statistics to a log or UDP broadcast, and so on.

As a bonus, it’d be nice if such tasks were to be non-blocking and thread-safe.

Since, the calls should be to non-blocking, asynchronous processes – they will require a separate thread to run on, in order to avoid interfering with the main process. So, how can we launch such thread?

We can invoke a new thread to a method call where the thread will run inside a “while” loop with a Thread.Sleep call at the end each iteration. We’d need to start such thread before all else (maybe in the constructor of the enveloping class), and than forget all about it.

Of course, nothing else would have much use for such method as it’s stuck in an infinite loop. And some may suggest that using infinite loops in your domain logic is a bad idea to begin with. I’d also point out that this can potentially cause potential run-away processing and thread-safety concerns. So, what’s to be done?

I find use of TimerCallback (http://tinyurl.com/c5grkcd) delegate a more elegant approach. Such delegated method would execute in a system thread pool separate from the thread which instantiated the callback. And, if need be, it can be aborted with a Timer.Change call.

We can start with declaring and instantiating some basic players in all this:

  1. private readonly static AutoResetEvent _autoEvent = new AutoResetEvent(false);
  2. private static TimerCallback _tcb = MyPeriodicMethod;
  3. private static Timer _stateTimer = new Timer(_tcb, _autoEvent, dueTime: 0, period: 60000);

‘MyPeriodicMethod’ refers to the target method. and ‘period: ‘ is set to 60000 milliseconds (1 minute) – i.e. target method will be called every minute.

 

Moving onto the target method:

  1. private static DateTime _lastRunDatetime;
  2. /// <summary>
  3. /// Target method for the TimerCallback
  4. /// </summary>
  5. /// <param name="stateInfo">The state info.</param>
  6. [MethodImpl(MethodImplOptions.Synchronized)]
  7. private static void MyPeriodicMethod(Object stateInfo)
  8. {
  9.     if (DateTime.Now.Subtract(_lastRunDatetime).TotalMilliseconds < 60000)
  10.         return;
  11.  
  12.     var autoEvent = (AutoResetEvent)stateInfo;
  13.  
  14.     try
  15.     {
  16.         _synRoot.EnterReadLock();
  17.  
  18.         /* do all the "periodic" work here */
  19.  
  20.     }
  21.     catch (Exception ex)
  22.     {
  23.         /* handle exception */
  24.     }
  25.     finally
  26.     {
  27.         if (_synRoot.IsReadLockHeld)
  28.             _synRoot.ExitReadLock();
  29.  
  30.         _lastRunDatetime = DateTime.Now;
  31.     }
  32. }

 

Couple things of note here. Notice [MethodImpl(MethodImplOptions.Synchronized)] (line 6) method attribute. What this means is that only one thread at a time can call this method.

If for some reason two or more timer-calls run into each other, i.e. call from interval 1 is still running when call from interval 2 comes in – the call from interval 2 will wait for the first one to complete.

Also, notice the _lastRunDateTime variable (line 1). This keeps track of the last time the method was called. So, if there’s a prematurely arriving thread – we can short-circuit and force it back out of execution (line 10); if not, than the code is executed (inside a thread-safe critical section) and the _lastRunDateTime is updated (line 30).

C'est fini! Merci!

AppFabric DataCacheFactory pooling.

For one of my projects I've been using Windows Server AppFabric Caching (on-premise) (http://tinyurl.com/2usm9bs).

I've setup a simple cluster: 2 nodes, with 4 Gigabytes in total, running in a high-availability mode – data is auto-replicated, so I can take down one node for maintenance (or allow for one-node asteroid strike) and client applications will still keep on trucking.

After a couple days of tweaking settings it all started working out nicely: (at worst) double-digit millisecond read times, good locking synchronization, region partitioning and all else seemed to be working as advertised. However, as the volume ramped up I started noticing some slowdowns. After digging a bit into it I identified possible culprit: DataCacheFactory.

MSDN API documentation didn't address this much, the documented approach was simple: instantiate DataCacheFactory and call GetCache() (or in my case GetDefaultCache())
to access the data. Great. But what does DataCacheFactory do exactly?

According to Jason Roth from MS (http://tinyurl.com/d7sk5s4):

"Creating the factory involves connecting to the cluster and can take some time. But once you have the factory object and the cache that you want to work with, you can simply reuse those  objects to do puts and gets into the cache, and you should see much faster performance".

Ok, I get it. Basically, DataCacheFactory instantiation is expensive so doing this for every call will not scale up. The suggestion is to essentially use a static DataCacheFactory object and reuse it.

Unfortunately, one of my applications leveraging AppFabric has very demanding throughput requirements and thus it’s quite asynchronous (a few hundred concurrent threads) so having just one point of entry eventually started to cause issues. Simply put it - couldn't handle the load.

So, my options so far:
a) Instantiate new DataCacheFactory on each call and get slow performance.
b) Use a static DataCacheFactory instance and get frequent thread contentions (which is not quite the end of the world - in case of cache failure my code automatically falls back to using SQL but this sort of defeats the purpose of having distributed cache in place.. obviously).

Both options are sub-par. What to do? Answer - Pooling!

There's no built-in support in 1.0/1.1 AppFabric for pooling of this sort, so I had to build my own. Where do we keep this pool? Nothing too fancy, just a simple List:

  1. /// <summary>
  2. /// Pool of cache data access objects
  3. /// </summary>
  4. private List<DataCache> _dataCachePool = new List<DataCache>(CachingProviderConfiguration.PoolSize);

 

I'm using a thread-safe lazy singleton pattern for the Caching provider so there's one instance in AppDomain and the constructor will run once per AppDomain as well:

  1. private static volatile Lazy<CachingProvider> _instance = new Lazy<CachingProvider>(() => new CachingProvider(), LazyThreadSafetyMode.ExecutionAndPublication);
  2. public static CachingProvider Instance { get { return _instance.Value; } }

 

..And this is where we want to initialize this pool, it may take some time but only happens once per AppDomain and then we're free of this burden:

  1. /// <summary>
  2. /// Prevents a default instance of the <see cref="CachingProvider"/> class from being created.
  3. /// </summary>
  4. private CachingProvider()
  5. {
  6.     for (int i = 0; i <= CachingProviderConfiguration.PoolSize; i++)
  7.         /* initializing cache data access objects pool */
  8.         _dataCachePool.Add(this.GetDataCacheInstance());
  9. }

DataCache object instantiation is where (the instigator of all this) – DataCacheFactory is accessed, otherwise the code is fairly trivial:

  1. /// <summary>
  2. /// DataCache property. Returns DataCache instance.
  3. /// Sets up data points for clustered nodes, configuration and security.
  4. /// </summary>
  5. private DataCache GetDataCacheInstance()
  6. {
  7.     var servers = new List<DataCacheServerEndpoint>();
  8.     var configuration = new DataCacheFactoryConfiguration();
  9.  
  10.     servers.Add(new DataCacheServerEndpoint(CachingProviderConfiguration.Node1, CachingProviderConfiguration.Node1Port)); /* lead node */
  11.     servers.Add(new DataCacheServerEndpoint(CachingProviderConfiguration.Node2, CachingProviderConfiguration.Node2Port)); /* backup node */
  12.  
  13.     configuration.Servers = servers;
  14.  
  15.     /* set default properties for local cache (local cache disabled) */
  16.     configuration.LocalCacheProperties = new DataCacheLocalCacheProperties();
  17.  
  18.     /* disable tracing to avoid informational/verbose messages */
  19.     DataCacheClientLogManager.ChangeLogLevel(TraceLevel.Off);
  20.  
  21.     /* no security is required, keep it light */
  22.     configuration.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
  23.  
  24.     configuration.RequestTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.RequestTimeout);
  25.     configuration.ChannelOpenTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.ChannelOpenTimeout);
  26.     configuration.TransportProperties.ReceiveTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.ReceiveTimeout);
  27.     configuration.TransportProperties.ChannelInitializationTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.ChannelInitializationTimeout);
  28.  
  29.     //configuration.TransportProperties.ConnectionBufferSize = CachingProviderConfiguration.MaxBufferSize;
  30.     configuration.TransportProperties.MaxBufferPoolSize = CachingProviderConfiguration.MaxBufferPoolSize;
  31.     configuration.TransportProperties.MaxBufferSize = CachingProviderConfiguration.MaxBufferSize;
  32.  
  33.     _dataCacheFactory = new DataCacheFactory(configuration);
  34.  
  35.     return _dataCacheFactory.GetDefaultCache();
  36. }

 

Almost done! So how do we access this pool? Ideally, we must balance the throughput across each item in the pool so a round-robin access comes to mind.

But you don't really have to do this. A random access to an item in the pool for each call will do just as well, considering random generation is uniformly distributed and we don't expect the pool to be in triple digits. One last piece of code:

  1. private Random _instanceIndexGenerator = new Random();
  2. /// <summary>
  3. /// Gets randomly chosen data cache access object from the pool.
  4. /// Load Balancer on the pool.
  5. /// </summary>
  6. private DataCache DataCache
  7. {
  8.     get
  9.     {
  10.         return _dataCachePool[_instanceIndexGenerator.Next(CachingProviderConfiguration.PoolSize)];
  11.     }
  12. }

 

..which can be accessed inside the provider as such: DataCache.Get("your-object-key-goes-here")
       
That's it! Happy coding!

Modify .NET configuration in run-time.

Being able to change file-based configuration settings in runtime is a nice facet in the ASP.NET world. Low-friction and seamless, this little feature allows for traffic to go on uninterrupted and helps avoid flushing AppDomain memory just to reload config. settings.

It's a different story when dealing with Console or Service applications, though. One has to restart the application/service to load up new config. settings.

To avoid losing data or interrupting a running process: one work-around is to hack out event handling routines prior to shutdown/restart and deal with graceful housekeeping there.

This is easy with Service (‘OnStop’ event: http://tinyurl.com/8vok4vj), little more difficult for Console (‘SetConsoleCtrlHandler’ interop: http://tinyurl.com/8bvnqzk). However, even with these tricks you must still restart the executable.

The solution which works for me (and I have to deal with large footprint, high-volume, crazy threaded Services/Console apps) is to simply force a periodic refresh against the config. file in runtime. Here's how I put it together:


1. Let's say, there's a class responsible for dealing with configuration upkeep for an application: SomeApplication, this upkeep class is called: SomeApplicationConfiguration.

public class SomeApplicationConfiguration : BaseConfiguration

(note BaseConfiguration, more on this later)
   
   
2. There’re simple static properties in such class, each corresponding to a configuration value for my SomeApplication ..application:

  1. public static int SomeNumericConfigurationValue
  2. {
  3.     get
  4.     {
  5.         return GetValueFromConfiguration<int>("SomeNumericConfigurationValue") != (default(int)) ?
  6.             GetValueFromConfiguration<int>("SomeNumericConfigurationValue") : 10000;
  7.     }
  8. }

(the default implementation would be used if, for some reason, this value is missing from the config. file so we replace it with some hard-coded default, a value of 10000 here)
     
       
3. Next, is the GetValueFromConfiguration method (we're almost at the 'meat' of the solution, bear with me).

This is a static method and it's part of the BaseConfiguration class mentioned in the first section. It uses some TypeConverter magic (http://tinyurl.com/cjvf6j3) to help with type conversion:

  1. public static T GetValueFromConfiguration<T>(String key)
  2. {
  3.     TypeConverter typeConverter = TypeDescriptor.GetConverter(typeof(T));
  4.  
  5.     if (AppSettings.AllKeys.ToList().Contains(key))
  6.         return (T)typeConverter.ConvertFromInvariantString(AppSettings[key]);
  7.     else
  8.         return default(T);
  9. }
               

4. Ok, so, the AppSettings property is ‘where it’s at’. This is the property which forces a periodic refresh of the configuration values against the .config file.

First, we keep a private static DateTime _refreshTime = DateTime.Now; field to use as a "bookmark" keeping track of the last time BaseConfiguration class pulled values directly from the file.

Secondly, we'll need a way to prevent threading collisions: I usually prefer the light-weight ReaderWriterLockSlim (http://tinyurl.com/btwngm4) type:

private static ReaderWriterLockSlim _syncRoot = new ReaderWriterLockSlim();

Thirdly, and lastly, here's the AppSettings property:

  1. public static NameValueCollection AppSettings
  2. {
  3.     get
  4.     {
  5.         if (DateTime.Now.Subtract(_refreshTime).Minutes > 15) /* refresh every 15 minutes */
  6.         {
  7.             try
  8.             {
  9.                 _syncRoot.EnterReadLock();
  10.                 ConfigurationManager.RefreshSection("appSettings");
  11.                 _refreshTime = DateTime.Now;                        
  12.             }
  13.             finally
  14.             {
  15.                 if (_syncRoot.IsReadLockHeld)
  16.                     _syncRoot.ExitReadLock();
  17.             }
  18.         }
  19.  
  20.         return ConfigurationManager.AppSettings;
  21.     }
  22. }


This is basically a wrapper around ConfigurationManager.AppSettings collection, with one caveat: it invokes ConfigurationManager.RefreshSection("appSettings") method in thread-safe manner every 15 minutes or so (of course, 15 minutes isn’t written in stone).

What this means, is that on the average, the updated value for configuration key will propagate to the AppDomain in 7.5 minutes. (0 minutes in best case scenario, 15 minutes in worst case. If we trust uniform distribution of random access - the average is 15/2 = 7.5).

Here's a compressed version of all this:

Derived Configuration Class
  1. public class SomeApplicationConfiguration : BaseConfiguration
  2. {
  3.     public static int SomeNumericConfigurationValue
  4.     {
  5.         get
  6.         {
  7.             return GetValueFromConfiguration<int>("SomeNumericConfigurationValue") != (default(int))
  8.                        ? GetValueFromConfiguration<int>("SomeNumericConfigurationValue")
  9.                        : 10000;
  10.         }
  11.     }
  12. }

Base Class
  1. public class BaseConfiguration
  2. {
  3.     private static ReaderWriterLockSlim _syncRoot = new ReaderWriterLockSlim();
  4.     private static DateTime _refreshTime = DateTime.Now;
  5.  
  6.     public static NameValueCollection AppSettings
  7.     {
  8.         get
  9.         {
  10.             if (DateTime.Now.Subtract(_refreshTime).Minutes > 15) /* refresh every 15 minutes */
  11.             {
  12.                 try
  13.                 {
  14.                     _syncRoot.EnterReadLock();
  15.                     ConfigurationManager.RefreshSection("appSettings");
  16.                     _refreshTime = DateTime.Now;
  17.                 }
  18.                 finally
  19.                 {
  20.                     if (_syncRoot.IsReadLockHeld)
  21.                         _syncRoot.ExitReadLock();
  22.                 }
  23.             }
  24.  
  25.             return ConfigurationManager.AppSettings;
  26.         }
  27.     }
  28.  
  29.     public static T GetValueFromConfiguration<T>(String key)
  30.     {
  31.         var typeConverter = TypeDescriptor.GetConverter(typeof(T));
  32.  
  33.         if (AppSettings.AllKeys.ToList().Contains(key))
  34.             return (T)typeConverter.ConvertFromInvariantString(AppSettings[key]);
  35.         else
  36.             return default(T);
  37.     }
  38. }

 

That about does it. Happy coding!

Geeks Anonymous, lol.

image

I’ve been a frequent to SoCal Code Camps (if you’re in the area, check ‘em out: http://www.socalcodecamp.com/ they’re held couple times a year).

The next one coming up just posted the schedule – with an addition to a new session “Geeks Anonymous”. Anyway, I thought it hilarious. Kthxbai!

What’s ‘default’ for?

Sometimes there's a need to communicate explicitly that value variable is yet to be "initialized" or in other words - we’ve never changed it from its' default value.

Perhaps "initialized" is not the right word since a value type will always have some sort of value (even a nullable one) but it's just that - how do we tell?

Of course an 'int' would be 0, an 'enum' would the first defined value of a given enum and so on – we sure can make this kind of check "by hand" but eventually it would get a bit messy.

There's a more elegant way with a use of little-known functionality of: 'default'

Let’s just say we have a simple Enum:

Simple Enum
  1. namespace xxx.Common.Domain
  2. {
  3.     public enum SimpleEnum
  4.     {
  5.         White = 1,
  6.  
  7.         Black = 2,
  8.  
  9.         Red = 3
  10.     }
  11. }

 

In case below we set the value of the enum to ‘White’ which happens to be a first and therefore default value for the enum. So the snippet below will set value of the ‘isDefault’ Boolean to ‘true’.

'True' Case
  1. SimpleEnum simpleEnum = SimpleEnum.White;
  2. bool isDefault; /* btw this one is 'false' by default */
  3.  
  4. isDefault = simpleEnum == default(SimpleEnum) ? true : false; /* default value 'white' */

 

Here we set the value to ‘Red’ and ‘default’ will tell us whether or not this the default value for this enum type. In this case: ‘false’.

'False' Case
  1. simpleEnum = SimpleEnum.Red; /* change from default */
  2. isDefault = simpleEnum == default(SimpleEnum) ? true : false; /* value is not default any longer */

Same 'default' functionality can also be applied to DateTimes, value types and other custom types as well.

Sweet ‘n Short. Happy Coding!

Using ConcurrentQueue for thread-safe Performance Bookkeeping.

Just a small tidbit that's sprung up today.

I had to book-keep and emit diagnostics for the average thread performance in a highly-threaded code over a period of last X number of calls and no more. Need of the day: a thread-safe, self-managing stats container.

Since .NET 4.0 introduced new thread-safe 'Collections.Concurrent' objects and I've been using them frequently - the one in particular seemed like a good fit for storing each threads' performance data - ConcurrentQueue.

But I wanted to store only the most recent X# of calls and since the ConcurrentQueue currently does not support size constraint I had to come up with my own generic version which attempts to restrict usage to numeric types only: unfortunately there is no IArithmetic-like interface which constrains to only numeric types – so the constraints here here aren't as elegant as they could be. (Note the use of the Average() method, of course you can use others as well as make your own).

 

FIFO FixedSizedConcurrentQueue
  1. using System;
  2. using System.Collections.Concurrent;
  3. using System.Linq;
  4.  
  5. namespace xxxxx.Data.Infrastructure
  6. {
  7.     [Serializable]
  8.     public class FixedSizedConcurrentQueue<T> where T : struct, IConvertible, IComparable<T>
  9.     {
  10.         private FixedSizedConcurrentQueue() { }
  11.  
  12.         public FixedSizedConcurrentQueue(ConcurrentQueue<T> queue)
  13.         {
  14.             _queue = queue;
  15.         }
  16.  
  17.         ConcurrentQueue<T> _queue = new ConcurrentQueue<T>();
  18.  
  19.         public int Size { get { return _queue.Count; } }
  20.         public double Average { get { return _queue.Average(arg => Convert.ToInt32(arg)); } }
  21.  
  22.         public int Limit { get; set; }
  23.         public void Enqueue(T obj)
  24.         {
  25.             _queue.Enqueue(obj);
  26.             lock (this)
  27.             {
  28.                 T @out;
  29.                 while (_queue.Count > Limit) _queue.TryDequeue(out @out);
  30.             }
  31.         }
  32.     }
  33.  
  34. }

 

The usage case is straight-forward, in this case I’m using a FIFO queue of maximum size of 200 to store doubles to which I simply Enqueue() the calculated rates:

Usage
  1. var RateQueue = new FixedSizedConcurrentQueue<double>(new ConcurrentQueue<double>()) { Limit = 200 }; /* greater size == longer history */

 

That’s about it. Happy coding!

Moving, moving, moving.

I'm moving my blog content to GeeksWithBlogs.com who have graciously accepted my application to host here. Over a period of time I'll be putting up my older posts from original sources to here (after careful editing to fit new format). The dates and ordering might be odd but I'll do my best to minimize the confusion. Cheers!