Geeks With Blogs
Alex's Blog-o-monium

For one of my projects I've been using Windows Server AppFabric Caching (on-premise) (http://tinyurl.com/2usm9bs).

I've setup a simple cluster: 2 nodes, with 4 Gigabytes in total, running in a high-availability mode – data is auto-replicated, so I can take down one node for maintenance (or allow for one-node asteroid strike) and client applications will still keep on trucking.

After a couple days of tweaking settings it all started working out nicely: (at worst) double-digit millisecond read times, good locking synchronization, region partitioning and all else seemed to be working as advertised. However, as the volume ramped up I started noticing some slowdowns. After digging a bit into it I identified possible culprit: DataCacheFactory.

MSDN API documentation didn't address this much, the documented approach was simple: instantiate DataCacheFactory and call GetCache() (or in my case GetDefaultCache())
to access the data. Great. But what does DataCacheFactory do exactly?

According to Jason Roth from MS (http://tinyurl.com/d7sk5s4):

"Creating the factory involves connecting to the cluster and can take some time. But once you have the factory object and the cache that you want to work with, you can simply reuse those  objects to do puts and gets into the cache, and you should see much faster performance".

Ok, I get it. Basically, DataCacheFactory instantiation is expensive so doing this for every call will not scale up. The suggestion is to essentially use a static DataCacheFactory object and reuse it.

Unfortunately, one of my applications leveraging AppFabric has very demanding throughput requirements and thus it’s quite asynchronous (a few hundred concurrent threads) so having just one point of entry eventually started to cause issues. Simply put it - couldn't handle the load.

So, my options so far:
a) Instantiate new DataCacheFactory on each call and get slow performance.
b) Use a static DataCacheFactory instance and get frequent thread contentions (which is not quite the end of the world - in case of cache failure my code automatically falls back to using SQL but this sort of defeats the purpose of having distributed cache in place.. obviously).

Both options are sub-par. What to do? Answer - Pooling!

There's no built-in support in 1.0/1.1 AppFabric for pooling of this sort, so I had to build my own. Where do we keep this pool? Nothing too fancy, just a simple List:

  1. /// <summary>
  2. /// Pool of cache data access objects
  3. /// </summary>
  4. private List<DataCache> _dataCachePool = new List<DataCache>(CachingProviderConfiguration.PoolSize);

 

I'm using a thread-safe lazy singleton pattern for the Caching provider so there's one instance in AppDomain and the constructor will run once per AppDomain as well:

  1. private static volatile Lazy<CachingProvider> _instance = new Lazy<CachingProvider>(() => new CachingProvider(), LazyThreadSafetyMode.ExecutionAndPublication);
  2. public static CachingProvider Instance { get { return _instance.Value; } }

 

..And this is where we want to initialize this pool, it may take some time but only happens once per AppDomain and then we're free of this burden:

  1. /// <summary>
  2. /// Prevents a default instance of the <see cref="CachingProvider"/> class from being created.
  3. /// </summary>
  4. private CachingProvider()
  5. {
  6.     for (int i = 0; i <= CachingProviderConfiguration.PoolSize; i++)
  7.         /* initializing cache data access objects pool */
  8.         _dataCachePool.Add(this.GetDataCacheInstance());
  9. }

DataCache object instantiation is where (the instigator of all this) – DataCacheFactory is accessed, otherwise the code is fairly trivial:

  1. /// <summary>
  2. /// DataCache property. Returns DataCache instance.
  3. /// Sets up data points for clustered nodes, configuration and security.
  4. /// </summary>
  5. private DataCache GetDataCacheInstance()
  6. {
  7.     var servers = new List<DataCacheServerEndpoint>();
  8.     var configuration = new DataCacheFactoryConfiguration();
  9.  
  10.     servers.Add(new DataCacheServerEndpoint(CachingProviderConfiguration.Node1, CachingProviderConfiguration.Node1Port)); /* lead node */
  11.     servers.Add(new DataCacheServerEndpoint(CachingProviderConfiguration.Node2, CachingProviderConfiguration.Node2Port)); /* backup node */
  12.  
  13.     configuration.Servers = servers;
  14.  
  15.     /* set default properties for local cache (local cache disabled) */
  16.     configuration.LocalCacheProperties = new DataCacheLocalCacheProperties();
  17.  
  18.     /* disable tracing to avoid informational/verbose messages */
  19.     DataCacheClientLogManager.ChangeLogLevel(TraceLevel.Off);
  20.  
  21.     /* no security is required, keep it light */
  22.     configuration.SecurityProperties = new DataCacheSecurity(DataCacheSecurityMode.None, DataCacheProtectionLevel.None);
  23.  
  24.     configuration.RequestTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.RequestTimeout);
  25.     configuration.ChannelOpenTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.ChannelOpenTimeout);
  26.     configuration.TransportProperties.ReceiveTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.ReceiveTimeout);
  27.     configuration.TransportProperties.ChannelInitializationTimeout = TimeSpan.FromSeconds(CachingProviderConfiguration.ChannelInitializationTimeout);
  28.  
  29.     //configuration.TransportProperties.ConnectionBufferSize = CachingProviderConfiguration.MaxBufferSize;
  30.     configuration.TransportProperties.MaxBufferPoolSize = CachingProviderConfiguration.MaxBufferPoolSize;
  31.     configuration.TransportProperties.MaxBufferSize = CachingProviderConfiguration.MaxBufferSize;
  32.  
  33.     _dataCacheFactory = new DataCacheFactory(configuration);
  34.  
  35.     return _dataCacheFactory.GetDefaultCache();
  36. }

 

Almost done! So how do we access this pool? Ideally, we must balance the throughput across each item in the pool so a round-robin access comes to mind.

But you don't really have to do this. A random access to an item in the pool for each call will do just as well, considering random generation is uniformly distributed and we don't expect the pool to be in triple digits. One last piece of code:

  1. private Random _instanceIndexGenerator = new Random();
  2. /// <summary>
  3. /// Gets randomly chosen data cache access object from the pool.
  4. /// Load Balancer on the pool.
  5. /// </summary>
  6. private DataCache DataCache
  7. {
  8.     get
  9.     {
  10.         return _dataCachePool[_instanceIndexGenerator.Next(CachingProviderConfiguration.PoolSize)];
  11.     }
  12. }

 

..which can be accessed inside the provider as such: DataCache.Get("your-object-key-goes-here")
       
That's it! Happy coding!

Posted on Friday, August 17, 2012 8:40 AM | Back to top

Copyright © Strenium | Powered by: GeeksWithBlogs.net