Posts
15
Comments
54
Trackbacks
0
Thursday, May 19, 2016
Semaphores–queuing with multiple workers–C#

One of the challenges of working with queues is how to properly and neatly process each item using multiple workers, luckily on .Net we have Semaphores. “In computer science, a semaphore is a variable or abstract data type that is used for controlling access, by multiple processes, to a common resource in a concurrent system such as a multiprogramming operating system.” https://en.wikipedia.org/wiki/Semaphore_(programming)

Below is a simple program that demonstrate the use of semaphore with a parametized number of threads and HttpRequest as Payload.

References:

https://msdn.microsoft.com/en-us/library/system.threading.semaphore(v=vs.110).aspx


private const int maxThreads = 10;
private Semaphore semaphore = new Semaphore(maxThreads, maxThreads);
private Queue queue = new Queue();
private void button1_Click(object sender, EventArgs e)
{
    addTestData();
     while (queue.Count >=1)
    {
        semaphore.WaitOne();
        var item = queue.Dequeue();
        ThreadPool.QueueUserWorkItem(ProcessCurrentItem, item);
    }
  
    int count = 0;
    while (count < maxThreads)
    {
        semaphore.WaitOne();
        ++count;
    }
}

async void ProcessCurrentItem(object item)
       {
           string responseData = "";
           HttpClient hc = new HttpClient();

           hc.DefaultRequestHeaders.Add("host", "www.mywebsite.com");
          hc.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36");
           hc.DefaultRequestHeaders.Add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
           hc.DefaultRequestHeaders.Add("scheme", "https");
           hc.DefaultRequestHeaders.Add("method", "GET");

           //Get the headers associated with the request.

           try
           {
               HttpResponseMessage response = await hc.GetAsync("https://www.mywebsite.com/" + item.ToString());
               responseData = response.ToString(); 

               //DO WHAT YOU WANT WITH YOUR DATA HERE

           } 

           catch (Exception ex)
           {
               Console.WriteLine(ex.Message.ToString());
           }
           Console.WriteLine(item.ToString());
           semaphore.Release();

       }

private void addTestData()
{
    queue.Enqueue("1.json");
    queue.Enqueue("2.json");
    queue.Enqueue("3.json");
    queue.Enqueue("4.json");
    queue.Enqueue("5.json");
    queue.Enqueue("6.json");
    queue.Enqueue("7.json");
    queue.Enqueue("8.json");
    queue.Enqueue("9.json");
    queue.Enqueue("10.json");
    queue.Enqueue("11.json");
    queue.Enqueue("12.json");
    queue.Enqueue("13.json");
    queue.Enqueue("14.json");
    queue.Enqueue("15.json");
    queue.Enqueue("16.json");
    queue.Enqueue("17.json");
    queue.Enqueue("18.json");
}

Posted On Thursday, May 19, 2016 9:20 PM | Comments (0)
Monday, February 29, 2016
Lock Pages In Memory

SQL Server together with other RDBMS, are the most memory consuming applications on our servers, and this is because, RDBMs usually cache objects into the memory to take advantage of the speed that physical memory offers.

Sadly when windows feels that its physical memory is currently not enough for a driver and/or processes that is requesting some resources, it is forced to trim some of currently running application’s memory working set. Now that is bad news for SQL Server because windows will be forced to push the objects from the memory to the servers’ paging file. You can verify if windows is doing this to your SQL Server by investigating SQL Server Logs for this entries

A significant part of sql server process memory has been paged out.This may result in a performance degradation. Duration: 0 seconds. Working set (KB): 1086400, committed (KB), memory Utilization: 50%.

Currently there are 2 settings that we can play with to avoid or at least alleviate this situation

1. Properly set SQL Servers’ Max Memory settings, by setting aside enough memory for the OS and other running Processes like Antivirus and Server Monitoring Software.

2. Enable SQL Server Locked Page in Memory (LPIM)

NOTE: Enabling LPIM without fully Understanding how SQL Memory Works and without knowing how your DB Server Behaves specially if it’s in a Virtual Environment might give you negative effects.

For this Post we will be focusing on LPIM. When using LPIM Windows cannot simply touch the memory space used by SQL Servers’ Buffer Pool, it is locked and cannot be paged. SQL Server does this by using Address Windowing Extensions (AWE).

When LPIM is enabled you cannot simply view how much really SQL Server is using by viewing Task Manager. As you can see on the screen shot below, SQLSERVR.EXE is only using 49, 536Kb

image

You can however use RamMap (A free RAM tool from Sysinternals: www.sysinternals.com). To view how much memory AWE is using.

image

 

Or you can also use sys.dm_os_process_memory SQL Server Dynamic Management Views (DMV).

 

How to Enable LPIM

Use Windows Group Policy tool (gpedit.msc) to enable this policy for the account used by SQL Server. You must be a system administrator to change this policy.

1. On the Start menu, click Run. In the Open box, type gpedit.msc.

2. On the Local Group Policy Editor console, expand Computer Configuration, and then expand Windows Settings.

3. Expand Security Settings, and then expand Local Policies.

4. Select the User Rights Assignment folder.

The policies will be displayed in the details pane.

5. In the pane, double-click Lock pages in memory.

6. In the Local Security Setting – Lock pages in memory dialog box, click Add User or Group.

7. In the Select Users, Service Accounts, or Groups dialog box, add an account with privileges to run sqlservr.exe.

8. Log out and then log back in for this change to take effect.

Note about SQL Server 2008 R2 Standard Edition (64-bit): Microsoft SQL Server 2008 R2 Standard Edition (64-bit, all versions RTM and later) also requires trace flag 845 to be added as a startup parameter so that SQL Server can use locked pages for the Buffer Pool when the SQL Server service account is granted the Lock Pages in Memory security privilege

Note about SQL Server 2012 Standard Edition (64-bit): Microsoft SQL Server 2012 Standard Edition (64-bit) does not require you to enable any trace flag to allow SQL Server use locked pages for the Buffer pool when the SQL Server service account is granted the Lock Pages in Memory security privilege.

 

How to determine if LPIM is enabled

Option 1 (Tested SQL 2005 )

DECLARE @LockPagesInMemory VARCHAR(255);

SET @LockPagesInMemory = 'UNKNOWN';

DECLARE @Res TABLE

(

[output] NVARCHAR(255) NULL

);

IF (SELECT value_in_use

FROM sys.configurations c

WHERE c.name = 'xp_cmdshell'

) = 1

BEGIN

INSERT INTO @Res

EXEC xp_cmdshell 'WHOAMI /PRIV';

IF EXISTS (SELECT *

FROM @Res

WHERE [output] LIKE 'SeLockMemoryPrivilege%'

)

SET @LockPagesInMemory = 'ENABLED';

ELSE

SET @LockPagesInMemory = 'DISABLED';

END

SELECT LockPagesInMemoryEnabled = @LockPagesInMemory;

Option 2 (Tested SQL 2008)

select osn.node_id,

osn.memory_node_id,

osn.node_state_desc,

omn.locked_page_allocations_kb

from sys.dm_os_memory_nodes omn

inner join sys.dm_os_nodes osn on (omn.memory_node_id = osn.memory_node_id)

where osn.node_state_desc <> 'ONLINE DAC'

NOTE: A non zero value for locked pages allocation means Locked pages in memory is enabled

Option 3 (TestedSQL 2008)

select

(physical_memory_in_use_kb/1024)Memory_usedby_Sqlserver_MB,

(locked_page_allocations_kb/1024 )Locked_pages_used_Sqlserver_MB,

(total_virtual_address_space_kb/1024 )Total_VAS_in_MB,

process_physical_memory_low,

process_virtual_memory_low

from sys. dm_os_process_memory

NOTE: A non zero Locked_pages_allocation_KB means Locked pages in memory is enabled.

Option 4

Using Exec xp

exec xp_readerrorlog 0, 1, 'locked pages'

exec xp_readerrorlog 0, 1, 'lock pages in memory'

Technical References/Further Reading:

Great SQL Server Debates: Lock Pages in Memory
https://www.simple-talk.com/sql/database-administration/great-sql-server-debates-lock-pages-in-memory/

How to reduce paging of buffer pool memory in the 64-bit version of SQL Server
https://support.microsoft.com/en-us/kb/918483

Support for Locked Pages on SQL Server Standard Edition (64-bit) systems

https://support.microsoft.com/en-us/kb/970070

How to enable the "locked pages" feature in SQL Server 2012
https://support.microsoft.com/en-us/kb/2659143

Posted On Monday, February 29, 2016 5:22 PM | Comments (0)
Friday, January 29, 2016
SQL Server 2016 Dynamic Data Masking (DDM)
SQL Server 2016 Dynamic Data Masking
Posted On Friday, January 29, 2016 11:38 PM | Comments (0)
Tuesday, June 17, 2014
Highcharts Export Tool Step by Step for IIS7 and .Net 2.0/4.0

1. Download Highcharts Export Module for ASP.Net From Tek4

2. Create New Website using IIS

3. Copy the contents of Exporting_Demo_website to the root folder of the newly created Web Site


4. Configure Application Pool entry for the new website
1. Open IIS Configuration Console 
2. Go to Application Pools
3. Right click on the name of the new application and select Advance Settings
4. Set Managed Pipeline Mode - Classic




5. Configure ISAPI and CGI restrictions
1. Open IIS Configuration Console 
2. Select the IIS server 
3. open ISAPI and CGI Restrictions
4. Select the .Net version that you require
5. put a check mark on "Allow extension path to execute"


6. Restart IIS Services

Test the Export Utility


Notes: 
1. Make sure to use the correct web.config file
2. You can test the site by opening default.aspx 
Posted On Tuesday, June 17, 2014 5:43 PM | Comments (1)
Saturday, December 1, 2012
ASP.Net MVC 4 Razor Engine For Beginners Part 1
Microsoft MVC Razor Engine ASP.Net Primer Beginner How To Step by Step Visual Studio MVC4
Posted On Saturday, December 1, 2012 10:43 PM | Comments (23)
Monday, June 13, 2011
Set your Most Popular Sites as your home page in Internet Explorer 9, IE9
Set your Most Popular Sites as your home page in Internet Explorer 9, IE9
Posted On Monday, June 13, 2011 8:20 PM | Comments (7)
Monday, May 16, 2011
CENTOS 5.6 httpd dead but subsys locked [RESOLVE]
CENTOS; CENTOS 5.6 httpd dead but subsys locked;
Posted On Monday, May 16, 2011 8:43 PM | Comments (9)
Wednesday, April 13, 2011
ADO.NET Entity Framework 4.1 Released
ADO.Net, Entity Framework
Posted On Wednesday, April 13, 2011 9:58 AM | Comments (1)
Thursday, April 7, 2011
Using Netbios-NS to get Information about a computer within a LAN
NBNS, Network Programming, Remote Host, Hostname, Mac Address
Posted On Thursday, April 7, 2011 12:45 PM | Comments (1)
Wednesday, March 23, 2011
Connect to SSIS on Named Instance of SQL Server
SQL Server
Posted On Wednesday, March 23, 2011 9:59 PM | Comments (0)
Tag Cloud