Geeks With Blogs

News This is the *old* blog. The new one is at blog.sixeyed.com
Elton Stoneman
This is the *old* blog. The new one is at blog.sixeyed.com

[Source: http://geekswithblogs.net/EltonStoneman]

I've had a need on a couple of projects for a monitoring system to record progress of a long-running transaction. Enter Heartbeat - a library which does just that. It's a project of mine on github: Heartbeat progress monitor.

The typical use-case is for monitoring a lengthy batch load, where Heartbeat will record progress at set intervals. The intervals can be time-based, or progress-based or both - so you can have an overnight process which writes a status update every 15 minutes, and for every 10,000th item it processes. Alternatively, for a Windows Service have a Heartbat to pulse every hour so you know it's still in a working state and how much work it's done since starting.

Heartbeat is threadsafe, so it's ideal for asynchronous processing which is actioned by spawning worker threads or using Task parallelism in .NET 4.0.

Usage

Usage is straightforward:

1. Initialise Heartbeat

Create a Heartbeat instance, passing in the object to be monitored, and the interval values for each type of pulse, and then start the timer:

    long countInterval = 7000;           
    double timerInterval = 300; //0.3 seconds
    var heartbeat = new Heartbeat(this, countInterval, timerInterval);
    heartbeat.Start();

2. Integrate Heartbeat with the working process

If you only want timed pulses, you don't need to do anything else – the Heartbeat instance will pulse at the set interval and write a log to the database. If you want additional pulses when your count interval is reached, then you need to increment the count when each item is processed:

    //do work...
    heartbeat.IncrementCount();

You can also subscribe to the OnPulse event, and on each pulse you can stop the log being written, or add your own custom message to the log:

    heartbeat.OnPulse += new Heartbeat.OnPulseEventHanlder(RunTimer_OnPulse);
    heartbeat.Start("RunTimer started, timerInterval: {0}, runTime: {1}".FormatWith(timerInterval, runTime));
    void RunTimer_OnPulse(PulseEventSource source, ref bool writeLog, ref string logText)
    {
        writeLog = true;
        logText = "RunTimer_OnPulse, source: {0}, text: {1}"
                    .FormatWith(source, RandomValueGenerator.GetRandomString());
    }

3. Tell Heartbeat when you're finished

Call SetComplete or SetFailed to log that the work is finished:

    var heartbeat = new hb.Heartbeat(this, countInterval, 0);
    heartbeat.Start("RunCount_NoHandler, countInterval: {0}, countTo: {1}".FormatWith(countInterval, countTo));
    try
    {
        for (int i = 0; i < countTo; i++)
        {
            heartbeat.IncrementCount();
        }
        var zero = 0;
        var dbz = 1 / zero;
        heartbeat.SetComplete("RunCount_NoHandler finished");
    }
    catch (Exception ex)
    {
        heartbeat.SetFailed("RunCount_NoHandler failed, message: {0}".FormatWith(ex.FullMessage()));
    }

If you don't make either call, then Heartbeat will write an UNKNOWN status log when the Heartbeat object is disposed.

Configuration

Rather than specify the pulse intervals for each Heartbeat instance you use, you can set default values at application level in your config file:

 <!-- set heartbeat defaults to pulse every 10 minutes & every 3,000 increments -->
 <sixeyed.heartbeatenabled="true"
                     defaultPulseTimerInterval="600000"
                     defaultPulseCountInterval="3000" />

The config section is optional, but if you don't have any config and don't set at least one of the pulse intervlas, your Heartbeat will never fire.

The Heartbeat Database

Heartbeat pulses are written to a simple database, consisting of a log table and two reference data tables:

To create the database (with C:\ as the default file location), run CREATE-Heartbeat.sql in the Database folder. In a working system I'd expect the Heartbeat tables to be added to an existing system database, so the script is just to get you started.

Each instance of a job has a unique ID, and the log records the full CLR assembly name of the object being monitored, the pulse intervals being used, the log time and status. The log also records the number of counts for the type of pulse, and the pulse statistics (elapsed milliseconds for timed pulses, number of items for count pulses). In raw form, the results of a time- and count- heartbeat session look like this:

HeartbeatLogId    HeartbeatInstanceId    ComponentTypeName    StatusCode    PulseTimerInterval    PulseCountInterval    LogDate    LogText    TimerPulseNumber    CountPulseNumber    TimerMilliseconds    CountNumber

376    BF341091-D0EB-498F-803A-B623EAD5BF16    Sixeyed.Heartbeat.Tests.HeartbeatTest, Sixeyed.Heartbeat.Tests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null    START     800    1000    2010-09-30 20:03:37.990    RunCountAndTimer_NoHandler, countInterval: 1000, countTo: 5692, timerInterval: 800, runTime: 4532    0    0    0    0

377    BF341091-D0EB-498F-803A-B623EAD5BF16    Sixeyed.Heartbeat.Tests.HeartbeatTest, Sixeyed.Heartbeat.Tests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null    WORKING     800    NULL    2010-09-30 20:03:38.803        1    NULL    811.2014    NULL

378    BF341091-D0EB-498F-803A-B623EAD5BF16    Sixeyed.Heartbeat.Tests.HeartbeatTest, Sixeyed.Heartbeat.Tests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null    WORKING     800    NULL    2010-09-30 20:03:39.613        2    NULL    1622.4029    NULL

411    BF341091-D0EB-498F-803A-B623EAD5BF16    Sixeyed.Heartbeat.Tests.HeartbeatTest, Sixeyed.Heartbeat.Tests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null    SUCCEED     800    1000    2010-09-30 20:04:01.610    RunCountAndTimer_NoHandler finished    29    5    23618.4415    5692

Storing the results in SQL Server provides the opportunity for simple SSRS dashboards, facilitated by the reference tables which give friendly names for statuses and allow you to store friendly names for applications (in HeartbeatApplications – xxx can be aliased to "User Load" for querying).

Implementation Notes

Heartbeat and Task.Factory.Start go very nicely together, but you need to be careful with how state is passed to the tasks. As Heartbeat implements IDisposable, if your worker class is also IDisposable then the Heartbeat instance could go out of scope and be disposed before your tasks are run. The safest pattern is to pass the task method a copy of the Heartbeat instance and any other variables it needs: 

    for (long i = 0; i < countTo; i++)
    {
        var heartbeat = _heartbeat; //don't pass the instance directly
        var taskIndex = i;
        tasks[i] = Task.Factory.StartNew(() => DoWork(heartbeat, taskIndex, finalTaskIndex));
    }
    private void DoWork(Heartbeat heartbeat, long taskIndex, long finalTaskIndex)
    {
        heartbeat.IncrementCount();
        //do work...
        if (taskIndex == finalTaskIndex)
        {
            heartbeat.SetComplete("StubComponent.Process finished");
        }

Also note the check to see if this is the final task – if so, the Heartbeat is set to complete. The task which is started last may not be the final task to complete, so the end time is not guaranteed to be accurate, but for processes running for several hours, the discrepancy is likely to be minimal.

Still to come…

  • Fancy SSRS reports showing a breakdown of jobs by completion status (failed, succeeded), average duration, historical run stats etc.
  • Options for using Heartbeat in distributed systems like NServiceBus and BizTalk.
   

 

Posted on Thursday, September 30, 2010 8:30 PM .NET 4.0 , github | Back to top


Comments on this post: Heartbeat: a progress monitor for long-running processes

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
Hi Mike,

good point and I was originally looking at log4net with a database appender. The main reason for not choosing that (or the event log or performance counters) is that the data you're logging there is unstructured.

With the custom solution, I can query a single batch process and see the progression through its workload - with structured attributes telling me elapsed time and work item count, as well as the unstructured log text.

I'll get some sample SSRS reports togeher in a follow-up post with the dashboard I have in mind, which is something that I think you'd only get from a custom, structured data approach.

Elton.
Left by Elton on Oct 01, 2010 9:10 AM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
Rather than a SQL db, why not use WMI here?
Left by andyclap on Oct 01, 2010 1:44 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
I like this article, i should try to learn this.
Left by Home Gadgets on Oct 21, 2010 5:22 AM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
I think this Heartbeat progress monitor will be helpful for all kind of projects, not only for long projects. And the illustration with the programming code was just great. Thanks for sharing this useful information.
Left by The Lotto BlackBook on Feb 25, 2011 5:10 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
I never knew you could monitor the progress of long-running processes like DBCC CHECKDB. But thanks that you did this job very well and shared with us. Keep sharing...
Left by lawn seeding on Mar 28, 2011 3:04 AM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...

The information has given here are so much informative.I was looking for all of this information
for a long time; it makes me so much pleased when I have found this information.Can I just say
what a aid to search out somebody who truly knows what they're talking about on the internet.
You positively know learn how to convey a problem to mild and make it important. More individuals must read this and understand this facet of the story. I cant consider you're no more well-liked since you undoubtedly have the gift.I hope he will write more blogs like this. Another thing all the information I have found from here are outstanding.
Left by 800 number fax on Jul 26, 2011 3:28 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
This is up to the mark. Analysis proves it's appropriateness with the logic.I agree with your conclusions and will eagerly look forward to your approaching updates.
Just saying thanks will not just be adequate, for the tremendous lucidity in your writing.I will instantly grab your rss feed to stay abreast of any updates. Authentic work and much success in your business efforts!Property “great document!” judgment which has no substance can be drab in addition to connected with interval.This is for the comfort of readers. From that aspect, to was successful to reach audience.
Left by call capture system on Jul 26, 2011 7:17 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
This is actually a Super knowledge gaining article and all thanks to bing search engine
get me on here.Just now I discovered the web for exactly this kind of information. Thank
to your post that search has to end right now. You wrote the post in a very understand
able way. With that, I added your blogs as one of my favorites!Will come back again -
taking you feeds also, Thanks.
Left by pediatric dentist mckinney tx on Aug 14, 2011 2:47 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...

There are several strong point of view which make us to believe on these information contribution which would be extremely paradigm shifting and conceptually stunning nevertheless I will try to alert my other friends , relatives, and significant others to spread this message
I got here. Keep informing such important logical information.

Outdoor furniture
Left by Outdoor furniture on Sep 22, 2011 8:30 AM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
So, regular health check of the network and service is needed to keep the service's standard up to the mark. As well as getting latest stuff like virtual VoIP system.I will try to alert my other friends , relatives, and significant others to spread this message.
Left by rubbish removal london on Sep 23, 2011 6:37 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...

This gonna be add new dimension to the information era surely.It looks like has a great potential to make
informatics effective than ever.The description was fine. Though there are several tools and equipments
for spreading news, this will make it's own place among them I hope.Looking forward for more interesting
and useful stuff like this.
Left by skip bins brisbane on Oct 10, 2011 12:39 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...

The article is actually the best topic on this registry related issue.It proved being in reality obliging to me plus I am constructive to each and every one the commenters here! It’s usually good when you can not just be informed, but in addition entertained! I am helpful you had pleasurable writing this.It network sense the attention of many readers expect from you in the door.

photofacials Atlanta
Left by photofacials Atlanta on Oct 17, 2011 3:41 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...


This is actually a Super knowledge gaining article and all thanks to bing search engine get me on here. I loved reading your post and added to the bookmarks. The views you tried to put up was clearly understandable. My sister also appreciated
after reading this article. I will browse for more sooner.
Left by internet fax number on Dec 08, 2011 5:11 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
Thanks for this lib. It's turning out to be pretty handy for keeping an eye on big batch loads.
Left by 800 numbers on Dec 09, 2011 8:29 PM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...

I recently came across your blog and have been reading posts. I thought I would leave a comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.
I wish you A very Happy Christmas and a prosperous New Year!!
north carolina wedding photographer
Left by north carolina wedding photogra on Jan 01, 2012 7:25 AM

# re: Heartbeat: a progress monitor for long-running processes
Requesting Gravatar...
I’ve been browsing on-line greater than three hours these days, yet I by no means discovered any fascinating article
like yours. It’s beautiful price sufficient for me. In my opinion, if all web owners and bloggers made good content
material as you did, the web will probably be much more useful than ever before.Thanks for all your efforts.
Left by xbox 720 on Apr 23, 2012 6:22 AM

Your comment:
 (will show your gravatar)


Copyright © Elton Stoneman | Powered by: GeeksWithBlogs.net