Paul Chapman

.Net Musings

  Home  |   Contact  |   Syndication    |   Login
  12 Posts | 0 Stories | 5 Comments | 0 Trackbacks

News

Twitter












Archives

Post Categories

Image Galleries

Silverlight links

Tuesday, May 26, 2009 #

How to add a Silverlight application to an Azure project.

Thursday, May 14, 2009 #

How to set up an Azure Site, with a webservice function

Wednesday, May 13, 2009 #

Project Sourcecode: NewsMashup

In Part I of this series of articles I discussed the database that lies behind the application. Moving from the backend to the client side, it is now time to discuss the code which accesses the database and returns the data to application.

I have a standard class I use when accessing databases. The version I am using here is developed to connect to SQL Server but with some minor changes it would not be difficult to adapt to other databases, and I already have one for ODBC.

public Database(string Connection)
{
    connection = new SqlConnection(Connection);
    InitialiseCommand();
}

public Database(SqlConnection Connection)
{
    connection = Connection;
    InitialiseCommand();
}

public Database()
{
    connection = new SqlConnection();
    InitialiseCommand();

}
 

The above code shows the constructor for the data access layer. Essentially the constructor expects either a connection, connection string or nothing to be passed to the constructor. Each calls another function; InitialiseCommand() which sets up the command object and is below.

private void InitialiseCommand()
{
    try
    {
        if (dbCommand == null)
        {
            dbCommand             = new SqlCommand();
            dbCommand.CommandType = CommandType.StoredProcedure;  // default to Stored Prc

            if (connection != null)
            {
                dbCommand.Connection = connection;
            }
        }
    }
    catch (System.Exception ex)
    {
        throw ex;
    }
}

 

This initialises the Command object and sets the type to Stored Procedure, if the Connection object has been initialised the command objects connection is set to the connection object.

To execute stored procedures we need two. One to return a datatable, and one which does not

    public void Execute()
    {
        try
        {
            if (dbCommand.Connection.State != ConnectionState.Open) dbCommand.Connection.Open();
            dbCommand.ExecuteNonQuery();
        }
        catch (System.Exception ex)
        {
            throw ex;
        }
    }

    public DataTable Execute(string TableName)
    {
        SqlDataAdapter  da;
        DataSet         ds;
        DataTable       dt = new DataTable();

        try
        {
            da = new SqlDataAdapter(dbCommand);
            ds = new DataSet();
            da.Fill(ds,TableName);

            if (ds.Tables.Count > 0)
                dt = ds.Tables[0];
        }
        catch (System.Exception ex)
        {
            throw ex;
        }

        return dt;
    }
}

 

 

The first of these two methods is fairly simple. Essentially what it does is confirm the connection is open, if not the connection is opened. Then it executes the stored procedure. The whole lot is then wrapped in a try catch, which is thrown back to the calling application.

The second is more interesting. It uses a Data Adapter and Dataset to fill a table returned back to the calling programme.

These two methods cover all the various types of database action I might want to undertake.

The real magic here however is in the objects which inherit from this class.

public class NewsMashupDB : Database,IDisposable

The class inherits from the class defined in the code above. The constructor for the object is as follows;

public NewsMashupDB() 
    : base()
{
    
}

public NewsMashupDB(string ConnectionString) 
    : base(ConnectionString)
{

}

Thats it. There is two here – one initialises the connection string, the other does not. All the calling application need do is retrieve the connection string. Executing a stored procedure is equally simple. Here we have the stored procedure which executes the CreateStory stored procedure;

public void CreateStory(string Headline, string Description, string Url, string Supplier, long PubDateValue)
{
    SqlParameter paramHeadline      = new SqlParameter("Headline", SqlDbType.VarChar);
    SqlParameter paramDescription   = new SqlParameter("Description",SqlDbType.VarChar);
    SqlParameter paramUrl           = new SqlParameter("Url",SqlDbType.VarChar);
    SqlParameter paramSupplier      = new SqlParameter("Supplier", SqlDbType.VarChar);
    SqlParameter paramPubDate       = new SqlParameter("PubDateValue", SqlDbType.BigInt);

    CommandText = "NewsMashup.CreateStory";

    try
    {

        paramHeadline.Value = Headline;
        paramDescription.Value = Description;
        paramUrl.Value = Url;
        paramSupplier.Value = Supplier;
        paramPubDate.Value = PubDateValue;

        dbCommand.Parameters.Clear();

        dbCommand.Parameters.Add(paramHeadline);
        dbCommand.Parameters.Add(paramDescription);
        dbCommand.Parameters.Add(paramUrl);
        dbCommand.Parameters.Add(paramSupplier);
        dbCommand.Parameters.Add(paramPubDate);

        Execute();
    }
    catch (System.Exception ex)
    {
        throw ex;
    }
}

Except for the code setting up the parameters required the execution of the stored procedure takes two lines; one to initialise the command text, the other to execute the stored procedure.

A stored procedure which returns data is equally simple.

public DataTable RetrieveNewsFeeds()
{
    DataTable dt = new DataTable();

    try
    {
        CommandText = "NewsMashup.RetrieveNewsFeeds";

        dt = Execute("NewsFeeds");

    }
    catch (System.Exception ex)
    {
        throw ex;
    }

    return dt;
}

It is clearer here than in the other method, again only two lines of code; one to initialise the CommandText the other to execute the stored procedure. This version of the Execute returns the datatable containing the results of the stored procedure’s execution.

This shows examples of the code. The rest should be in the attached zip file


Tuesday, May 12, 2009 #

If you have had a chance to look at the downloadable videos from MIX 09 (http://videos.visitmix.com/MIX09/All) then you will have heard that it will soon be possible to develop Azure hosted applications and have good old Transact SQL as a backend! It was in Nigel  Ellis’ session on ‘What's New in Microsoft SQL Data Services’ which gives all the details.

However as of the time of writing this functionality, while due in the next couple of months, is not yet available. All is not lost. I am going to use the local Azure development environment to host an application and point it at my local SQL Server instance until such time that I can host it online.

While writing this application there were a number of gotcha’s – some because Azure is beta, some not. The follow is how I wrote a small application to show news items, that constantly updates. The backend is a SQL database undated on a regular basis by an Azure Worker Role, then read by a Web Service called from a Silverlight application, itself hosted on an Azure WebRole.

In developing this application, which I will be adapting for a much larger online one I am working on I will work from the Backend to the Silverlight application which makes up the client end.

Creating our database

According to the SQL Data Services session access to the database; either locally or from SQL Data Services will be via SQL Management Studio. Once the a database has been created it is time to create the tables that will store the details of our new stories;

This project is actually part of a much larger one I am creating. For this reason I want to logically separate parts of the database using schemas. This for example will allow one database to have two tables called ledger – one in a schema called ‘Sales’ and one called ‘Purchases’. You will notice that all of the database objects created in the following examples of T/SQL are preceded by the word ‘NewsMashup’. To create this schema execute the following code against the SQL database;

 

A Tables to store the sites from which feeds will be taken – this allows us to store a hyperlink to the parent site.

One to store the details of feeds from which stories will be taken;

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [NewsMashup].[Feeds](
	[Id] [bigint] IDENTITY(1,1) NOT NULL,
	[Url] [varchar](max) COLLATE Latin1_General_CI_AS NOT NULL,
	[Supplier] [bigint] NULL,
 CONSTRAINT [PK_NewsFeeds] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
)WITH (PAD_INDEX  = OFF, IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

GO
SET ANSI_PADDING OFF
GO
ALTER TABLE [NewsMashup].[Feeds]  WITH CHECK ADD  CONSTRAINT [FK_NewsFeeds_Supplier] FOREIGN KEY([Supplier])
REFERENCES [NewsMashup].[Supplier] ([Id])
GO
ALTER TABLE [NewsMashup].[Feeds] CHECK CONSTRAINT [FK_NewsFeeds_Supplier]

 

 

One table to store Stories as they are picked up

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [NewsMashup].[Stories](
	[Id] [bigint] NOT NULL,
	[Headline] [varchar](max) COLLATE Latin1_General_CI_AS NOT NULL,
	[Description] [varchar](max) COLLATE Latin1_General_CI_AS NOT NULL,
	[Url] [varchar](max) COLLATE Latin1_General_CI_AS NOT NULL,
	[Supplier] [bigint] NOT NULL
) ON [PRIMARY]

GO
SET ANSI_PADDING OFF

 

 

and finally one table to store the sites from which the feeds will be taken

ALTER Procedure [NewsMashup].[RetrieveNewsFeeds]
As
Begin
	Select		NewsMashup.Supplier.Id			As Supplier,
				NewsMashup.Supplier.Name, 
				NewsMashup.Supplier.Url			As HomePage,
				NewsMashup.Feeds.Url
	From		NewsMashup.Feeds 
				INNER JOIN NewsMashup.Supplier ON 
					NewsMashup.Feeds.Supplier = NewsMashup.Supplier.Id
	Order By	NewsMashup.Supplier.Name
End

Creating Stored Procedures to access and write data

In placing all database queries in a stored procedure you gain a number of advantages. Stored Procedures can be faster than inline code, partly this is because the database is able to partly compile and optimise your query. With inline code this happens at execution time. For the purposes of an application to display news headlines there are four stored procedures; the first to Create our Stories;

ALTER Procedure [NewsMashup].[CreateStory]
	@Headline		VarChar(Max),
	@Description	VarChar(Max),
	@Url			VarChar(Max),
	@Supplier		BigInt,
	@PubDateValue	BigInt
As
Begin
	Declare @Counter		BigInt
	
	Set @Counter = (Select Count(*) As PreviousCounter From NewsMashup.Stories 
					Where Url = @Url)

	If @Counter = 0 
	Begin
		INSERT	INTO NewsMashup.Stories
			(Headline, Description, Url, Supplier, Id)
		VALUES	(@Headline,@Description,@Url,@Supplier,@PubDateValue)
	End 
End

It’s fairly simple this function. We first check to see if the link has already been added to the database. If not then the story is added. The next two procedures are both involved in retrieving headlines; we need two – one is used to retrieve the latest 10 stories from the database, the other retrieves any subsequent stories that may have been added to the database. This allows new stories to be added to the display as and when they are picked up.

CREATE SCHEMA [NewsMashup] AUTHORIZATION [dbo]

Retrieving the most recent 10 headlines 

ALTER PROCEDURE [NewsMashup].[RetrieveHeadlines]
AS
BEGIN
	
	SET NOCOUNT ON;
	
	SELECT     TOP (10) NewsMashup.Stories.Id, NewsMashup.Stories.Headline, NewsMashup.Stories.Description, NewsMashup.Stories.Url, NewsMashup.Supplier.HasDescription, 
						  NewsMashup.Supplier.Name AS Supplier
	FROM         NewsMashup.Stories INNER JOIN
						  NewsMashup.Supplier ON NewsMashup.Stories.Supplier = NewsMashup.Supplier.Id
	ORDER BY NewsMashup.Stories.Id DESC
END

Retrieving the latest new headlines

ALTER PROCEDURE [NewsMashup].[RetrieveHeadlines2]
	@LatestHeadline	BigInt
AS
BEGIN

	SET NOCOUNT ON;

	SELECT     Id, Headline, Description, Url, Supplier
	FROM         NewsMashup.Stories
	WHERE     (Id > @LatestHeadline)
	ORDER BY Id 
END

 

The final stored procedure retrieves the list of feeds from which headlines will be extracted.

ALTER Procedure [NewsMashup].[RetrieveNewsFeeds]
As
Begin
	Select		NewsMashup.Supplier.Id			As Supplier,
				NewsMashup.Supplier.Name, 
				NewsMashup.Supplier.Url			As HomePage,
				NewsMashup.Feeds.Url
	From		NewsMashup.Feeds 
				INNER JOIN NewsMashup.Supplier ON 
					NewsMashup.Feeds.Supplier = NewsMashup.Supplier.Id
	Order By	NewsMashup.Supplier.Name
End

 

That just about creates the database structures required to stored the News Headlines picked up by the application. Part II I will discuss the .Net libraries used to access these procedures.

 

 


Tuesday, March 31, 2009 #

Creating a vertical scrollable region in Silverlight

Wednesday, January 07, 2009 #

The current Azure CTP has a problem correctly connecting to Web Services. This article details how you can complete one of the Azure Labs to correctly connect to a Web Service.

Friday, January 02, 2009 #

I have been playing around with Microsoft Azure. I was interested because one of my current development projects has a shoestring budget (read no budget). MS Azure will allow for the hosting costs to increase with the site's success.

So it was time to have a go at using Azure's storage functionality to store the data for the site. To this end I downloaded a couple of storage demonstrations to see how it works. With Azure being a beta and a fair new technology looking at demos is often the best way of learning it. These demonstrations were  Windows Azure Walkthrough: Simple Table Storage and Windows Azure Blog Source Code from PDC.

Both exhibited the same problem; and threw the following error

System.Data.Services.Client.DataServiceClientException: System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:10002
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetRequestStream()
at System.Data.Services.Client.DataServiceContext.SaveAsyncResult.BeginNextChange(Boolean replaceOnUpdate)

The answer appears in the comments of the latter post. It turned out that the Table Service for the solution was not running. To check it's status when running the Azure solution two icons appear in the system tray labelled. It is the development storage we are interested in. To prevent this error simple start the Table Service.

image

I found that once this had been started once, the error (so far) has not re-occurred, even with other projects.


Thursday, November 20, 2008 #

The basis for much growth in science has been the sharing of ideas. Ideas get passed around, question, examined, peer reviewed and eventually built on. This system has worked for centuries, taking man from the discovery of fire - to the moon and back. Computer software is much the same. Ideas thought up, built on and improved - this has taken computing from machines requiring a legion of white clad acolytes and air conditioned offices to something I can slip into my shirt pocket, read emails, send instance messages, store my music, video and pictures. Excel built upon changes and improvements to the Spreadsheet originally written in the early 80's.

Innovation has thrived in environments where one can freely improve and build upon the work of others - can you imagine what would have happened if Daimler Benz had opted to enforce a patent on the motor car! No Ford, no Chrysler, manufacturing may have not changed with the assembly line. And forget learning to drive when each car has to be different in order not to infringe some patent!

So anyway it seems some wit in the US Patent has decided to give Apple a patent on the Dock which forms part of OS X. It would help but from this developers view the Dock does not seem to be that original - it is a menu, some nice bells and whistles but a menu none the less. It was Edison who said 1% inspiration, 99% perspiration. That is true - It would have taken some time to write the code - and that is protected certainly in the UK by copyright. Even if I could get the source code - I cannot copy it to implement my own Dock. So what if I want to improve on the Dock - first I have to pay a large fee to Apple's patent lawyers. Why, thinking up the dock is just the start - I'm still going to have to do the 99% of sweat to implement my dock and my whiz bang improvement. So why bother, why sweat for my 99% and give a lawyer most of the money. So one does not build upon the original, but wastes time trying to find another way when the Dock might fit the bill.

Why do companies do this? Apple produces some excellent hardware - better in many respects than the PC - but it is the PC which has most of the market - and it is open, anyone can write software for the PC and anyone can build a compatible PC and sell it. Companies which start to rely on patents cease to be innovative, because the competition cannot build upon the original product and force the original company to innovate. The company rationale moves therefore from innovative to just protecting a patent; which itself becomes less use as people avoid infringing the patent - although this point does not stop companies being formed for the sole purposes of profiting from these things.

It does not help but to a greater or less extent the dock is not that original; at it's heart it is a menu - something you could have picked up in any restaurant  back when a PC was something in a sci-fi movie!

Something not a million miles from the Dock has been seen on Risc OS, Silicon Graphix. You could drag applications to the toolbar in Windows 95! Long before OS X came out.

This is not the first time Apple have tried this - anyone remember Windows 1/2 - all those luvly tiled windows because of some daft agreement between MS and Apple - ended up in court eventually and Apple lost - why? because Rank Xerox had done it before - I was using overlapping windows on Unix Workstations long before the MAC.

So come on Apple - the New Macbooks show what you can do when you innovate. So stop the lawyers and innovate. And will someone teach the US patent office a little history of computing please.


Thursday, September 04, 2008 #

In the years since the .Net runtime was first released I've seen a number of ways of accessing a SQL database from within VB.Net (or C#). One person I worked with had a nifty utility which would take any database and generate all the code to access it. It has been on my list of jobs to do to implement my own such tool but like other jobs, such as winning the National Lottery or winning a high stakes poker game I have never got around to it. OK the poker game is something my wife will have to do - she is better at poker than me.

What I do have is a class which encapsulates the functions I need to execute stored procedures on the database. All my database access is executed by inheriting from the following class;

Imports Microsoft.VisualBasic
Imports System.Data
Imports System.Data.SqlClient

Imports System
Imports System.Xml
Imports System.Xml.Linq

Public MustInherit Class DataEngine
    Protected _Connection As SqlConnection
    Protected _Command As SqlCommand

    Sub New()
        _Connection = New SqlConnection()
        _Command = New SqlCommand()
    End
Sub

    Sub New(ByVal ConnectionString As String)
        _Connection = New SqlConnection(ConnectionString)
        _Command = New SqlCommand()
    End
Sub

    Protected Property ConnectionString As
String
       
Get
            Return _Connection.ConnectionString
        End
Get
        Set(ByVal value As String)
            _Connection.ConnectionString = value   
        End
Set
    End
Property

    Protected Function ExecuteStoredProcedure(ProcedureName As String, Table As String) As DataTable
        Dim _DataTable      As DataTable
        Dim _DataAdapter    As SqlDataAdapter
        Dim _DataSet        As DataSet = New DataSet()

       
Try

            If _Connection.State = ConnectionState.Closed Then
                _Connection.Open()
            End
If

            With _Command
                .CommandText = ProcedureName
                .CommandType = CommandType.StoredProcedure
                .Connection = _Connection
            End
With

            _DataAdapter = New SqlDataAdapter(_Command)

            _DataAdapter.Fill(_DataSet,Table)
            _DataTable = _DataSet.Tables(Table)
        Catch ex As Exception
            Throw ex
        End
Try

        Return _DataTable
    End
Function

    Protected Sub ExecuteStoredProcedure(ProcedureName As String)
       
Try
           
            If _Connection.State = ConnectionState.Closed Then
                _Connection.Open()
            End
If

            With _Command
                .CommandText = ProcedureName
                .CommandType = CommandType.StoredProcedure
                .Connection = _Connection   
     
                .ExecuteNonQuery()
            End With 

        Catch ex As Exception
            Throw ex
        End
Try
    End Sub


    Protected Property Command As SqlCommand
       
Get
            Return _Command
        End
Get
        Set(ByVal value As SqlCommand)
            _Command = value
        End
Set
    End
Property

End
Class

First thing to notice about this class is that it is marked MustInherit you cannot directly create an instance of this class. What you do is create a class which inherits from this class. In this way the common tasks required; which summarised is open a connection to the database and execute stored procedures. There are two types essentially; a stored procedure which returns a table, and one that does not.

The class has two constructors. One that creates the basic database access objects; the connection and the command, and another which allows us to pass a connection string.

Next is a property which exposes the connection string to any child of this class. This is the other means for setting the connection string, although to be honest as a general rule I prefer to do that at the when an instance is created.

The next two procedures take advantage of overloading, allowing us to pass an optional table name for the results.

Finally the last property allows access to the command object.

We can use this class to access the database created earlier with the following class;

Imports Microsoft.VisualBasic
Imports System.Data
Imports System.Data.SqlClient
Imports System.Configuration.ConfigurationManager

Public Class SimplicitaDatabase
        Inherits DataEngine

    Public Sub
New

        ConnectionString = ConnectionStrings("SimplicitaDB").ConnectionString

    End
Sub

    Public Function Register(Name As String,Address1 As String, Address2 As String, Town As String, County As String, Postcode As String, Tel As String,Fax As String, Email As String, WebAddress As String,Password As String) As DataTable
        Dim dt As DataTable = New DataTable()


        Dim SqlParamName        As SqlParameter = New SqlParameter("Name",SqlDbType.VarChar)
        Dim SqlParamAddress1    As SqlParameter = New SqlParameter("Address1",SqlDbType.VarChar)
        Dim SqlParamAddress2    As SqlParameter = New SqlParameter("Address2",SqlDbType.VarChar)
        Dim SqlParamTown        As SqlParameter = New SqlParameter("Town",SqlDbType.VarChar)
        Dim SqlParamCounty      As SqlParameter = New SqlParameter("County",SqlDbType.VarChar)
        Dim SqlParamPostcode    As SqlParameter = New SqlParameter("Postcode",SqlDbType.VarChar)
        Dim SqlParamTel         As SqlParameter = New SqlParameter("Telephone",SqlDbType.VarChar)
        Dim SqlParamFax         As SqlParameter = New SqlParameter("Fax",SqlDbType.VarChar)
        Dim SqlParamEmail       As SqlParameter = New SqlParameter("Email",SqlDbType.VarChar)
        Dim SqlParamWebAddr     As SqlParameter = New SqlParameter("WebAddress",SqlDbType.VarChar)
        Dim SqlParamPassword    As SqlParameter = New SqlParameter("Password",SqlDbType.VarChar)
       
       
Try

            SqlParamName.Value      = Name
            SqlParamAddress1.Value  = Address1
            SqlParamAddress2.Value  = Address2
            SqlParamTown.Value      = Town
            SqlParamCounty.Value    = County
            SqlParamPostcode.Value  = Postcode
            SqlParamTel.Value       = Tel
            SqlParamFax.Value       = Fax
            SqlParamEmail.Value     = Email
            SqlParamWebAddr.Value   = WebAddress
            SqlParamPassword.Value  = Password

            With Command.Parameters
                .Clear()
                .Add(SqlParamName)
                .Add(SqlParamAddress1)
                .Add(SqlParamAddress2)
                .Add(SqlParamTown)
                .Add(SqlParamCounty)
                .Add(SqlParamPostcode)
                .Add(SqlParamTel)
                .Add(SqlParamFax)
                .Add(SqlParamEmail)
                .Add(SqlParamWebAddr)
                .Add(SqlParamPassword)
            End
With

            dt = ExecuteStoredProcedure("RegisteredUser.Register","Result")

        Catch ex As Exception
            Throw
        End
Try
       
        Return dt
    End
Function

    Public Function RetrieveNewsfeeds As DataTable
        Dim dt As DataTable = New DataTable

        With Command.Parameters
            .Clear()
        End
With

        dt = ExecuteStoredProcedure("NewsAggregator.RetrieveNewsFeeds","NewsFeeds")

        Return dt
    End
Function

    Public Function RetrieveLatestStories As DataTable
        Dim dt As DataTable

        With Command.Parameters
            .Clear()
        End
With

        dt = ExecuteStoredProcedure("NewsAggregator.RetrieveLatestStories","NewsStories")

        Return dt
    End
Function

    Public Sub WriteNewsStory ( Title As String, Link As String, Desc As String, Supplier As String)
        Dim SqlParamTitle       As SqlParameter = New SqlParameter("Title",SqlDbType.VarChar)
        Dim SqlParamLink        As SqlParameter = New SqlParameter("Link",SqlDbType.VarChar)
        Dim SqlParamDesc        As SqlParameter = New SqlParameter("Desc",SqlDbType.VarChar)
        Dim SqlParamSupplier    As SqlParameter = New SqlParameter("Supplier",SqlDbType.VarChar)

        SqlParamTitle.Value     = Title
        SqlParamLink.Value      = Link
        SqlParamDesc.Value      = Desc
        SqlParamSupplier.Value  = Supplier

        With Command.Parameters
            .Clear()
            .Add(SqlParamTitle)
            .Add(SqlParamLink)
            .Add(SqlParamDesc)
            .Add(SqlParamSupplier)
           
        End
With

        ExecuteStoredProcedure("NewsAggregator.WriteNewsStory")
       
    End
Sub
End
Class

And that is the code for accessing the database created in part one.


Wednesday, September 03, 2008 #

For one of my projects I need to have Business News headlines displayed on the front page, to give me greater control as to which headlines and from which feeds I use I'm going to have the aggregator call each feed and store the article details in a SQL database.

Schemas

I'm going to use schemas in designing my database. The use of these changed in SQL 2005 so that each one is a distinct namespace, independent of the user who created it. This will allow me to group related tables together, but it also allows you to use the same table name for different items. For example in an accounts system you have Sales accounts and Purchase accounts. You could of course create one table called SalesAccount and another called PurchaseAccount. With schemas we create ourselves two schemas one called SalesLedger and one called PurchaseLedger. Each schema has a table called 'Accounts'.

So first task is to create the schema that will hold our News. The following line of T/SQL will create our Namespace;

CREATE SCHEMA [NewsAggregator] AUTHORIZATION [dbo]
GO

Creating our tables

Now we have our schema the next thing is the tables that will store the feeds we are going to read, and the news headlines. Not a lot special in these ones to be honest. We need two tables; one to store the feeds that we will be aggregating and the second to store the headlines. The following T/SQL will create these tables;

CREATE TABLE [NewsAggregator].[NewsStories](
                  [Id] [bigint] IDENTITY(1,1) NOT NULL,
                  [Title] [varchar](127) NOT NULL,
                  [Link] [varchar](max) NOT NULL,
                  [Desc] [varchar](max) NULL,
                  [Supplier] [varchar](20) NOT NULL,
                  [DateAdded] [datetime] NOT NULL
                      
CONSTRAINT [DF_NewsStories_DateAdded]
                      
DEFAULT (getdate())
             )
ON [PRIMARY]

 

CREATE TABLE [NewsAggregator].[NewsFeeds](
                  [Id] [bigint] IDENTITY(1,1) NOT NULL,
                  [Link] [varchar](max) NOT NULL,
                  [Supplier] [varchar](30) NOT NULL,
                  [Weight] [smallint] NULL
             )
ON [PRIMARY]

So there is our two tables. I have added a 'weight' field to the table of feeds. How this works will be clear later, however it is here to allow us to control how many articles from each feed we are displaying.

Now when developing I like to keep things separated. For this reason I like to do all my database access via Stored Procedure. The practical upshot of this is that any differences in the dialect of SQL you are using should be hidden behind the stored procedure making it a lot easier to covert the whole thing to another database.

Stored Procedures

Our feeds are retrieved by making a call to RetrieveNewsFeeds. Taking note of the fact that I am using a schema called NewsAggregator the code to create this stored procedure is as follows;

CREATE PROCEDURE [NewsAggregator].[RetrieveNewsFeeds]
AS
BEGIN
SET NOCOUNT ON
;
     SELECT Id,
            Link
,
            Supplier
     FROM   NewsAggregator
.NewsFeeds
END

This simply returns the results of select statement. I'm not interested in the order so to speed things along no sorting is done. Next having retrieved our news feeds we need to have a means of writing the headlines to our database;

CREATE PROCEDURE [NewsAggregator].[WriteNewsStory]
      @Title      VarChar(127),
      @Link       VarChar(MAX),
      @Desc       VarChar(MAX),
      @Supplier   VarChar(20)
AS
BEGIN

      Declare @StoryCount BigInt

      SET NOCOUNT ON;

      Set @StoryCount = (Select Count(*) As Counter From NewsAggregator.NewsStories
                         Where Link = @Link)

      If @StoryCount = 0
      Begin
            INSERT INTO NewsAggregator.NewsStories
                        (Title, Link, [Desc], Supplier)
            VALUES      (@Title,@Link,@Desc,@Supplier)
      End
END

This is fairly simple as a stored procedure. We first look for the story link in the database to see if it has been read before. If not then the story gets added to the database.

Now we have our feed suppliers, and we can write the stories to our database. The final stored procedure is to recover the files from the database.

First off the procedure opens up a cursor to access the list of feed suppliers. Normally I would not use a cursor which are notoriously slow in SQL Server but it does provide a simple mechanism for progressing through a table.

CREATE PROCEDURE [NewsAggregator].[RetrieveLatestStories]
AS
BEGIN
    
Declare @Supplier VarChar(30)
     Declare @Weight SmallInt
     Declare @Result As Table
     (
          Title VarChar(127),
          Link VarChar(Max),
          [Desc] VarChar(Max)
     )
    
     Declare
FeedSuppliers Cursor
For
          SELECT Supplier, 
                 Weight
          FROM   NewsAggregator.NewsFeeds

     Open
FeedSuppliers
     Fetch FeedSuppliers Into @Supplier,@Weight

     While @@Fetch_Status = 0
     Begin
          Insert Into @Result (Title,Link,[Desc])
          Select   Top (@Weight)Title,Link,[Desc]
          From     NewsAggregator.NewsStories
          Where    Supplier = @Supplier
          Order By DateAdded Desc

          Fetch FeedSuppliers Into @Supplier,@Weight
     End

     Close
FeedSuppliers
     Deallocate FeedSuppliers

     Select * From @Result
END

 

While we are moving through the table of suppliers we retrieve from the list of stories the latest for each supplier based on the weight. If the weight is 1 then we retrieve 1 story, 2 and 2 stories 3, 3 stories etc. This weight places a bias in the display of stories allowing us to control the stories from each supplier.
 
and the following will store the news headlines