Geeks With Blogs

News Dave's Mug View David Oliver's profile on LinkedIn Add to Technorati Favorites Blog Directory for Guildford, Surrey
Dave Oliver's Blog Enterprise Technology Thought Leadership in a FTSE 100

Does your software or database run like a dog? Running out of space? Or doesn't recover well in a disaster? ... then your storage configuration may be one of the reasons. This is a short little post just to give you a heads-up on the basics. You may know bits of it already, hence why I have broken it up into a Q&A format so you can skip what you already now.

The Hard-Disk

Well we've all seen a hard-disk but the likelihood is that it was in a desktop and you're the only person that uses it at anyone time? Have you noticed that if you run more than one thing that accesses the disk at the same time, it sometimes slow down?

Hard-disk's are essentially serial devices where a head can only really read or write one thing at a time, they have got cleverer in recent years but still it's a bunch of heads hovering over a set of platters, each head taking it's turn to read or write.

These day's in servers and storage devices you are likely to find are SAS drives or Serial Attached SCSI. You would think that Parallel SCSI, the old standard, would be faster but as speeds increases the problem of timings and keeping data in sync or 'skewed' was proving difficult and expensive. SATA is self 'skewing' so we will see much larger increases in bandwidth in the next few years, typically it is 3Gb/s now this is likely to rise to 12Gb/s in the next few years.


Increasing performance has meant getting more disks then spreading segments of a file across them so each set of heads can read a part of the file at the same time as others speeding up the loading of a file. Hard-Disk in this configuration are said to be 'Stripped'.


Hard-disks have a life, they have moving part and will ware out in-time. However, we can't accurately predict when it will break-down so the simplest form of protecting the data on the disk from it's inevitable failure is making a copy. Backing up the data to another media takes time. To reduce the risk, data is copied to another disk called a mirror at around the same time it as the original. If the original fails the mirror takes over until the original is replace, then the mirror copies it contents to the new original. This is called unsurprisingly 'Mirroring'.

Parity/Error Correction?

Redundancy data is created and stored to allow problems to be detected and/or fixed. The redundancy data is calculated from sets of actual data values.

In detail, since computer data is stored as binary numbers it can use Boolean operators to transform data. One of these operators is Exclusive OR (XOR). The interesting and useful thing about XOR is that if performed twice in a row, it "undoes itself". This allows for calculating any single missing value from a set of values.

What is RAID?

Standard configurations combinations of disks in a Striped, Mirrored, with or without parity combinations are called the RAID (Redundant Array of Independent Disks) with a corresponding numeric for common communication of the configuration.

There are various combinations giving different trade-offs of protection against data loss, capacity, and speed. RAID levels 0, 1, and 5 are the most commonly found, and cover most requirements. Here is a brief summary,

  • RAID 0 (striped disks) distributes data across several disks in a way that gives improved speed and full capacity, but all data on all disks will be lost if any one disk fails.
  • RAID 1 (mirrored settings/disks) could be described as a real-time backup solution. Two (or more) disks each store exactly the same data, at the same time, and at all times. Data is not lost as long as one disk survives. Total capacity of the array is simply the capacity of one disk. At any given instant, each disk in the array is simply identical to every other disk in the array.
  • RAID 5 (striped disks) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk
  • RAID 6 (striped disks with parity) can recover from the loss of two disks.
  • RAID 10 (or 1+0) uses both striping and mirroring. "01" or "0+1" is sometimes distinguished from "10" or "1+0": a striped set of mirrored subsets and a mirrored set of striped subsets are both valid, but distinct, configurations.
    At the heart of any storage solution disks will be in a RAID configuration of some description. The rule of thumb is that the more disks the more expensive so the idea is to keep the costs down by asking where would be the best place to put files. Unchanging files perhaps best placed on RAID 0 or 1 arrays and database files on RAID 5, not everything on a RAID 10 for example.

    What is NAS?

    As you can imagine the more disks a server has the more expensive it is and it's always the way, one server is running out of room whilst one near by has plenty of space spare. Things get messy if servers try to access each others storage and I/O starts to suffer. So the cool idea is to create servers dedicated to serving files to everyone else, i.e. the File-Server. However having a server just serving out files is a waste of CPU and RAM so dedicated devices were created to do the job, these are called Storage Arrays and array's being the row of hard-disk drives themselves.
    Attaching Storage Array's to a network makes them a NAS, or Network Attach Storage.

      What is a DAS?

    A DAS or Direct Attached Array, is a Storage Array that is directly attached to server or workstation because the machine is performing a specific role which means using a lot of data, needs the performance or the protection. Typical examples of this would be high-powered graphical work stations, office media libraries or sometimes found in the home attached to Media-Centre's to store Movies and Music.
    A good example of DAS is the LaCie 4Big Quadra which can attached to a server or desktop via USB, Fiewwire or eSATA.

What is a LUN?

When you request an amount of storage a set of hard-disks are logically grouped together, this grouping is called a LUN or Logical Unit Number. Storage Array's are divided up into LUN's.

    What is a SAN?

    However as you can imagine all this extra disk accessing traffic on the network to read and write files from servers to NAS devices is going to increase latency and take up bandwidth. This is were placing storage devices onto it's own dedicated network is best practice, this storage network is called a SAN or Storage Area Network.

A SAN concentrates on speed, ease of management and amazing resilience to disaster. It is about moving large volumes of data round quickly and safely.

SAN devices are at the cutting edge of storage technology and have advanced into specialist technologies that typically mean using the fastest mediums possible such as a dedicated fibre-optic networks for example. The Storage Array's themselves can also go a step further than NAS, take the HP Storage Works 9100 for example, 640 terabytes capacity.

The Storage Works 9100 has a shed load of RAM in it acting as a cache protecting you from the limitations of the hard-disks themselves. Ultimately Hard-Disks have there limitations however their days are numbered! As the size increases on Solid State Disk or SSD, these are increasingly being used in Storage Array's such as the EMC Symmetrix DMX-4. Most EMC devices are rebranded as IBM or Dell. The trend is only likely to continue.

Good Questions to Ask

The section covers some Q&A's to help ...

So what other applications are using the same Hard-Disk on the SAN? Could this explain those unpredictable slow downs?

Certainly Yes! Let me give you a typical example. SQL Server creates a file called TempDB, this is were it stores stuff it's workings out. If accessing TempDB slows down, SQL Server will slow down, looking at Task Manager isn't going to tell you that.

A typical reason why TempDB is slowing down is because the file is sharing a Hard-Disk with some other files that are getting accessed at the same time.

It is recommended that a TempDB file is created for each processing core just make sure that they aren't all sitting on the same Hard-Disk. For optimum speed make a TempDB file sit on a Hard-Disk all on it's own. Sounds like a waste? No, not if several databases and their queries are hanging off that TempDB file specially if you can get more performance out of existing SQL Server machines you won't need to buy more.

Should the file block size change?

Some applications read data in specifically size chunks at a time. SQL Server's is 16 kilobytes for example. So it makes sense that the Hard-Disks to also reads and write with the same block size as the applications it's serving.

Can storage help with Disaster recovery?

Many storage solutions are built with DR in mind and will not only copy data to another disk in a mirror setup but copy that data to another Storage Array in another Data-Centre. As you can imagine with the complexity of this operation there is more than one way to do it. For example Symantec Veritas Volume Replicator will create an exact copy in another Data-Centre be transferring only block level changes over an IP network keeping network traffic down.

However there is a debate whether technologies like Veritas Volume Replicator are good alternative to say SQL Server Database Mirroring, as Volume Replicator sends exact block level changes where Database Mirroring will send SQL commands that can update an infinite level of data depending on it's instruction, both of which can be sent over the same IP network but only works with SQL Server 2005 and above. So you can see there is pro's & con's.

The Checklist

This section covers question that your storage team will want to know about the software don't always rely on Business Analysis, Architects or Projects Managers to supply this information, better to have it twice than not at all even if you can't fill it all in.

Questions about the Software

  • Initial Amount of Storage - how large will the system be when it goes live
  • Storage growth - 6mths-  12 mths, 18mths, 3yrs, 5yrs
  • User Growth 6mths -  12 mths, 18mths, 3yrs, 5yrs
    Storage & Capacity planning teams can work out when is the best time to buy more storage.
  • SLA - Survival Critical,Multiple Business Processes Critical, Business Process Supporting, Day to Day, Nice to Have - Business Day - 8am-6pm, 24hrs, World Region EMEA/ASIAPAC/AMERICAS
  • Most Frequent Time of Use -  End of Month? End of Day? No Peak?
  • Business Users and Location  
  • Support Team

Storage Teams are operational so will want to know who they need to talk to if things go wrong or want to organise upgrades or outages. 

    Questions about the Files
  • File Name/Set - Size, Growth, 12 mths, 18mths, 3yrs, 5yrs
  • Description of Use
  • Type -  Sequential/Random
  • Performance Required <25ms <100ms
    Answering these types of question will help determine what type of RAID the file will sit on and the DR whether a local Data-Centre or who knows, it could be on the other side of the world!

In Summary

You don't have to become an expert in Storage, I'm not. Having an appreciation certainly is an advantage as it is a key subject of enterprise computing which many can wrongly believe they are abstracted from but clearly aren't, infact the opposite, total dependent on. Getting your head around storage could be one of those things in grow that 'Developer X-Factor' that turns you from a good developer to a great developer!

Technorati tags: , ,
Posted on Tuesday, December 30, 2008 12:03 PM Development Technologies , Technical Architecture | Back to top

Comments on this post: A Developers Guide to Storage. What you need to know but were afraid to ask.

Comments are closed.
Comments have been closed on this topic.
Copyright © Dave Oliver | Powered by: