During the last days and weeks, there's an increasingly heated debate about the performance of NHibernate vs. some other commercial and noncommercial ORMs. This debate was triggered by the launch of a new website called ORMBattle.NET, which is allegedly "devoted to direct ORM comparison" (and hosted by a commercial competitor of NH...). The comparisons are largely based on batch processing tests like this one:
protected override void InsertTest(int count)
{
using (var transaction = session.BeginTransaction()) {
for (int i = 0; i < count; i++) {
var s = new Simplest(i, i);
session.Save(s);
}
transaction.Commit();
}
} |
I first came across this topic because I'm following Ayende's blog, and initially the whole shebang was nothing but white noise for me, simply too unimportant to spend time with it...
But as time goes on, I had to notice that some people, who I suppose to be really clever, spend considerable efforts on this topic - largely to 'prove' that the 'comparison tests' are technically flawed or generally senseless, because they miss the whole point of an ORM. You can follow this debate for example here, here, here, here, here, or in the various NHibernate forums. This post makes the point that there are other things to an ORM than batch performance, like for example community support and the like.
I wholeheartedly agree to all the arguments made in the aforementioned posts. Additonally, I' feel that one should consider the following points; they seem much more relevant to me than any performance comparisons, let them be 'fair' or not:
- The ORM is in almost every cases not the slowest component of an application's 'persistence complex', but it is the database system and/or related connection issues (e.g. low bandwith, slow server machine etc.).
- If you are using an ORM in your project, then you most likely have a complex domain model, which has to be mapped somehow to a database (If that's not the case, then probably the use of an ORM is generally a bad idea...). To my experience, performance isn't an issue at all in such projects (supposed that you don't do real silly things in your code). Rather you have to focus on such things as functional correctness, robustness, security and the like. It is simply not relevant if 'retrieving a Customer' takes 10 or 200 milliseconds.
- If, in the project planning phase, you decide to use a specific ORM, then really important questions may be:
- Does the ORM framework provide enough extensibility points to meet potential future requirements?
- Does the development team have sufficient know-how for the product?
- Is there a rich, consistent, and well documented API?
- How's the technical support?
- How is the availibility of external experts (i.e. freelance developers who can support the original dev team, if required)?
- Has the product in question in general a good standing in the industry?
- What are the expectations for the future of this product (remember: business software usually may live up to 30 years...)?
- ...
These points don't really play a role in the discussion, at least as far as I followed it.
What remains is a slight feeling of irreality. This picture (shamelessly stolen from Ayende's last post) perfectly summarizes my thougths:
So, what does that mean (at least to me)? Well, essentially two things:
- Saying "Currently (i.e. after all the fixes) NHibernate is 8-10 times slower than EF, Lightspeed and our own product on this test." is like saying There are better hammers if you have thousands of screws.
- Demonstrating that NHs performance in these 'tests' can be radically improved is equivalent to saying But you can do much better if you use our hammer _that_ way.
As I said: I don't get it...