Friday, September 04, 2009
#
I posted last year regarding a dependency management tool I was contemplating. Well, I can happily announce that I did create and launch the tool as I had envisioned it. (I just actually completed a lunch and learn today where I unveiled it to a few more people within the company)
In essence, it works by spidering the BUILD environment nightly and records all dependencies from code to code, from code to database and from database to database. We have over 100 .Net solutions with multiple projects within and all the references down to the method and database object level are recorded. It was very satisfying recently to clean up 1/3 of the code from a project as it was found to be no longer referenced (around 100 out of 300 stored procedures were removed). This alone shaved about 1 month of developer effort out of a major refactoring initiative.
I can't post or easily demo the tool as I developed it to work within our company intranet, but I thought I should point out some of the libraries I settled on to make this tool possible (these differ from the ones I was considering when I first posted about this last year). If you haven't played with these but are the type that like to create innovative developer tools or efficiencies, these are a must for your arsenal:
- Spidering DLLs: Mono.Cecil
- Spidering Code: NRefactory
- Spidering Databases: SMO
Thursday, August 13, 2009
#
I have been recently dealing with a somewhat tangled set of components where over time, developers have added references as needed to get things done quickly, creating circular and upstream references, resulting in low level layers depending on high level layers. While looking for a nice little refresher online on dependency injection (incidentally, this is a great refactoring technique to fix this situation fast), I found this nice concise definition on bad design provided in an article by Robert C. Martin, Thoughtworks
This seems like a great step to facilitate design reviews in a repeatable manner while avoiding digression into personal preference or opinion. I look forward to reviewing designs on the basis of Rigidity, Fragility and Immobility. For the full article, visit http://www.objectmentor.com/resources/articles/dip.pdf
------------------------------------------- ARTICLE EXCERPT -------------------------------------------
Have you ever presented a software design, that you were especially proud of, for review by a peer? Did that peer say, in a whining derisive sneer, something like: “Why’d you do it that way?”. Certainly this has happened to me, and I have seen it happen to many other engineers too. Clearly the disagreeing engineers are not using the same criteria for defining what “bad design” is. The most common criterion that I have seen used is the TNTWIWHDI or “That’s not the way I would have done it” criterion.
But there is one set of criteria that I think all engineers will agree with. A piece of software that fulfills its requirements and yet exhibits any or all of the following three traits has a bad design.
- It is hard to change because every change affects too many other parts of the sys-
tem. (Rigidity)
- When you make a change, unexpected parts of the system break. (Fragility)
- It is hard to reuse in another application because it cannot be disentangled from
the current application. (Immobility)
Moreover, it would be difficult to demonstrate that a piece of software that exhibits none of those traits, i.e. it is flexible, robust, and reusable, and that also fulfills all its requirements, has a bad design. Thus, we can use these three traits as a way to unambiguously decide if a design is “good” or “bad”
------------------------------------------- ARTICLE EXCERPT -------------------------------------------
Thursday, July 24, 2008
#
The Challenge
Imagine a development shop that maintains 100 .Net solutions, each composed of multiple projects. Imagine roughly half of these being shared domain services or shared common libraries, the other half being end web applications servicing multiple large clients. Imagine any given project having the ability to reference classes/methods on any suitable re-usable service. Imagine trying to get a grip on it all - a great architectural paradise or a great architectural nightmare. Welcome to my world!
For a while I have been thinking about a way to manage dependencies. Dependencies hold us back, they make us scared to change things. But when we don't evolve, we don't improve, we spend time doing complete rewrites or adding layer over layer of code bloat and we don't refactor to achieve enough of my favorite thing..... (c'mon guys, you know it's coming).... RE-USE.
I am dreaming of a searchable database of dependencies, most it being automatically generated and updated, although some could be specified manually if indirect. Basically a master tool that would spider our build and production server(s) and document dependencies to any artifact - code, databases, folders, etc. If we want to remove/update/rename anything, we can be confident in our actions. It would point out orphans that we can clean up so as to keep our systems/code clean. It would plug into an automated build process and send dependency reports and alerts.
The Research
Voice inside my head says "Start with the code". I want to know for every method/property what method/property calls it in any project. Here's what has transpired so far:
First I figure a little app that calls the .Net framework could be in order, that spits the results to a database. I know System.Reflection has metadata on dlls built in. On further inspection, System.Reflection only exposes things in the manifest, which does include assembly references (a start), but does not provide an insight into method references. Next I look at the CodeDom namespace - if similar to XML DOM, I could create a solution to parse the source code and I could iterate the dom to find function calls. In fact, there is a CreateParser() method - too bad for me there's no implementation. Maybe in a future .Net version?
I took a look at NRefactory that is an API that comes with the SharpDevelop IDE but can't find much documentation on this to see if it could do the job.
Examined some out-of-the-box add-ins/tools, NDepends being a possibility. I figured I would have an automated build output an XML file from NDepends and then I could have code that parses it for the dependencies and puts them in a database.. NDepends looks like a good tool with some interesting features and a query language, but it does not output method dependencies (only type and assembly ones), although you can see them if you drill down using the application. There is also licensing per individual.
I thought dlls could maybe be reversed engineered; there is a tool that comes with .Net called ildasm.exe. Its output though is intermediate language code which I'm not familiar with and would be kindof brutal to parse. I found out Reflector has a "hidden" API if you reference the EXE. A nice MSDN article that has a promising name by James McCaffrey: http://msdn.microsoft.com/en-us/magazine/cc163641.aspx. May try this out, does not seem like the perfect high-level API but may not be too complicated knowing what exact statements to look for in intermediate language.
Then found out that FxCop also has a "hidden" API if you reference the EXE. Here's a nice doc Jason Kresowaty created to document this API: http://www.binarycoder.net/fxcop/pdf/fxcop.pdf. This looks like the abstract API I was looking for. This is my best bet right now.
The Execution
Concluding my research, my goal in the next few weeks is to:
- Use System.Reflection to log all the assembly references found in dlls
- Use FxCop or Reflector API to log all the method references found in dlls
- Log all the source code in the database (if this is impractical, use indexing service) so I can query it. What the hey, why not use SQL Server's scptxfr.exe utility and log all the stored procs/views/triggers too.
- Have a GUI for the development teams that displays dependencies between assemblies and allows you to drill down to methods. Have GUI that allows you to search for ad hoc strings in the source code..
I'll let you know how it turns out.
P.S. If you know of some wicked tool that can do this for me and allow me to RE-USE and conserve time, let me know!
Monday, June 09, 2008
#
Apparently there was a little stir among the senior developers today whether multiple returns in a function/method's implementation is an absolute evil - some taking both absolutely agree or absolutely disagree. My comment:
If it's a long function, I try to avoid multiple returns and have a single exit point. If a short function (15 lines or less), it's not really that important, as long as the code is readable. Usually the discussion of exit points hides a bigger issue that the functions haven't been decomposed enough and you have a long function that needs help in understanding - solution: break up/out the function.
With a long function, I have already added a return statement right at the start that executes based off a simple short circuit condition, and this was more readable/intuitive compared to exit points being nested deep in a function or nesting the entire implementation in an IF statement.
In general, in implementing I try to write so someone else has a quick understanding of my code and it's self documenting. If someone has to trace through my code to figure out where/when it's returning and so forth, going back and forth with different parameters that potentially could be passed in, they might as well have implemented it since they spent way too much time understanding my code. There's more that I could say regarding functional decomposition, case statements, else clauses, error handling, etc. Ideal path of execution is a broad topic that goes beyond single point of exit.
I will add that multiple exit points is bad if you 'forget' to do something common to any execution path due to leaving the function too early - ie. things like cleanup or logging. But did you know that you can wrap your entire implementation later with a Try... Finally and the Finally will execute regardless of how you exit the function (ie. via a return OR an exception thrown)? Kindof kewl...
Thursday, June 05, 2008
#
As the creative and interactive staff in my company are under my dictatorship... errrr care, I have had the opportunity to expand my understanding of what makes a great application in ways computer science can't teach you. I ran across a thought provoking article on A List Apart, that although targeted towards designers, could be generalized to any profession:
http://www.alistapart.com/articles/oncreativity
"Except for personal projects, self-expression has no place in design, but constraint is vital to design. No component fuels creativity more than constraint. Constraints are a designer’s best friend. They’re signposts, not shackles. Indeed, without constraint, creativity (and design) is irrelevant."
Tuesday, June 03, 2008
#
“What's in a name? That which we call a rose
By any other name would smell as sweet."
Smells wonderful, but what if no one plants it, no one waters it and even the gardener can't remember what the heck the flower that smelled so nice was called.
I’ve been somewhat absent in the Microsoft events and hooplah in the past. These meetings and events used to be mostly about technology previews and the next cool tool or language that will revolutionalize software development (in recent while, I’ve noticed things are changing however for the better with the community initiatives). What really interests me is platform agnostic design principles and best practices. I cringe when I attended a Microsoft event this year and saw yet again that CSS class called ‘MenuLeft’ where the presenter showed how easy it was to change the implementation to have it float right. You see, to me a name is important.
The goal
Closer to my point now… I feel a part of our responsibility in our goal of moulding new developers into star programmers/designers lies in realizing their roadblocks to learning and in making adjustments to our teachings to compensate. Do not spend time teaching them technology, they will get drawn into that as long as they have a curious nature, teach them design and good coding practices… and find the right names to make them remember.
For example, when I held some database design reviews some time back, I didn’t even bother using industry accepted words like ‘normalization’ at first. I found myself making up the term ‘row-based design’ and teaching how ‘column-based design’ had problems. Seemed liked they got it and future database additions were in good form without me being involved. New team members are since then being taught about how ‘row-based design’ is better.
A new goal of mine is to focus on teaching object oriented principles in order to have it applied correctly and to leverage the benefits in new development. Before working where I’m at, I was in the Java camp and spent some time decompiling Java class libraries in order to learn how they were implemented (Sun developers are awesome by the way in their implementation of the Java packages). Along with an experienced OO mentor for a few months some time ago and a lot of self-learning and trial and error, I have found some ways to do things that make future maintenance/refactoring much smoother.
Horror
So I talk to people in the .Net camp that know what they’re doing and find out the latest and greatest in OO talk. What do I hear about? Stuff like Liskov Substitution Principle. I wonder – what is this great thing, I never heard of it, it must be past my current knowledge base. I look up the definition: “In class hierarchies, it should be possible to treat a specialized object as if it were a base class object.”
I’m like wow… what a bad name. This is like intro to OO type of stuff and we try to teach people this way? A few weeks later, I hear that term again. I want to appear knowledgeable, so look it up. Oh yeah, I forgot, it’s polymorphism by inheritance. Gotta remember that… I have a decent amount of OO experience (not an expert, but I’ve thought about how best to do things quite a bit) and yet I need a reference or search engine at my side. This is bad, I can tell you I will need to take some time to come up with my own new vocabulary in order to spread the word. While I’m at it, I might look for an alternative for Inversion of Control (lovingly shortened by the community to IoC).
The ideal
It makes me think – what is the best protocol for naming? I want to avoid pushing theory and constantly re-inforcing/reminding people the definitions; I want them to just intuitively get it. Object oriented development is a relatively new field of study (especially in the MS world). As with most areas, I would predict an evolution, how about:
Phase 1 - Scientific Naming
Phase 2 – Layman Descriptive Naming
Phase 3 – Intent-Based Naming
Phase 1 (Scientific Naming) is the talk of academics and ‘smart’ people. We are not at the point where we can benefit from our full pool of young potential and community. We lack the naming that will get old school management on board with new ideas. Scientifically named concepts will yield slow adoption and recognition.
Phase 2 (Layman Descriptive Naming). Here we are getting somewhere, we haven’t encoded why we are doing things, but we are now making some good patterns easier to remember. This is the difference between saying you have conjunctivitis and getting a lot of blank looks from everyone, or just saying ‘pink eye’ with nodding heads all around. We need names even non-technical people can relate to – this ensures that our university students can get hopping with good principles starting from day one. I even find myself avoiding the widely accepted word ‘polymorphism’ in favor of something like ‘plug and play’. My advice: pick up the grade 6 dictionary for teaching IT, throw away the university textbook. Refer to the official name as a sidebar (there may be multiple official names), but use the layman name to really get an understanding across.
Phase 3 (Intent-Based Naming) is an idea for a utopian naming convention. Imaging the software engineering field to be mature at a point where we have a deep understanding of why we use the principles we learn. How many times have we seen a good concept mis-applied or used in the wrong context? Junior developers are passionate to try out every advanced trick in the book yet lack the experience to answer “It depends…”. What if we actually put our experience in the nomenclature? Easier said than done and I’m not sure if it is possible to come to such a place for everything, but names like ‘Seperation of Concerns’ kindof hits the mark for me.
Wednesday, May 28, 2008
#
OK, so my first non-intro post will not be on technology - I will post something relevant to this site but first I need to get this out of my system :)
If you know me, you know I have an addictive personality. One of my latest addictions is this TV show Battlestar Galactica. I was given the first 3 seasons and got hooked after the first episode. Since then, I have been developing my master theory on how this 'frakking' thing ends (this is the last season). I revealed many aspects of my theory at the start of this season to some co-workers. One co-worker of mine, we'll call him Mr. Pal (haha), upon hearing my latest recap, didn't recall that I had predicted some things months ago. So I have decided for posterity and in order to strengthen my position as suffering from the Cassandra syndrome on this and other things (see Greek mythology), that I should document my master theory. Rather than spewing one master theory that has a 100% chance of being wrong, I have decided to break it down into mini-theories where I could see a few hit the mark. Here we go guys, no sense in watching the rest of the show now, it's all here!
Mini-theory 1 - Kara Thrace is the half cylon
Yup, you heard it here first. The half cylon that is going to change the world... you suckers thought it was Athena's kid. There's no way they'd make it that straightforward (if anything else, it's the chief's baby but I'll get to that later). Kara is THE half cylon, the first born from a human and cylon - at least the current Kara, I won't commit to the old Kara due to a secondary 'ovary' mini-theory I had at some point that I won't go into here. In any event, remember when she was with her mom on her death bed in her visions and she was talking to Kara, Kara who was about to take the journey in the nebula said "How do you know I can do it?". Her mom turned to her and whispered "Because you are my daughter..." Spooky, that was the apple that fell on Newton's head. Kara will change the world, she will lead both races to Earth and she will be the harbinger of death - yet to see who's death...
Mini-theory 2 - The 12th and final cylon is... Helo
That's right folks. It's going to be a big reveal at the end, while all of you are choking on your popcorn I'm going to be posting on this blog how I knew it all along! Most people expect it won't be obvious (ie. Baltar), yet it should be a pretty defined character. Some people say to make it interesting it should be an Adama, Kara or the president. Those *could* all be possible if the last 5 cylons are special since they are copies of actual humans (remember that Kara never met the others until her fiance's funeral, and the father son had a long stint seperated). But I've got Helo pegged (otherwise I would be thinking Cally). I bet when you go back to all the previous episodes, you will notice a pattern of behavior that is suspicious. I bet the last cylon has always known he was a cylon. Let me ask you a few questions. Who protected the cylons from extinction from the virus? Cylon trait (or used to be): they protect their own. Who gave up his seat on the raptor (first episode) for... Baltar? Cylon trait: they love Baltar. Who protected Athena from Kara and then from the crew, even though she was cylon? Cylon trait: they protect their own and seem to sleep around with other cylons. Now his last performance in letting Kara go on her mission while Gaeta was losing his leg is part of some evil cylon plot. Yeah, the one that looks to be the most "good" is usually the hidden villain. Which means that his child with Athena is actually a full cylon, so even if mini-theory 1 fails, the first born half cylon is NOT Athena/Helo's baby, it's the chief's baby.
Mini-theory 3 - Last episode - big battle and endless time loop
Fuzziness smears this mini-theory, but I'm connecting some dots. This is how the last episode will go. They will find earth - both cylon and human together. I figure Kara will find it with the humans and the cylons will have followed them with the help of the final five. What will follow is a big battle, we're going to see alot of bloodshed, most of the ships if not all but a raptor will be destroyed - after all, Kara is the harbinger of death. Earth might get screwed over too altogether. Bear with me, I'm stretching a bit here.... Anyways, remember that incision that Kara had at the hospital? Well, what they didn't show you was that Kara has been cloned by cylons - hence why she is half cylon, her clone will look exactly like her since it has the exact same human DNA. Confused yet? Well her clone will then be sent through that nebula wormhole back in time and her mom which is alive in the past will adopt Kara as her own daughter. Aside: wormholes are linked to time travel (look it up on wikipedia) and when Kara "blew up" going in the nebula, what you really saw was the reaction as matter/energy of the ship was going back in time. Her ship was brand new when it came back since what you were looking at was that same raptor 20 odd years ago. But the inside of the ship for very sound scientific reasons was insulated from the time travel. So anyway... now we see why "This has happened before and will happen again". Now we see why Kara's "mother" never really gets attached to her - she's not her biological mother. Who's Kara's mother? Herself?... In the words of Saul on finding out he's a cylon, "woah" Meanwhile the show will fade out with little Kara drawing her nebula pictures and me being credited as a psychic genius
Mini-theory 4 - the visions are controlled by cylons
Those darn cylons and their mental suggestion powers. Seems so convenient that Baltar gets visions of Six and these hallucinations give him abilities to see things and secrets that he would likely not be able to perceive otherwise, but that someone with inside knowledge would know. Seems also interesting that these control him like a puppet and make him take action he would not otherwise take. Something bigger is at work here - the cylons, maybe the 12th cylon only, have abilities to implant suggestions and hallucinations on weak human and cylons alike. Since some of these occurred away from the Battlestar crew, it makes me think they are not implanted by one individual or cylon, but maybe a specific model. Six also gets hallucinations, so I would think it's not her model.
Mini-theory 5 - 13th cylon - Baltar??
OK, now this is me really trying to make things happen. I'm not very confident on this one, but now that I have your attention why not continue to assert the impossible? We have 12 cylons... or do we? Is it me or did I notice that the number 12 matched the number of colonies. But then we have that 13th colony called Earth, it makes me suspect that there is a 13th master cylon called Baltar. Another 12 is the 12 signs of the zodiac (Greek/Roman), with the 13th being the Sun but I haven't taken time to explore that further and I digress... Baltar is so cylon, they make him painstakenly have the profile of one. Cylons don't give up their own as we know - even Athena refused to reveal the other cylons and she was loyal to the fleet apparently, Baltar knew Sharon was one and had a test to detect them yet kept quiet - why? Another thing about those cylons, they really like to sleep with other cylons. Chief and Sharon, Tory flirting with the chief, Tory had a little fling with Sam most people forgot about, Sharon and Helo (my pick at 12th cylon), Sam and Kara (my pick as half cylon). If Baltar was a cylon that would solidify that - he's slept with Tory, Kara, Six, D'Anna. A family that lays together stays together I guess, haha. Not to mention all the things he's done in previous episodes to mark him as a cylon. Deciding to settle on that planet like a chump which of course the cylons found so conveniently with his girlfriend's help.
So there you have it my fellow Galacticans, the editorials of one with true psychic abilities. True psychics cannot be discredited until the future comes about and show they are full of BS, but until then... :) I will have this record for Mr. Pal and others to make me the Cassandra of the 21st century. Here's to an awesome Season 4 and no more US May Long holidays that delay the next episode
Yes, that's right; I have a blog now! Reminiscent of a few years ago when people looked at me weird when I told them I didn't have a cellphone (of course now I'm hooked to the crackberry), I have 'conformed'. The fact that I waited this long may give you a glimpse into my personality (you thought independent thinker right?).
I've considered it before and usually just gave up on the thought because I want to post quality material and am limited on the time needed to do so, but then again, quickly formulated thoughts and ramblings is the essence of a blog - isn't it? Now after several people have suggested that I put my thoughts on the web, I have decided to see if I can post some content that might interest some people.
This is not a 'promise to post everyday to my blog' message, but just a promise to post once in a while and document some of my strange ideas... and maybe some of the well-thought out stuff too. In any event, welcome to my blog.
Woohoo - first post!