Ulterior Motive Lounge

All right, Mr. DeMille, I'm ready for my close-up

Martin L. Shoemaker — Tue, 26 May 2009 23:58:43 GMT

Back in January, Brian H. Prince from Microsoft interviewed me about the UML features in Visual Studio Team System 2010. Today, he informed me that the interview is finally live on Channel 9.

Ulterior Motive Lounge: A Map of Carnivore Park

Martin L. Shoemaker — Sat, 10 Jan 2009 07:35:07 GMT

As I’ve written elsewhere, I love maps. And I love a story that includes maps. Maps help me to see where the action takes place; and they give me tantalizing clues to action yet ahead. When I first opened a hardcover of The Hobbit and unfolded the big map of Wilderland, I was hooked.

So for those who love maps like I do, here’s a map of Carnivore Park. (Click picture for a larger image.) This is where the rest of the story takes place; and following it is a map guide that may contain clues to what lies ahead.

Carnivore Park lies within a dormant volcanic island, bought and run by Owner. The volcano cone makes a natural barrier to dinosaurs – and to prying eyes, of course. The west end faces open ocean. The east end faces Central America, though a long way away. The east end beach has been artificially built up, along with two breakwaters. A concrete apron allows cargo and passengers to disembark and take the switchback up the slope to the entrance to The Park.

The interior of The Island may seem natural, but it’s actually nearly 100% sculpted or planted. The topsoil has been imported from rich fertile swamps on the mainland. The river has some contribution from springs and runoff, but is heavily fed by a hidden desalination plant. The ridge through the Park is actually a concrete wall, with interior chambers and bolt holes. The plateaus of the visitor center are partly natural, but largely shaped concrete. Roads run along the ridges, sometimes fenced from the dinosaurs, sometimes separated by “natural” barriers, and sometimes exposed.

The Visitor Port

Ship and helicopter landings.

Carnivore Park

The main park buildings reside on a set of artificially enhanced plateaus. Trees and other obstacles are carefully arranged to block any view of dinosaurs from the Pavilion or the rest of the control area; but the Lodge is elevated so that the higher (and more expensive) storeys have a view into the main Park.

The Pavilion

This is the entry to the Park. It includes the Operations Center, the Gift Shop, the Quick Bite Café, the Boring Exposition, and the Laboratories.

Gift Shop

Souvenir shop. Sells Park merchandise. Also sells LoungeWare.

Quick Bite Café

A cafeteria.Specializes in curried “dinosaur” dishes (actually chicken).

The Boring Exposition

Auditorium named for Wayne H. Boring. Used for reviews and presentations.

Operations Center

The nerve center for the Park.

The Laboratories

A sign on the door reads “Here there be dragons!”

The Lodge

Guest lodging at the Park. Across from The Pavilion.

Drumbeats

Nightclub at the top of The Lodge.

The Food Chain

Restaurant in The Lodge.

The Pool

A place to unwind when not out seeing the dinosaurs.

On The Rocks

Bar near The Pool.

The Petting Zoo

An area near the Pool where small, cute dinosaurs are available for petting and feeding.

The Park

A giant, dormant volcano cone, perfect for confining dinosaurs.

The Garden

Where the herbivores run “free”. Visitors can approach by ascending walking trails.

Carnivore Country

An area where various smaller carnivores roam.

Rex’s Crib

An area set aside for Tyrannosaurs Rex specimens, including extra reinforced fencing and other safeguards.

Raptor Country

An area dedicated to Velociraptors and their prey.

The Backlots

Fenced areas deep within the public viewing areas, well-screened by trees. These are areas where workers prepare and care for dinosaurs (and dinosaur meals) before release into the viewing areas. Each has a number of isolation pens, carefully screened by trees from other pens.

River Country

A boat tour (not yet approved by Money Man) through swamp and river dinosaur habitats. Also passes through The Glasshouse.

The Glasshouse

A large glass enclosure for flying dinos.

The Power Center

Includes a geothermal power station, water pumps, and fence control.

Not that I'm ungrateful, but I think you're in the wrong place...

Martin L. Shoemaker — Sat, 13 Dec 2008 12:42:01 GMT

In an effort to understand and better serve my readers, I keep an eye on my referral logs. I like to know who likes what they read here, and then learn what else they read.

But this one has me boggled. One of my most recent referrals is from the headlines page for fanpop's community for The L-Word. It's a link to my recent post for Ship It On The Side.

I appreciate the link. Really, I do. But this site has pretty much nothing to do with the lesbian scene in L.A. Sorry.

Ship It On The Side Episode 3

Martin L. Shoemaker — Thu, 11 Dec 2008 07:45:15 GMT

Ship It On The Side Episode 3 -- Use Cases is now released. Complete with goats!

The UML Learning Path

Martin L. Shoemaker — Thu, 11 Dec 2008 02:41:34 GMT

(Click picture for a larger image.)

No, I'm not going to name any of the devs who inspired this post. They wouldn't know who I am, anyway.

But it takes an extremely high degree of arrogance to go from "I don't see a way to use this" to "This has no value, no matter who says they're getting value out of it. So I'll dismiss it, and I'll mock them" Either arrogance, or more likely, insecurity: "I don't understand this; so since those people think it's important, either they understand something I don't, or they're fools. I'll mock them, so everyone thinks they're fools. That will make me look smart."

And that insecurity manifests in a lot of places on a lot of topics, not just UML: Agile Development, Orchestrated Development, CMMI, Test Driven Development, C#, Java, Ruby, linux, .NET... Any time you move from "I don't see it" to "It's worthless", look around: if other developers are putting those tools to productive use, then it's not worthless. It just doesn't help you. So do you call it worthless, and imply they're fools? Or do you openly mock them, demonstrating that you're a fool?

Or do you follow the only exit path in this diagram? There is only one, after all. Once you get UML, you've gotten it for good. You may not use it all the time, but you'll understand when and why you should use it. But the only exit path is the middle: you recognize that UML (or Agile, or Orchestrated, or...) is having some value on some projects, so it's not worthless; but you just can't see the value. You remain open-minded.

UML Jokes

Martin L. Shoemaker — Wed, 10 Dec 2008 06:09:15 GMT

In case the Lounge isn't satisying your hunger for UML humor, check out UML Jokes, a contest for UML jokes. Sadly, it expired in February, I didn't start the Lounge until September, and I just found the site now. So congratulations to the winners! And if there's another round, look for a Lounge submission.

Ship It On The Side

Martin L. Shoemaker — Sat, 29 Nov 2008 21:59:02 GMT

Curtis Gray informs me that the first Ship It On The Side podcast is now published.Listen to us talk about building and shipping great software while holding down day jobs.

Thanks, comment spammers!

Martin L. Shoemaker — Thu, 27 Nov 2008 04:46:56 GMT

Comment moderation is now activated. Thanks, comment spammers!

Doctors are the Stupidest Users!

Martin L. Shoemaker — Fri, 21 Nov 2008 04:33:43 GMT

Note to doctors: No, I don't really mean that title. I use provocative titles to get attention and capture an attitude. What? What are you doing? You're going to stick that thermometer where?

Doctors are the stupidest users. If you've ever had to write software for doctors, you've discovered this: the phrase "RTFM" was made for doctors. They just can't be bothered to read even the simplest help docs. They can't bother to learn even simple tools that a bright grade-schooler can master.

OK, that's the programmer perspective. Now let's look at it from the doctor's perspective. On her desk is a small mountain of medical journals she needs to read to keep up with her specialty.

Next to those is a small mountain of textbooks for the new specialty she's trying to learn.

Next to those is a small mountain of new regulations and guidelines she must comply with to maintain her license.

Next to those is a small mountain of insurance guidelines she'll probably never have time to read but should if she wants to make sure she's charging within guidelines.

Next to those is a small mountain of insurance paperwork that demands her scarce attention. She has no time for it; but if she doesn't keep up with it, patients may be denied treatment that would've been approved if she had.

Next to those is a small mountain of accounting statements that she'd rather not bother with, but has to if she wants to pay her student loans and malpractice insurance.

And next to all of those is a large mountain of patient histories, test reports, specialist reports, and hospital reports she needs to keep straight in order to treat her patients.

And somewhere, buried under all of those papers, are a few pictures of the family she hopes to see once or twice this week.

And along comes this programmer person who says, "What's the deal? This is easy! Just read this, and this, and then try this, and learn this, and look at this, and it's easy! You're a doctor. You're supposed to be smart. This should be easy!"

After she schedules the programmer for an emergency barium enema, the doctor goes back to her work.

This is part of the reason (by no means the entire reason) that there's a profession of Medical Technician. These people's responsibility, in part, is to be the doctor's user interface to programs that the doctor just has no time to master. They have time to specialize in arcane software. They usually have more technical experience.

But really: should it be this way? Not that I want to put Medical Technicians out of work; but shouldn't the programmers spend more effort understanding the doctors and their needs, rather than requiring the doctors to understand programs and programmers?

For years, I've spoken and taught about a principle I call Conservation of Complexity. For any given task, there's a minimum complexity required for that class. You can't make it less complex. (But trust me, some idiot can always make it more complex...) If I'm automating that complexity, I take some of that work onto myself, and leave some for the user; but no matter what I do, I can't reduce the complexity below the minimum. And being lazy by nature, I'll want to do the minimum necessary to meet the requirements: minimum complexity for me, and let the user do the work.

But we can cheat! We can't reduce the complexity of one instance of the task; but we can reduce the complexity of multiple instances, especially when those instances are performed by multiple users. In so doing, we can reduce the net complexity in the system.

Let's say the user and I split the complexity, C: I get 0.5 C, and he gets 0.5 C. Now if 100 users do the task 100 times, we have:

Cnet = 0.5 C + 100 * 100 * 0.5 C = 5000.5 C

Cave = Cnet / n = 5000.5 C / 10000 = 0.50005 C

So we've cut the complexity in half. That's great, right? That's why we write code, right?

But suppose some idiot -- i.e., me -- didn't make that app very usable for the user. Oh, it's easier than working by hand, or I won't sell any copies; but say it's 0.8 C. In that case:

Cave = 0.80005 C

So I took an easy way out. The system is still less complex on the whole. We're still winning, right?

But now, let's go the opposite way. Let's say I put in the extra effort to reduce the user's work to 0.2 C. And let's assume this takes me a simply ghastly effort, 2 C. That means that:

Cave = 0.2002 C

By me working twice as hard, the system as a whole works roughly 80% less.

Now these numbers are just examples, of course. But I think they make my point: if all I worry about is getting code out the door, I may not work hard; but my users have to work a lot harder. Too often, we programmers see our perspective too clearly, and the other guy's perspective too dimly. (In other words, we're human.) But the smarter thing to do is to put forth the extra effort to reduce the user's complexity. Because after all, there's money to be made in reducing your user's workload!

I'm reading Alan Cooper's The Inmates are Running the Asylum; and with practically every page, I'm reminded of Conservation of Complexity. And of the doctors.

Also see Joel Spolsky's discussion on mental models. He approaches the same idea: the programmer has a mental model of how the code works; the user has a mental model of how the code works; and great software results when the programmer's mental model is very close to the user's. It's easier for a team of motivated programmers to change their handful of mental models than to change all those users.

Ulterior Motive Lounge Intermission 3: Plan for the Future, Learn from the Past

Martin L. Shoemaker — Wed, 19 Nov 2008 19:55:42 GMT

UPDATE: My word, what a lot of typos! I blame the bright sunlight and the glare it created on my Tablet PC. And I wasn't gonna move, because bright sunlight in Michigan is a rare treat right now. Fixed!

In our last Episode, The UML Guy and Geek Girl explored how State Diagrams help you to model the rich range of flow in a user interface. Today, The UML Guy has an announcement. (Click picture for a larger image.)

To this point, the Lounge has been somewhat random and piecemeal. (In other shocking news, rain has been somewhat wet!) I approached UML topics as I found them important or interesting, or when a particular film inspired them. Sometimes I approached them because of specific questions from readers.

One thing common to many good UML books is that they want to show you that UML is a language, not a process. I approve of that, in theory. But as an instructor, when I tried to structure my UML classes that way, students struggled with this approach. Without a process, they didn't know why they would draw a particular diagram, or when. Eventually, I decided they needed to learn UML in the context of a "just enough" process. From that decision and a lot of thought, I ended up with my minimalist Five Step UML process: Define, Refine, Assign, Design, Repeat.

Define your customer's goals and requirements in Use Case Diagrams. Add Class Diagrams as a visual glossary.
Refine those requirements into processes and rules using Activity Diagrams for the Use Cases. Use State Diagrams, Sequence Diagrams, or Communication Diagrams if they help; but primarily rely on Activity Diagrams.
Assign the steps in those rules to parts of the system by adding Swimlanes to the Activity Diagrams.
Design the relations of those parts through Class Diagrams. You may also need Object Diagrams, Component Diagrams, or Deployment Diagrams. We haven't seen those in the Lounge yet; but they're enough like Class Diagrams that you should be able to read them if you can read Class Diagrams.
Repeat the process across Use Cases. Then repeat at different scales: with Classes, or across Components or Processors. Look at more detailed Use Cases. Repeat, repeat, repeat.

This has proven much more successful as a teaching tool, and also as a light-weight process. Even an Agile process. It's just enough UML to help you ask and answer questions. You'll learn more as you apply it and grow with it.

But even just showing students a UML-based process in the abstract doesn't do much. They need to see UML in action. What they need is a Case Study.

And very early on, I knew that some day, the Lounge would need Case Studies. The Back to the 80s series (here and here and here and here) is something of a mini-Case Study; but for Volume II, I wanted something meaty: a single film that would serve as a complete project, with a wide range of development disciplines and tasks. After discussion with Editor Bill and feedback from Curtis Gray, we think we've found the ideal film. It's not quite an 80s film; but let's face it, it's not like I've ever enforced that rule in the past. And the potential in this film is just too good to pass up!

All right, all right, I'll be honest... I get to draw freakin' dinosaurs and UML! There was no way I could pass this up once I thought about it.

So join The UML Guy, Geek Girl, The Reader, and a bunch of new cast members and old friends as they head off to the tropics. But fair warning: assuming you know how this story will play out could be hazardous to your health. I don't control this strip any more. The characters have minds of their own; and so do the dinosars...

Quantity IS Quality

Martin L. Shoemaker — Sun, 16 Nov 2008 00:20:52 GMT

On a mailing list where I hang out, a participant recently said (paraphrased): “He believes that popularity proves quality. I believe that there is almost no correlation between quality and popularity.”

We hear this sort of thing all the time. There’s an implication among self-appointed elites that “the masses” — i.e., everyone who’s not them — just can’t recognize quality. It’s assumed that “popular” is proof that something is bad. You see this attitude in film snobs who insist that an Oscar nomination for The Return of the King is some sort of travesty, because the film is a popular fantasy and not some art house flick or some historical epic. To be fair: you see it in Lord of the Rings fans who for years have been telling others who didn’t like the books that they just didn’t appreciate great literature. And before the films, you could see it among the literati who snubbed Lord of the Rings because it’s a popular fantasy rather than a dreary, post-modern, self-referential, obscurantist yawn. You see it in opera buffs who assume the rest of us are subintelligent because we don’t share their passion for opera. You see it in young rebels who look down on the lives of the conformist “sheeple” and who demonstrate that they are individuals and not “sheeple” — by all dressing and talking and acting and piercing alike. And you even see it in gourmets who extol the virtues of French food over more pedestrian fare like food from McDonald’s.

But the truth is: they’re wrong, every single one of them. They proceed from two clearly false assumptions: that there is one clear, objective, inarguable standard of quality; and that of all human beings, they somehow have been born with/been granted/achieved the unique ability to pronounce what the standard is.

But the fact is just the opposite. If I can avoid butchering the Latin too poorly, de gustibus non disputandum: with taste there can be no dispute. Or in the modern vernacular: there’s no accounting for taste. When someone tries to tell you that his tastes are objectively correct, he’s demonstrating how self-centered he is or how shallow his thinking is.

Does that mean there are no things that are objectively better than some other things? Can’t we all agree that Shakespeare is better than “The Simpsons”? Nope: I could gather up quite a debate on both sides of that issue; and the pro-Simpsons side would be every bit as educated and erudite as the pro-Shakespeare side.

Can’t we all agree that French food is better than McDonald’s? No, for multiple reasons: many people dislike new tastes, and prefer comfort and familiarity; not everyone likes the spices in French food; and if you grew up with French food every day, you might see it as “normal” and McDonald’s as a new experience, where novelty makes it attractive.

And so on, and so on, and so on. If you take any “objective” measure of general quality (as opposed to quality for a particular purpose, which may be assessed much more precisely) and examine it all the way down to its roots, you find personal tastes, past experiences, biases, and other responses that aren’t objective at all. There’s no objective measure of quality.

Except one. See, like many things that are immeasurable in the small, quality is measurable in the large, through statistics. No one person can absolutely proclaim that a certain thing is a quality product; but we can measure with reasonable precision how many people accept and endorse the quality of a product, by virtue of their purchases. In other words, the list writer I paraphrased has it exactly wrong: the closest thing we have to an objective measure of quality is popularity. If a significant number of people enjoy a product, then the odds that you will like it are higher. We’re all individuals, not ruled by statistics; but statistics are a useful piece of information to help you find products to try. Quantity purchased is a valid measure of quality. The market identifies products that the largest number of people accept as quality products.

And before anyone chimes in about betamax, QWERTY keyboards, CDs vs. albums, Microsoft, or any other oft-cited “evidence” that the market can produce the “wrong” answer: go reread my post, because you still missed the point. Don’t force me to go haul out the evidence that shows the conventional wisdom is wrong in every one of these cases, or I’ll produce so much it crashes the server.

Concern vs. Worry vs. Obliviousness

Martin L. Shoemaker — Sat, 15 Nov 2008 23:56:39 GMT

Concern	Worry	Obliviousness
"Has this happened yet?"	"Oh, no, what if this happens?"	"This could never happen."
"How likely is this to happen?"	"Oh, no, what if this happens?"	"This could never happen."
"How can we tell if this happens?"	"Oh, no, what if this happens?"	"See no evil, hear no evil, speak no evil..."
"Can we prevent this from happening?"	"Oh, no, what if this happens?"	"This could never happen."
"What will be the impact if this happens?"	"Oh, no, what if this happens?"	"No problem..."
"Can we prepare for that impact?"	"Oh, no, what if this happens?"	"Cross that bridge when we come to it."
"Can we manage that impact if it happens?"	"Oh, no, what if this happens?"	"Cross that bridge when we come to it."
"Who will be responsible for watching and managing this?"	"Oh, no, who will we blame if this happens?"	"I didn't do it!"
"We'd better do something about this!"	"Somebody'd better do something about this!"	"This could never happen."
"There's nothing to do about this right now, so let's focus on what we can do."	"Oh, no, what if this happens?"	"Looks like smooooooth sailing ahead."

Which column describes your project management techniques?

Disclaimer: This has nothing to do with any of my current projects or clients. The person who inspired this already knows who he is, and is firmly in column 1. Any other resemblance to actual persons or events is purely coincidental.

No llamas were injured in the making of this post.

The Department of Motor Vehicles and Clipboards

Martin L. Shoemaker — Sat, 15 Nov 2008 23:55:03 GMT

All right, I've had enough. It's time for me to issue a decree:

Henceforth, programmers who wish to use the clipboard must be licensed in the same fashion as automobile drivers are licensed by the Department of Motor Vehicles. (Yes, yes, we call it the Secretary of State's Office here in Michigan; but more people know it as the DMV.)

1. Before you may use the clipboard to copy code, you must take 20 hours of classroom instruction on how not to use your clipboard.

2. Before you may be fully licensed to use your clipboard, you must undergo a six-month probationary period during which you may only use the clipboard with a grown up, licensed programmer observing your use of the clipboard.

3. Upon completing your probationary period, you may apply for a clipboard license at your nearest Department of Responsible Programmers. There your code will be examined for any signs of clipboard abuse. (Note: Unlike the tests at the DMV, this examination will not be easy. It will be brutal. You have been warned.)

4. If your code is deemed acceptable by a Responsible Programmer, you will receive a clipboard license, and be authorized to copy code as needed.

5. At any time, your code may be inspected for clipboard abuse. You will receive one (1) point for each line of unjustified code copying. If you accumulate 100 or more points, you will have to return to Clipboard School, or lose your license.

6. If at any time you accumulate 200 points, your clipboard license will be immediately revoked. You will be eligible to reapply after 1 year.

7. If at any time you accumulate 500 points, you will be shot. While we know this is harsh, it's the only economical option. We can't afford the cost of maintaining your code.

8. Clipboarding while drunk is grounds for immediate revocation of your license.

Yes, this means you!

Dr. Strained Memory or: How I Learned to Stop Worrying and Love Compact Framework Memory Management

Martin L. Shoemaker — Sat, 15 Nov 2008 23:53:26 GMT

OK... So why has my DODO been incalculable? Well, because we've been chasing down memory problems in our .NET Compact Framework 2.0 app that runs under Windows CE 5.0.

And if that just sent a chill down your spine, then you've probably been here before. You have my sympathies.

If you haven't been here before but you're planning a CF/CE app: Be afraid. Be very afraid.

The following is a brief summary of what we've learned (the hard way) about memory management in a Compact Framework application.

"In the beginning, the Universe was created. This made a lot of people angry, and has been widely regarded as a bad idea."

My client's application comes in two versions: one that runs on a laptop, and one that runs on a handheld computer running Windows CE 5. Since both versions run .NET -- .NET Framework 2.0 on the laptop, .NET Compact Framework 2.0 on the handheld -- they figured to share a lot of code between the two platforms, including a homebrew implementation of a Model-View-Presenter architecture. The implementation of View (and Presenter, to a lesser extent) had to change across platforms; but for the most part, what was a good architecture for the laptop was seen as a good architecture for the handheld.

First mistake, there...

And although I wasn't here when that mistake was made, I probably would've made the same mistake myself. I didn't realize that on the handheld platform, you need different coding priorities.

"Do you know what the secret of life is? This. One thing. Just one thing. You stick to that and the rest don't mean ****."

When you code, the quality of the result depends significantly on your priorities: what do you want from the code? Every developer has an idea of the "correct" coding priorities. Some value small code, or fast code, or readable code, or clever code. Steve McConnell once advised deciding and announcing your priorities for a given project before work begins, so everyone knows what's important for this project. You might make a prioritized list, such as:

Correctness.
Maintainability/readability.
Speed.
Extensibility.
Efficient use of memory.
Schedule.
Modularity.
Reusability.

And so on. Just let the team know what's expected from them, and you're more likely to get it from them.

But for many teams, that list is too much bother. They can get by with a simple mantra:

Make it run.

Make it right.

Make it fast.

Make it run, so users can give you feedback. Make it right from their feedback. And then optimize only when necessary to make the users happy, because optimization can be difficult to read and maintain and is prone to bugs.

Well, we've learned a new mantra for the handheld:

Make it small.

Make it run.

Make it right.

Make it fast.

Repeat, repeat, repeat...

You have to make it small first; because on CE 5, if it's not small, it will never run right, and it will be fast only in crashing. And then, when you take steps to make it run and right and fast, you'll inevitably make it larger, and risk crashing due to that.

So on CE 5, you have to design for small first. Adding small afterward is a great way to push your DODO to Infinity. I should know.

"A Man's Gotta Know His Limitations."

So here's the short summary of what we've learned about our limits. First, keep in mind that this code comes in two general types: Native, and Managed. Native Code is code compiled directly for the platform, and responsible for managing and allocating its own memory and other resources. Managed Code is .NET Compact Framework code, which runs inside the Common Language Runtime, a Native Code "shell" that loads and runs and manages Managed Code. It takes care of allocating memory; and if -- if -- you let it, it does its darnedest to clean up that memory when you're done with it. And Managed Code may very well load and call Native Code for certain purposes. (Native Code can load and run Managed Code as well; but for this discussion, that's irrelevant.)

Some of what follows applies only to Native Code. Some applies only to Managed Code. And some applies to both.

"I want to be an explorer, like the Great Magellan." "Oh, you're too late! There's nothing left to explore!"

CE 5 uses Virtual Memory and memory mapping to give each running process a "private" memory space. I'm no memory management guru; but as best I understand it, this means that each process thinks it "owns" the machine. The Operating System (CE 5) maps into part of that memory space. The app and all its data map into another part. On handheld devices, memory is usually pretty limited (128 MB on board, in our case, and 1 GB on card), and probably less than the addressable space in a process's memory map.

The OS itself loads into a given memory location. I can't say where, because I'm not a memory guru. I'm trying to give an overview here, not details. But the important part is that your process then receives an address space in which your process lives in the space from 0 GB to 1 GB, and "global" storage lives in 1 GB to 2 GB. The OS lives beyond that space.

"Game over, man! Game over!"

OK, so your process's memory map lets it address 1 GB (between addresses 0 GB and 1 GB); but here's the kicker, and here's where our troubles began. The actual memory accessible to your app -- also known as the Virtual Memory, i.e., the chunk of memory mapped into that 1 GB space -- is restricted to 32 MB on Windows CE. Period. No matter what you do. If your app needs more than 32 MB to store your currently loaded code and data, that's it. You're gonna crash. In our case, we didn't crash; but we made calls to SQL Mobile in a low-memory condition, and it'll crash like clockwork, every time.

So that's simple, right? Stay under 32 MB, right? What kind of freakin' handheld app needs 32 MB, anyway?

Well, one that was ported from the laptop without sufficient consideration for Make it small, make it run, make it right, make it fast, that's what kind.

And oh, if only it were as simple as waving a magic wand and saying "Keep it under 32 MB"...

"Take your stinking paws off me, you damned dirty ape!"

Maybe, maybe you can find a way to do what you need within 32 MB. (Probably not: when you tell customers that they're getting portability as a tradeoff for power and features, they never hear the tradeoff part. They'll want your handheld app to be portable and as powerful as your laptop app and as fast. Learn to tell them "no" as early as possible.)

But you're not the only one who gets to use that 32 MB. No, in fact, every app in the system has its stinking paws on your 32 MB, including the OS!

See, when an app is actively running on the handheld, its Virtual Memory is mapped into a special reserved 32 MB slot in the system, slot 0. Your app itself is loaded at the bottom of slot 0, growing up. Any unmanaged DLLs it loads are loaded at the top of slot 0, growing down as each is loaded. (Each DLL is allocated space in 64K chunks, so DLLs that are close to 64K in size -- or 128K, or 192K, or... -- are loaded most efficiently.) And the space in between is available for your app's data. You can also store data in the "global" area from 1 GB to 2 GB; but that's a lot more difficult to use, so you don't want to if you don't have to.

But here's the tricky part: because handhelds have limited memory, CE 5 optimizes the memory used by DLLs. If a DLL is loaded in one app, CE doesn't load it into another app that also uses it. CE loads it once, period. But in order to do this, CE loads the DLL into the same location in every app that uses it -- and effectively, every app that doesn't use it. The unmanaged DLLs are loaded into the top of slot 0 itself, and just left there as different apps are mapped into slot 0 and back out again. So your app takes the hit for every unmanaged DLL loaded anywhere in the device. That can even include OS DLLs: if the OS runs out of room for its DLLs in its reserved memory, it loads them into slot 0. (In our case, after the vendor did some tuning, that only meant one unmanaged DLL loaded into slot 0 by the OS itself.)

"Invention, my dear friends, is 93% perspiration, 6% electricity, 4% evaporation, and 2% butterscotch ripple." "That's 105 percent."

Now if the 32 MB limit sounds scary, and the external DLLs chewing up your 32 MB sounds even scarier... Well, you're right, they are scary.

But before you set out to try to work around them, ask yourself one thing: are you using .NET Compact Framework 2.0 (or later)? Because if you are, then the CF team has already given you an invention where they put in the perspiration, electricity, evaporation, and butterscotch ripple. Sadly, it's not a 105% solution (more like an 80% solution in our case); but it's very likely better than you have time and resources to devise and implement and test and maintain yourself. So before you start jumping through hoops, let's look at the hoops they've already jumped through for you.

When you build a .NET app (Compact Framework or regular Framework), the real application is the CLR itself. It's the thing that runs, not your code; and then it loads your code, and uses your code to tell it what to do.

So what this means on the Compact Framework is that the CLR can be smart: it knows about the 32 MB limit, so it works overtime to try to get around it. The biggest technique it uses is that all of your Managed DLLs and your managed application are loaded into the global storage area between 1 GB and 2 GB. Then, as you call the code in your app and your DLLs, the CLR loads the pieces you use into an area within your 32 MB VM. This JIT (Just In Time) Heap starts at about 1 MB, and grows as needed; but if the CLR sees that memory is low, it "pitches" code out of the JIT Heap, keeping only the more recently used code. So the JIT Heap can be treated as a 1 MB block under normal circumstances.

And another way in which the CF CLR is smart: if any one object you allocate is larger than 2 MB (or it might be 1 MB, I'll need to check), it automatically allocates from the global storage, without making you jump through all the hoops involved. The CLR knows how to jump through the hoops. That's its job.

If you think you can write better memory management than the CLR provides, be my guest. Be sure to allow lots of time in your schedule for design, redesign, implementation, testing, fixing, retesting, refixing, reretesting, reredesign, nervous breakdowns... Now CLR memory management does not come without a cost, of course: the CLR itself requires another 1 MB block out of your VM. And other data used by the CLR can add up as well, in the form of different heaps:

The AppDomain Heap describes the assemblies, modules, and types used by your app. The CLR uses these to find and load and decipher code as you call it. It grows as the overall app+DLLS grow.
The Process Heap and the Short Term Heap are memory used and discarded by the CLR as it works. These are usually small, usually transitory, and pretty much always beyond your control.
And finally, the GC (Garbage Collected) Heap itself, where the CLR allocates all of your data (other than large allocations stuck in global storage). Anything in here can be released by your code at any time, and the CLR will Garbage Collect it to free up space when it sees a need.

So your real limit in a Managed app is: 32 MB, minus the size of Native DLLs in slot 0 (in our case, 5 MB, mostly due to SQL Mobile), minus 1 MB for the CLR, minus the sum of all the heaps (with a minimum around 2 MB). That's how much Native data you can afford. Because remember: your Native DLLs will allocate memory in your 32 MB VM as well.

And frankly, for many sorts of handheld apps, that's a lot. But for anything ambitious, anything that really makes customers excited... That's not enough.

In testing of our app (explanations below), the JIT Heap, the AppDomain Heap, and the GC Heap have climbed to nearly 13 MB. Add in 5 MB for Native DLLs, 1 MB for the CLR, and 1 MB for the Process and Short Term Heap, and we're up to 20 MB out of 32. That's enough to make me nervous. And for large SQL Mobile queries, that may be enough to crash. (SQL Mobile usually crashes when we make a large query with less than 2 MB VM free; but we once saw it crash with 3.4 MB free.)

"No. Try not. Do... or do not. There is no try."

So now you know the CF 2.0/CE 5.0 memory landscape. You know, kind of, what your limitations are.

So if your app starts crashing and memory errors seem to be the diagnosis, you need to throw out stuff that might waste memory, right?

No. Wrong. Stop. Remember this important mantra: We're not guessing, we're profiling.

With a lot -- a lot -- of experience, you can begin to guess where system performance problems are, and be right maybe half the time. (And by performance, I include memory usage, CPU usage, and other ways your app might bog down a machine.) But people with that much experience will usually tell you: don't guess, use a profiler.

Now on the laptop, there are many profiling tools available. On CE 5, not so many. One of the best we could find is the .NET Compact Framework Remote Performance Monitor. (That's a link to the CF 3.5 Power Toys from Microsoft; but the RPM in there works fine with CF 2.0.) This tool will tell you the size of your different heaps. It will also take "snapshots" of your GC heap, which will let you see how many objects of different classes are in use, and how much space they consume. (In our case, the worst offender is 14,408 Strings, requiring 0.89 MB of storage. The top 10 classes for memory usage consume over 2.6 MB out of a 3.3 MB GC heap. I won't know why until I do more research. I have my guesses, but we're not guessing, we're profiling.)

Another useful memory profiling tool is DUMPMEM. (This is targeted at Pocket PC 2002, but still works OK for CE 5.) This tool is a bit verbose, with a lot of nearly unreadable information; but near the end of the report, it spells out all of the apps running on your device, including which unmanaged DLLs each has loaded.

Beyond those, we found it easiest to write our own profiling tools. They're crude, but they helped us match memory usage to user activities in the application. They rely heavily on the MemoryManagement class from the OpenNETCF libraries.

"Tweaking? A project that needs tweaking?"

Finally, some general lessons we learned about memory management under CF 2.0 and CE 5.0:

Design for small from the start. Don't try to add it later. Yes, I repeat myself. It bears repeating.
We're not guessing, we're profiling. Yes, I repeat myself. It bears repeating.
Don't waste time trying to "tweak" your way to sufficient memory. Tweaking is usually a sign that you're guessing, not profiling. We did a lot of little tweaks that saved us a few K here, a hundred K there... Those all made some difference, but at a lot of cost; whereas finding and eliminating the 14,408 cached strings will probably make the difference we need. When you try to tweak to fit, it's like Human Tetris: you might twist yourself into exactly the right shape; but if you're wrong, you won't know it until you go splat! Instead of tweaking, profile, identify and locate the major leaks, and fix them. Repeat, repeat, repeat.
Build memory diagnostics in from the start. You'll curse me now. You'll thank me later.
Graph your diagnostics, and identify on the graphs what the user was doing at each major change. That will help you in locating the major leaks.
Don't try to outsmart the CLR and the GC. Maybe you are smarter, but do you have time to build and test and maintain your smarter solution?
Sometimes, when your code is leaking, it can appear that the GC is not collecting. It is, at least as much as it can, and I have the graphs to prove it. Trust the GC.
Never call GC.Collect() directly. It never helps -- never! -- and it often hurts. It never helps because the GC is optimized to not waste too much CPU time. If called, it doesn't do any real garbage collection until a trigger has happened: cumulative Managed allocations have passed a megabyte boundary; GDI+ resources (pens, fonts, brushes, etc.) are used up; or memory is used up. If none of those have happened, no garbage is collected. But it may hurt, because before testing the triggers, it suspends your whole app, including waiting for each thread to reach a "suspendable" state. So though calling GC.Collect() won't save you any memory, it can cost you time.
Maybe call the unmanaged heap compact API. This seldom has had any effect for us; but on rare occasions, it has reclaimed some Virtual Memory for us. We're still kind of undecided.
If you can, wait for CE 6.0. It has a radically expanded memory mapping scheme, including 2 GB of VM per process. If that's not enough, you don't understand that the "C" in "CF" and "CE" stands for "Compact", as in "tiny". If 2 GB is too tiny for you, you're on the wrong platform. But as of this writing, there are no production devices running CE 6.0.
Consider this: is a UMPC compact enough for your customers? It's small. It's portable. And it will run the same code as your laptop does. (For us, the answer is no; but maybe you'll have a different answer...)

"Think it'll work?" "It would take a miracle."

Actually, no, you won't need a miracle to live within the memory limits of CF 2.0 and CE 5.0. You'll just need awareness, design, profiling, patience, perseverance, and maybe a little bit of luck. And maybe I've given you a few more rabbits for your hat.

Don't Make Me Send Yoda Over There to Whack You With a Stick...

Martin L. Shoemaker — Sat, 15 Nov 2008 23:49:38 GMT

15.5 hours in to what I hoped would be a 5 hour day...

Working the last major bug on the bug list...

A bug that has literally consumed over 200 hours of development spread among various team members...

A bug that our best minds looked at, and looked at, and looked at, to no avail...

A bug that got the project lead so frustrated, he did the programming equivalent of taping electrical tape over the blinking VCR clock...

And today, the black-electrical-tape patch started showing massive bugs of its own...

So we hunted... And we searched... And I got an idea for some code I could scavenge from another project...

Still didn't work... But that gave me a hint of where to look, because none of our hypothesized causes could've afflicted this new code...

Whatever was wrong was low level and fundamental... And one of my coworkers told me the sort of thing it looked like to him... And that gave me another hint...

And so I started the process of elimination... Comment out code until it worked, uncomment until it broke again, and narrow in on the single line that made the difference...

And I found it:

Application.DoEvents();

That means "Let the UI do whatever it wants. I can wait for it."

My boss had been finding Application.DoEvents() all through parts of the code... And the only explanation mentioned anywhere was, "Well, we tried this, and it worked. So we kept it."

Yoda was right: Do, or do not... There is no "try"...

If you don't understand how the code works, it doesn't.

Don't make me send Yoda over there to whack you with a stick...

Take Pity on the Poor Maintenance Coder

Martin L. Shoemaker — Sat, 15 Nov 2008 23:47:45 GMT

It might be you.

Process is a Lie

Martin L. Shoemaker — Sat, 15 Nov 2008 23:44:59 GMT

Process is a lie. All of it.

If you're spitting and fuming because you believe in software development processes, bear with me. I have a point to make, and you'll enjoy it.

If you've never believed in software development processes and you're cheering me on, take a seat and a deep breath. I have a point to make, and you're not gonna like it. But you need to hear it.

Process is a Lie?

Process is a lie, because process is a plan for how we'll work. And plans are lies. All of them.

A plan says, "This is what will happen, and this is what we'll do." A more realistic plan says, "This is what will happen, and this is what we'll do. These are things that could happen instead, and this is how we'll handle them." And a truly realistic plan says, "This is what will happen, and this is what we'll do. These are things that could happen instead, and this is how we'll handle them. And in case something happens that we haven't foreseen, these are the resources we keep in reserve and the options we keep open."

And even truly realistic plans fail. Nineteenth century German Field Marshall Helmuth Graf von Moltke told us: "No battle plan survives contact with the enemy."

Some developers and teams are process averse. And one common reason is that they've experienced plenty of process failures.

And yet Field Marshall von Moltke made plenty of battle plans. He wasn't saying "Plans are bad." He was saying "Plans are never enough. But they're still essential."

Process is never enough. But it's still essential. Process failure might indicate a flaw in that particular process; but often, it indicates that no process survives contact with the enemy.

Now the enemy in this case is not the customer (at least, not if your customer relations are healthy). The problem itself is the enemy, and your team is the invading army, trying to find its weak points, pick it apart, and conquer it. And the problem fights back! It hides minor details until they become critical. It changes due to unforeseen circumstances. It's a lot bigger than it looks. And sometimes it draws a lot of its problem friends into the battle when you least expect them.

Your process is your battle plan; and it is wrong, in ways you can't foresee until they happen. But that doesn't make it wrong.

Wrong, but Not Wrong?

OK, now Martin's getting confusing: it's wrong, but it's not wrong. What gives? Well, a plan can be (will be) wrong in details and yet still be right overall. Like all plans, your process is what you expect to do, not what you actually end up doing. A process or plan establishes a basic framework for how you'll handle the simple, mundane matters that arise; but realism tells us that things are going to happen that aren't covered by the plan. The process guides you in handling what you can foresee, so that you can spare more attention and thought for the eventualities you can't foresee.

The plan or process lets you handle simple stuff without thinking, so you have more thought to devote to the hard stuff.

So a well-designed process is good, as far as it goes; but you have to be ready to step outside it or beyond it when the need arises. And a really good process includes a process for changing or sidestepping the process. The U.S. Constitution includes an amendment process for this reason.

One of the most common sources of process disillusionment is the ISO-9000 family of standards for quality management systems. Many an organization has implemented failed ISO-9000 initiatives; and while I won't name names to protect the not-quite-innocent, more than one organization has implemented "pretend" ISO-9000 initiatives to attain certification, but never really follow their own standards.

And yet ISO-9000 is, at its core, a good process for designing processes. It really amounts to this:

Document what your process is.
Document how you gather data and measure it.
Document how you correct it when it fails.
Document how you change it when needed.
Document how you handle exceptions when needed.
Document how you'll train people in the process.
Document that you're following the process, including data.
Document that you're correcting when needed.
Document the exceptions when they happen.
Document training results.
Document changes to your process.

Forgive me if I'm being dense, but what could be wrong with any of those items?

No, when ISO-9000 fails, it's almost never because of ISO-9000 itself. Rather, it's because your process stinks; and when you're process stinks, either your team abandons it, or they blindly follow it. In some cases, the stinky process comes from outside "experts" or consultants who don't know how your team actually works. Often it's a "canned" process that the team is "shoehorned" into, with disastrous results.

What's missing, then, is an often-overlooked step 0: Decide what your process really is. Don't let some outsider tell your team their process; ask your team what their process is. Ask them questions and get them thinking about it. And then document it, document your data, etc., etc., etc.

Yes, it's possible that your team needs to learn and adopt better processes. Absolutely. But the wrong process can be worse than no process at all. If your team isn't ready or able to be circus horses, then a circus horse process will fail. You can cajole and fume and order and threaten all you want, and it won't change anything: some teams aren't ready or suited for some levels of formality. Or informality: some people adapt well to more formal processes, and find informal processes to be chaotic time-wasters. They're proud circus horses, not smelly, dirty cattle.

CAFÉ, or CAFÉ? Your Choice…

So yes, you need a development process, whether you think you do or not. In fact, if you don't think you have a process, you're probably following the most common development process, CAFÉ:

Figure 5: CAFÉ – Code And Fix, Endlessly

And if you're really lucky, sometimes you hit the end of that process. Maybe. If you define "Bugs" in a very forgiving fashion.

The trick is finding the right process, one that fits well how you're already working. Then try to describe that process so that you and your team can reach a common process. Then document that process. Then apply that process, and see where it falls short, and make it better over time.

Most popular software development processes today are incremental, building small chunks of functionality and then adding more over time. Why? Because that way you never get too far down a wrong path. You get quick feedback on what you're doing right and wrong. And then you fix the code that's wrong. (In fact, these incremental processes are all, at their core, more structured descendants of CAFÉ.)

Well, developing your development process should also be incremental. Don't try to change everything overnight. Do try to change. Then look at what worked, and do more of it. Look at what didn't work, and fix it. Repeat, repeat, repeat. When it comes to your process, you want the other CAFÉ: Change And Fix, Endlessly.

The Horse and the Stag

Martin L. Shoemaker — Sat, 15 Nov 2008 23:42:54 GMT

AT ONE TIME the Horse had the plain entirely to himself. Then a Stag intruded into his domain and shared his pasture. The Horse, desiring to revenge himself on the stranger, asked a man if he were willing to help him in punishing the Stag. The man replied that if the Horse would receive a bit in his mouth and agree to carry him, he would contrive effective weapons against the Stag. The Horse consented and allowed the man to mount him. From that hour he found that instead of obtaining revenge on the Stag, he had enslaved himself to the service of man.

Aesop's lesson here is about the dangers of Lock In: once you choose a tool or platform, you've surrendered some control to that tool or platform. You need to be wary of Lock In. A tool may lack features you need, or may provide them is a substandard form. A tool can force you to take steps and work in ways that may be wrong for your project, or may not fit your skill set. And Lock In can also leave you stranded if your tool supplier goes out of business; and it can leave you just as stranded if your tool supplier comes up with a new tool and stops supporting the old one. So put not your faith in tools!

But doesn't that conflict with The Boy and the Nettles, where Aesop advised you…

If you choose a platform, use the platform. Grasp the nettle boldly.

…?

Well, there's not a conflict there, so much as a tension between two forces. It's important to know that both forces are at work: the benefits of a tool, and the risks of a tool. You don't get one without the other. The key is to find the tool with the most benefits for your project and the fewest risks for your project; and then study the risks, see how they affect you, and decide a plan to work around them. Find the right balance of benefits vs. risk.

Let's look at our three risk areas in more detail: lack of features, lack of fit, and lack of support.

Tool Risk: Lack of Features

One way to have the fewest risks is to choose a tool with a large number of features and capabilities. As I explained in The Boy and The Nettles, that's one benefit of .NET development: no matter what common development task you can name, it's in there. For .NET 1.0, I made a presentation called .NET is an Elephant. With art courtesy of my lovely wife Sandy, I created this image:

Figure 3: .NET is an Elephant

My point was to convey that there's a lot in .NET. If I were to actually list all of the feature sets, you wouldn't see the elephant for the words.

And then with .NET 2.0, I expanded the picture:

Figure 4: .NET is a Whole Lotta Elephants!

And again, I would need a whole circus parade to list even some of the .NET 2.0 features. And then 3.0, and 3.5, and WPF, and WCF, and WF, and Live, and Silverlight, and HealthVault, and Windows Automotive, and… Well, suffice it to say, lack of feature support should not be a worry for .NET developers. (Lack of time to learn them all, on the other hand…)

And you can find rich-featured tools for other platforms as well. Don't ask me for guidance! I've got too much to learn just keeping up with .NET. I can read up on what's being done with other tools and platforms, but there's no time for me to gain deep knowledge. (I would start by asking Dianne Marsh.)

Now a separate question is whether your tool provides good support for the feature you need. I won't repeat myself on the choices involved there. Go back and read The Boy and The Nettles for my advice on that.

Tool Risk: Lack of Fit

If your tool doesn't fit your problem or your constraints, that can be difficult; but often, you can create a technological bridge for this. It may not be efficient; but often, it's easier and faster than creating your own solution from scratch. And it may be more efficient, as well, unless you're a genius who's sure you can come up with a better solution right out of the gate.

As an example, for a while there was a fairly thriving consulting niche for people who could produce PDF files from online forms. Users would fill in the forms, the system would create a PDF, and then the system would either display the PDF in a Web page or email it to the user. At the time, the most popular and common tool for building PDFs was Adobe's Acrobat Writer; but that was a desktop application, and the system required access via the Web. So the niche was in creating Web server plug-ins (using various technologies) that would feed documents to Acrobat Writer, gather up the results, and return them to the server. Today, there are canned solutions for this; but at the time, any solution was better than none. Acrobat Writer didn't fit the Web, but we could make it fit.

But if your tool doesn't fit your team, that's a more serious problem. Don't assume your team can learn a new tool too quickly. Nothing against your team, it's just that power tools usually have a lot of depth and complexity beneath an easy, appealing surface. So make sure you give your team learning time. Get them books. Send them to conferences. And yes, though I have self interest in saying so, send them to classes! Sometimes an expert can show them in five minutes what they would spend weeks digging out on their own.

And if you can find them, hire some experts to come in as tool gurus or mentors.

Tool Risk: Lack of Support

Sometimes managers worry, "What if our tool vendor goes out of business?" If you're using .NET, I think this is a very low risk. A market cap of $340 billion doesn't disappear overnight. In fact, I'll make a bold prediction right now: Microsoft will still be a significant factor in the software development market ten years from now. I made that prediction a decade ago, and people laughed at me. They were sure that Linux was going to "finally kill Microsoft". (You have to imagine the glee in their eyes when they said this…) I was right, and they were wrong. Well, I'm predicting it again: no company is guaranteed to last forever (Lloyd's of London notwithstanding), but Microsoft will still be big a decade from now. I've got a free dinner I'm willing to bet on it. Any takers?

So Microsoft will still be around; but what if they stop supporting .NET? That's a more serious concern, since Microsoft has rather radically overhauled their core technologies over the past decade. Large companies with long-range planning like to amortize strategic technology investments over as long as 15 years! Well, Microsoft usually phases out support for old technologies over 10 years, and stops most active support over 5, give or take. And the "dominance" of a given Microsoft OS is even shorter. For example, they have released seven consumer OSes in the last 15 years: Windows 3.0, Windows 3.1 (different enough that I think it deserves to be listed separately), Windows 95, Windows 98, Windows Millenium Edition (painful flashbacks…), Windows XP Home, and Windows Vista Home. That's barely two years each!

But the underlying technologies are changing more slowly: 95, 98, and ME had a lot of common core; and XP and Vista are first cousins at the farthest. If I write .NET code, most of what I wrote works equally well on XP as on Vista.

And beyond that, I can offer only this observation, somewhere between faith and prediction: in my recent experience, when a new Microsoft tool makes an old one obsolete, I find that the new one is superior on so many fronts that it's a clear benefit to adopt it; and one of those fronts is that it's easier to learn and adopt. So instead of feeling that I might "lose time" by investing in and using new tools that may go away, I find that I gain time by using something that's easier and leads to a solution faster than the previous tools I was using. Plus the next new tools will usually build on pieces of the current new tools, so time spent learning the tools today usually pays off with the tools tomorrow.

But you say, "But we're a Mac shop!" Well, Apple will probably last the decade, too, so the same basic principles apply.

"But unlike Martin, I'm not a Microsoft toadie," you say. Or "But our critical tool provider is a lot smaller than Microsoft." Well, there are no guaranteed answers. There's no guarantee your company will be around in 10 years to support your customers. And there's almost a guarantee that whoever you assign to write your own solutions won't be around in 10 years to maintain it. 5 years is iffy. And let's be honest: unless it's a core piece of the code that the developers spend most of their time in, they'll forget how it works in six months. Well, maybe three. Or one… So writing your own solution isn't much less risky than trusting to a reliable tool vendor. (See The Boy and The Nettles for more thoughts on this.) Now good design practices can help mitigate that risk, but not eliminate it.

That sort of risk is one motivation for Open Source development: if it's Open Source, then you can always fix it yourself if you have to; and there's a chance other people in the community are maintaining it; and you can find support forums online. Sometimes that works; but sometimes Open Source projects get abandoned. And sometimes some good Open Source Version 1 products become Closed Source Version 2 products, once the producers have established a market for them. Nothing wrong with that; but it means you either have to stick with and maintain the old version, or go back to trusting a vendor to stay in business and support a product.

Honestly, while this risk is non-zero, I think it's exaggerated. I'll take a small risk with a really good tool any time. If you do your homework and check reviews on the tool, you can minimize the risk. And unless the tool is core to your business, making your own is a greater risk.

Aesop’s Fables of Software Development

Martin L. Shoemaker — Sat, 15 Nov 2008 23:39:12 GMT

2500 years ago, a man named Aesop gathered the accumulated software development knowledge of the ancient Greeks, and presented the knowledge to history. However, since so few people in the intervening millennia knew anything about software, they misinterpreted Aesop's lessons as simple morality fables, rather than as hard advice for software managers. Now you can learn what Aesop had to teach us about software development processes and management.

The pioneering programmer known as Aesop is still studied today; but we tend to forget his role as a programmer, due mainly to the limitations of his hardware platform:

Figure 2: Aesop's Computer

As any programmer can tell you, everything comes down to hardware problems. In Aesop's case, these included:

Hertz. Not giga, not mega, not even kilo. Hertz.
No GUI. Not even Windows 1.0.
It was imported from China. The shipping time for replacement parts was a bummer.
It was programmed in an obscure language: Φ. (Later Φ++.)

And as any programmer can also tell you, if it's not a hardware problem, it's a user problem:

Users didn't know what they wanted.
Users kept changing their minds.
Users were impatient.
Users didn't want to pay for quality.

(Sadly, while hardware has improved since Aesop, users never change…)

So because of problems beyond his control, Aesop never shipped a product; but he did learn a lot about development practices and project management. His best practices captured in a book: ΑΕΣΟΠ Φ ΑΒΑΚΥΣ. This was later mistranslated as "Aesop's Fables". When the art of programming was lost (in the Library of Alexandria, naturally), later translators misunderstood Aesop's masterwork. For the most part, they missed the point of his lessons entirely, interpreting them as simple morality lessons.

But now, thanks to recent X-ray images of scrolls from Pompeii (and way too much time on my hands), I've restored Aesop's lessons for you in this series of essays. Look for three overarching themes:

Better programming techniques.
Better process techniques.
How could we be so stupid?

Note to hardware techs and end users…

No, I don't really think hardware and users are the cause of all problems. We're all in this together, all contributing to problems and all contributing to solutions. But when talking with programmers, some jokes are just expected. (Don't even get me started on manager jokes…)

Murphy’s Dictionary: Short Cut

Martin L. Shoemaker — Sat, 15 Nov 2008 23:35:55 GMT

Short Cut (n): Something that will take longer and cost more, but might gain some immediate goal. (Odds of gaining that immediate goal average around 7.3%.)

Example: "Yeah, we know we needed to test that, but it would take all week. We need to ship today, so we're going to take this short cut instead: we'll test after we ship."

Commentary: Really, if the short cut were a good idea, why didn't you plan to do it that way in the first place? As Professor Tolkien wrote, "Short cuts make long delays." Or as I keep telling people: Short cuts take longer and cost more, every single time.

Murphy's Dictionary: Show Stopper

Martin L. Shoemaker — Sat, 15 Nov 2008 23:34:30 GMT

Show Stopper (n): The largest bug that isn't fixed yet.

Example: "Last week, the show stopper was that the brakes cut out. This week, the show stopper is that the buttons are cyan instead of teal."

Murphy’s Dictionary

Martin L. Shoemaker — Sat, 15 Nov 2008 23:32:32 GMT

What would you get if you married Ambrose Bierce's Devil's Dictionary…

CYNIC, n.

A blackguard whose faulty vision sees things as they are, not as they ought to be. Hence the custom among the Scythians of plucking out a cynic's eyes to improve his vision.

…with the engineering adage Murphy's Law…

If there's more than one possible outcome of a job or task, and one of those outcomes will result in disaster or an undesirable consequence, then somebody will do it that way.

…?

Murphy's Dictionary: an ongoing series of definitions of what words really mean on a typical development project.

Code Is Not Enough

Martin L. Shoemaker — Sat, 15 Nov 2008 23:31:00 GMT

So you're a coding ace. You know the latest languages. You use the latest tools. You even write your own tools, so your development environment practically reads your mind. You eat, sleep, and breathe code.

So why do you keep missing your deadlines? Code is not enough.

You love to code. You can hardly believe that people pay you to "work" at a hobby you often do for no pay at all. You're never without a few personal projects under development.

So why do you feel so stressed at work? Code is not enough.

Code may be the fun part of programming, but it's only a small part of the development process. It's not enough to see you through a tough project. It's not enough to help you develop business and meet requirements. It's not enough to make your project a success.

Code is not enough. And in these essays, I'll discuss the rest of the development process.

Ulterior Motive Lounge Episode 23: Girls Just Wanna Write Code

Martin L. Shoemaker — Sat, 15 Nov 2008 23:23:55 GMT

In our last Episode, Linguist Coder and Prototype Analyst kidnapped The UML Guy and demanded he explain arrows in UML. After a rescue by the police, The UML Guy returns to the Lounge. (Click the picture for a larger image.)

This one is for Jennifer Marsman, who is quite possibly the second-most-fanatical UML geek that I know. She's also the creator of Featured Women in Technology, an ongoing series of profiles on women who shine in technology fields. Jennifer does her part to show young women that technology isn't an all-men's club. And boy, is she right! Besides her profiles of Dianne Marsh, Katy Kneale, Janet Galore, Carey Payette, Amity Binkert, and Lidiane Souza, I would add (off the top of my head) Lynne Molenkamp, Dana Coffey, Michelle Leroux Bustamante, Mary Poppendieck, Julie Lerman, Betsy Weber, Sarah Dutkiewicz, Anne Gogh, Gina Loveless, Grace Hopper, Rachel Appel, Janine Moore, Maggie Longshore, Elizabeth Naramore, Lynn Langit, Jennifer Davis, and Kirstin Juhl. Plus many more whom I forget at the moment, but whom I respect as colleagues, and to whom I offer my sincere apologies for my feeble memory. And lest we forget, Jennifer Marsman.

The Lounge is a bit testosterone heavy so far. Partly due to the characters I used as inspirations, partly no doubt from my own experience, the Lounge cast so far has run about 7 males per female. I decided that was out of balance (although honestly, it's a better ratio than most places I've worked); so it was time to add Geek Girl to the cast. Think of her as a spokesperson for all the engineers and designers and evangelists and authors and speakers in the list above. With Crystal Bernard's accent, because Wings happened to be on while I was drawing this Episode. (It's amazing the tricks the mind plays. I could "hear" Geek Girl in my head, clear as a bell, and couldn't place the voice, even as the same voice was speaking in the background.)

Modeling user interfaces in UML is a common problem; and State Diagrams are easily the best answer I've found. They're a great way to show flow that's based on user decisions, not on some algorithm. With Events and Actions, they're a great way to show control events and how your system responds to them within a given form.

And perhaps most important, they're a great way to handle The Cancel Problem. In a modern GUI, it's often the case that a user can cancel at almost any point. Even worse: if you're building a Web app, they can always just close the browser, or even disconnect from the Web. That's not an immediate cancel; but it will be a cancel when the Web session times out. If there's some work that has to be done to clean up when the user cancels, trying to draw control flow with Activity Diagrams, or even with State Diagrams, becomes a nightmare as you try to draw arrows from every possible cancel point to the clean up work. But with a Superstate that contains all of the cancelable Substates, you can easily show that all of these Substates handle cancel in the same way: by performing this clean up work. In today's Episode, for example, no matter where you are in the Booking process when you cancel, you have to go to Releasing, or the suspect will get lost in the system.

User interfaces aren't the only things you can model with State Diagrams, of course. They're a powerful tool for modeling certain kinds of business processes. In particular, if you have a document-centric process, where some document (such as an Order for goods) passes to different departments as different people act on it and different events occur, then you can easily automate that process as a document object that passes through different States in response to some Events, and which does work within a State in response to other Events. This can be a convenient way to model processes built within a Service Oriented Architecture.

Another value of State Diagrams is that a number of tools will turn these directly into executable code. In particular, Windows Workflow Foundation is a powerful tool that lets you graphically design both Sequential Workflows (similar to Activity Diagrams) and State Workflows, and then execute them directly. There's some work involved in hooking them into a user interface, but there's amazing flexibility and power in being able to depict and edit a flow graphically and then change your mind without having to change a lot of code.

Ulterior Motive Lounge Episode 22: Time's Arrow

Martin L. Shoemaker — Sat, 15 Nov 2008 23:20:36 GMT

In our last Episode, Apathetic Coder and The UML Guy derived code from Sequence Diagrams. Today, the Lounge has a pair of unexpected visitors. (Click picture for a larger image. UPDATE: The larger image is broken somehow. I probably can fix that later tonight.)

This Episode is a hodgepodge of obscure references:

Star Trek.
Zatanna the Magician.
The Ransom of Red Chief.
Benjamin Franklin.
Entity Relationship Diagrams (ERDs).
Data Flow Diagrams (DFDs).
Episode 8.
Night Court.
Columbo.
Who Framed Roger Rabbit?
Plus a couple of coworkers with unusual interests: Scott Morris, who specializes in assembling rapid prototypes of systems; and Harry Gilbert, linguist and programmer and musician (among other talents).

If you don't know all of these references, this Episode can be completely baffling.

Kinda like the arrows can be in UML, until you get used to them.

To me, the UML arrows are natural. They're "right". They're what I learned, and they make sense.

But to people skilled in ERDs and DFDs, the arrows are precisely backwards. In those notations, arrows indicate the direction that data moves: in other words, from provider to client. UML's arrows run from client to provider.

For that difference, I can only apologize. The notation is what it is. It took me two years of teaching UML before some student had the guts to raise a hand and ask "Why are the arrows backwards?" Now at least I'm aware of the issue, and (usually) remember to bring it up early so that you can grapple with it.

At the same time, what's the client and what's the provider isn't necessarily cut and dried. In fact, it can be an arbitrary implementation decision. Does a book refer to its chapters, or does a chapter refer to its book? Or both? And if the answer is both, is that one association, or two? In code, it almost has to be two; and in UML, we would model that as two arrows. But if it's a single association, we model that as just a line, not an arrow. That's also a common way to model the association when we haven't yet decided what's the client and what's the provider.

So if the arrows are confusing, I apologize. I'll do my best to make really clear examples.

This Episode is for Harry and Scott, who demanded it. Now, they didn't hold me at gunpoint, but they got really close...