When Things Go Wrong

Platform Builder 6.0: Project Creation Failed

Bruce Eitman — Wed, 19 Nov 2008 22:00:20 GMT

I received an interesting question from an Engineer today. It read like this:

"I have an issue that has cropped up with using Platform Builder 6.0. I installed a newer version of a CE BSP on my system and now Platform Builder will not open my old project nor create a new one. When creating a new project I get an error message on the status bar that only says “…project creation failed”. If I restore the old BSP I can create new projects so I think the problem is somehow related to the new BSP. "

So what I knew was:

He is developing for Windows CE 6.0
He had an old BSP
He has a new BSP

What I guessed was that the old BSP and the new BSP were both in the Platform folder, and that they both had something in common that Visual Studio and Platform Builder where looking at. I suspect that the thing they have in common is the catalog file.

I suggested that he remove the catalog file (Catalog\MyBSP.pbcxml) from the old BSP and try again. What he did was to remove the entire old BSP. The result was that everything worked again.

I tried to reproduce the problem and could not.

Platform Builder: Bluetooth Component Inconsistencies

Bruce Eitman — Thu, 06 Nov 2008 22:10:26 GMT

Subtitle: Why adding Bluetooth Stack with USB doesn’t always add Bluetooth USB

Argh, one of my junior engineers just gleefully exclaimed to me “Anyway, I WIN! :-)” Now that just burns, but I take comfort in the fact that she got lucky.

We were both adding Bluetooth support to different workspaces that supported the same board. My project finished building before hers so I asked her to test my project. The USB Dongle that she was using didn’t work with my OS image; it popped up the little dialog asking for the driver to use. So her project finished building and she tried it, and it worked just like it was meant to work.

We compared notes on what we both had added. She added:

· Bluetooth Stack with Universal Loadable Driver

· Bluetooth Stack with Integrated USB Driver

I added:

· Bluetooth Stack with Integrated UART Driver

· Bluetooth Stack with Integrated USB Driver

· Bluetooth Stack with Universal Loadable Driver

Sure it seems silly to add the universal loadable driver and the others but I wanted to make sure that it worked for my customer. I thought that I was adding more than they needed, but that wouldn’t matter. I was wrong.

So I looked into reginit.ini for the registry settings for the USB Bluetooth. It wasn’t there. I searched for BTH and found that the USB settings would have been added if CE_MODULES_BTHUSB is set. I thought that should have been set since I added the USB Driver and a check of ceconfig.h confirmed that it wasn’t set.

Digging further, I looked at Public\CEBASE\OAK\MISC\winceos.bat, and found the problem. The problem is that in winceos.bat, the catalog components under Bluetooth Protocol Stack with Transport Driver Support become mutually exclusive. That is that the logic in winceos.bat only includes the first driver that it looks for and finds set, the first on is the UART driver. So really the catalog components should be a set as choose one of, not choose any of. That would make the catalog consistent with the batch files that that process the SYSGEN variables.

Windows CE: The Case of the Missed Interrupt

Bruce Eitman — Thu, 06 Nov 2008 18:49:18 GMT

I was recently working on an interrupt that was asserted, but the system didn’t respond by handling the interrupt. The interrupt that wasn’t being handled came from a CPLD into a GPIO pin. The CPLD asserts the interrupt in response to several external events. I thought that this would make a good example of what not to do in an ISR.

If only one of the external events was triggering, then the interrupt seemed to work well. If more than one triggered, not necessarily at the same time, then eventually the interrupt stopped being serviced. Seemed like the system never saw the interrupt, but of course it is handled in hardware so that seemed unlikely.

I reviewed the code. Here is what it did in response to this interrupt:

In the ISR:

1. Clear the GPIO interrupt from the CPLD

2. Mask off the interrupt in the CPLD, so that other interrupts to the CPLD can cause the interrupt when they arrive

3. Clear the interrupt in the CPLD

In the IST:

1. Handle the source of the interrupt

2. Call InterruptDone

In InterruptDone

1. Mask all interrupts to avoid read/modify/write errors from the ISR

2. Re-enable the interrupt in the CPLD

3. Clear the GPIO interrupt from the CPLD

4. Re-enable the interrupt from the CPLD, which was never disabled

5. Re-enable all interrupts

Do you see where the problem is? Neither did I for a couple of days. I poked around the system, when this failed the interrupts where all enabled, but not asserted. Not asserted? Well that turned out to be my test application detecting the problem and “fixing” it by reinitializing the external hardware so it actually made sense.

Actually if you read through what it is doing and only think about one signal into the CPLD, it does work. Well, sort of but only if the external interrupts are spaced apart, which they were.

The real problem is step #3 of handling InterruptDone. To understand, realize that we can only execute step #1 if the interrupt is not asserted, if it was asserted we would be in the ISR not in InterruptDone. After step #1, the interrupt can assert without going to the ISR. If it does assert before completing step #3, then the interrupt is cleared and the interrupt controller will not signal the ISR to run.

Windows CE: Closing Invalid Handles

Bruce Eitman — Thu, 23 Oct 2008 16:59:41 GMT

I pulled some old source code off the shelf recently, dusted it off and started modifying it. Okay, that was my first mistake… I really should have tested it first but that actually would not have helped this time.

When the system booted up, it successfully loaded drivers and then when it came to loading Explorer.exe all heck broke loose. Sometime the desktop would appear and other times it would not. When the desktop would appear, GWES would data abort. The system was a mess, and needed to be fixed.

We knew the problem wasn’t really in GWES, so we didn’t even try to find the source of the data abort. This was a good decision because it wouldn’t have told us anything even if we did track down where the problem was.

Instead, we when through my list of known problems that could stop the system from working. If I ignored the data abort, it looked like the scheduler stopped scheduling threads to run which could be why the desktop sometimes not be displayed, which happened more often than it being displayed. Maybe it is the old Intel bug that caused the timer interrupt to stop for 20 minutes. Looking at the code, showed me that this project was old enough that it had not been fixed, so we fixed it and had it output a character every 100 milliseconds. To my surprise, the interrupt stopped. Not only that, but all interrupts stopped.

Being a hopeful kind of engineer, I decided that maybe I could write an application that would continue to run long enough to read registers and see why the interrupts had stopped. I went back to my desk and wrote the following:

int WINAPI WinMain( HINSTANCE hInstance,

HINSTANCE hPrevInstance,

LPTSTR lpCmdLine,

int nCmdShow)

{

CeSetThreadPriority( GetCurrentThread(), 50 );

{

RETAILMSG( 1, (TEXT("L\n")));

} while( TRUE );

}

It isn’t the most remarkable code that I have ever written, but before I put too much into it, I wanted to make sure that it ran long enough that it would be useful. This code lives a little on the edge; it sets the thread priority fairly high and doesn’t play nice with the system by blocking to let other threads run. My hope was that once it started running, that the scheduler would let it keep running even after interrupts stopped.

The system that I am working with includes an application that is started using the HKEY_LOCAL_MACHINE\Init key. That application then starts other applications it finds on a CompactFlash card. So I grabbed a CompactFlash card that I had lying around and copied the new test application to it. But first before testing, I thought that it would be good to look at the files on the card to see if there was anything that should be deleted, like a bootloader or OS image that might not be for this system. That is when it hit me like a ton of bricks what the problem was because it was something that I had seen, and fixed, before on all of my current projects (the ones that aren’t so dusty.)

The Problem

What happened when I looked at the CompactFlash card was that I saw that it had a little exe on it that is special. This application is special in that if the application that runs other applications sees this file on the CF card, then it copies it to the flash drive so that it can run again even when the CF card is removed. The start of the problem is how it copies the file, which I won’t go into great detail about except to say that it creates a zero byte file even if the file isn’t on the CF card, so the file always exists on the flash drive, but may not really be an executable file.

The code then tries to run the file from the flash drive. The code appeared to be good on the surface. Here is some pseudo code that shows what it was doing:

Check for the application on the flash drive

If the application exists on the flash drive

Call CreateProcess to start the application

Close the handles returned from CreateProcess to avoid leaking handles

Now the problem comes from closing the handles returned from CreateProcess without first checking to see if the call to CreateProcess was successful. As it turns out, if you call CreateProcess on a zero byte exe file, CreateProcess will fail – I know that is a shocker. To exacerbate this problem, the PROCESS_INFORMATION structure being passed into CreateProcess was declared on the stack, but not initialized in any way shape or form. So the values contained in the handles were unknown and certainly not set to INVALID_HANDLE_VALUE which means that when CloseHandle is called the values could appear to actually mean something.

So while I titled this post to suggest that it is about calling CloseHandle with invalid handles, this application was a Perfect Storm just waiting to bite me in the … It really had three problems; creates a zero byte exe, doesn’t initialize the PROCESS_INFORMATION structure, and then didn’t check the result from CreateProcess before closing the handles. Any one of those problem, if fixed, would prevent the problem.

So in the end, by fixing this code we were able to boot the system successfully.

Platform Builder: Debugging the Boot Hive

Bruce Eitman — Tue, 14 Oct 2008 02:00:43 GMT

One of my engineers was working on a new system today and was having a problem setting up a NAND flash disk. The problem was that while he had made a change to the registry to set up the start address and size of the disk, the changes just were not visible in when the OS ran. We both have a lot of experience developing and building Windows CE systems, so we checked the obvious things:

1. He save Platform.reg where he made the changes

2. He copied Platform.reg to the _FLATRELEASEDIR

3. When makeimg ran, the changes made it to reginit.ini

4. He actually loaded the changed OS on the board and it was running

Then he came up with an idea, load the nk.bin in Visual Studio with Platform Builder 6.0. I have to admit that I didn’t think this would yield any useful information that we didn’t already have, but I wasn’t going to stop him. The next thing I heard from his office was that he had solved the problem. At that moment I realized the importance of what he had done.

If you haven’t opened an nk.bin with Platform Builder 6.0, you should do so to see what it can show you. When it comes to the registry, it can show you what values are in the boot hive vs. the system hive. In this case, it showed that his changes were not in the boot hive when they were needed, but instead were in the system hive which isn’t loaded until after the flash disk is already mounted.

To open an nk.bin in Platform Builder 6.0, use the Files menu select Open\File and select the file type “Windows CE Run-Time Images (*.bin; *.dio)”, then select your nk.bin. You can use this on nk.bin files created with previous versions of Platform Builder. When the nk.bin file is opened you will see it displayed with a list of files included in the OS, the system hive and the boot hive.

Even if you are working with older version of Windows CE, you might find it valuable to have Platform Builder 6.0 on hand just for the ability to view the registry in a more effective way.

Platform Builder: Data Misalignment

Bruce Eitman — Wed, 03 Sep 2008 20:04:20 GMT

I ran into a problem today with data misalignment. The problem would have taken a lot of time to find if I wasn’t already familiar with the problem (kicking myself now.)

I was porting an existing driver into an old platform to update the platform. So I know that the driver works, but when it started it would data abort. This particular driver is quite large and complex, so a simple read of the code would not be possible in a reasonable amount of time. I have some experience with tracking the cause of a data abort (Windows CE: Finding the cause of a Data Abort.) But this data abort was occurring in Coredll.dll, so I knew that it was because of something that the driver was passing into a function.

Thinking back on what I had done to port the driver into this platform here is what I did:

1. Copy the driver into the platform

2. Added some members to a data structure in a header file already in the platform

I know that the driver is well tested, but let’s review the data structure:

#pragma pack(1)

typedef struct _MYSTRUCT {

BYTE var1;

BYTE var2;

DWORD var4;

BYTE var5;

DWORD NewVar1;

DWORD NewVar2;

DWORD NewVar3;

} MYSTRUCT;

#pragma pack()

If you look at it closely, you may already have seen my problem. Of course depending on my driver code, this structure could be perfectly acceptable, but knowing my driver it is not. My driver increments NewVar1 using InterlockedIncrement() and decrements it using InterlockedDecrement(). If you aren’t familiar with these functions, these functions take as their parameter a pointer to a 32 bit location. InterlockedIncrment() then effectively does *ptr++ which when I pass in the address of NewVar1 is dereferencing a location that is not 32 bit aligned, which for the ARM processors that I work with causes an exception.

The alignment is not a problem when accessing the structure directly. If I had the following code:

MYSTRUCT *Ptr;

//Allocate some memory for Ptr…

Ptr->NewVar1++;

This would compile and run okay, although some extra assembly code would be generated to handle the alignment for me. The problem also would not exist if the structure wasn’t wrapped with the pragma pack, which tells the compiler not to add any padding to the structure to align the members.

The problem comes from passing the address of NewVar1 into another function.

The solution for my problem is to cause NewVar1 to be aligned by adding an extra byte before it in the definition.

Go to Summary of When Things Go Wrong

Tags: Data Abort

Platform Builder: My Stream Interface Driver Does Not Load

Bruce Eitman — Fri, 08 Aug 2008 17:07:05 GMT

You just created a new Stream Interface driver, but it isn’t being loaded. I hear this a lot and thought it would be good to look into the possible causes. Hopefully, if you are here you have already reviewed Windows CE: A Stream Interface Driver Shell which provides a very basic driver an all of code and other files to demonstrate a driver that actually loads.

The first things to check:

1. Is the driver in you OS image? Check to see that it is in %_FLATRELEASEDIR%\ce.bib.

2. Do all of the functions return a value that indicates success?

3. Are the driver’s registry settings in the OS image? Check for them in %_FLATRELEASEDIR%\reginit.ini.

4. Did you load the OS image with the new driver? This is worth checking, really it is.

So you have checked the basics, and the driver is in the image and the registry settings are there to load the driver, but it still isn’t being loaded. The next thing to check is that you have exported the basic functions that are needed for the device manager to load the driver.

If you don’t have the proper functions exported, device manager may not load the driver. When the device manager is loading a driver, it checks to see if the driver has the exported functions that it will need. The logic looks like this:

· The driver must have Init and Deinit functions.

· If the driver prefix exists in the registry then

o It must have Open and Close functions

o It must have at least one of the following functions; Read, Write, Seek and IoControl

o If the function PreClose exists, there must also be a PreDeinit function.

If the driver is not loaded becuase the driver failes the above test, the debug output from a debug build will be:

UDEVICE!CreateDevice: illegal entry point combination in driver DLL 'MyDriver.dll'

Dumpbin can be used to tell you which functions have been exported from a DLL. The EXPORTS argument tells dumpbin to show the exported functions. Here is the output from running dumpbin /EXPORTS on DriverShell.dll.

_FLATRELEASEDIR>dumpbin /exports DriverShell.dll

Microsoft (R) COFF/PE Dumper Version 7.10.4017

Dump of file DriverShell.dll

File Type: DLL

Section contains the following exports for DriverExample.dll

00000000 characteristics

488A2DD5 time date stamp Fri Jul 25 15:47:33 2008

0.00 version

1 ordinal base

10 number of functions

10 number of names

ordinal hint RVA name

1 0 00001230 XXX_Close

2 1 00001188 XXX_Deinit

3 2 00001250 XXX_IOControl

4 3 00001190 XXX_Init

5 4 0000122C XXX_Open

6 5 00001228 XXX_PowerDown

7 6 00001224 XXX_PowerUp

8 7 00001238 XXX_Read

9 8 00001248 XXX_Seek

10 9 00001240 XXX_Write

Summary

1000 .data

1000 .pdata

1000 .reloc

1000 .text

The output shows that the necessary functions are exported. But what if your dumbin output doesn’t include the functions? Of course the first thing to check is that your driver includes the functions. The next step is to export the functions using either a def file like I did in DriverShell or by indicating that the function should be exported in the source code itself.

These are the reasons that I can think of for a driver to fail to load, add a comment if you can think of other reasons. I have left out of this discussion reasons why a driver does load, but then fails, that is a different discussion.

For more see Windows CE: Device Driver Summary

Tags: Drivers

Windows CE: Why is reading from the driver so slow?

Bruce Eitman — Wed, 06 Aug 2008 00:41:23 GMT

I was working with a customer today and discovered that their application was calling into one of my drivers repeatedly to read 4 bytes at a time, each time reading the next 4 bytes of data. The driver reads a minimum of 4 bytes, or they probably would have read less, but the driver is capable of reading large quantities of data on each call from the application.

This really isn’t all that uncommon though and it actually will read the data correctly. The problem is that each call to the driver has some overhead. So if the calling into the driver takes a few microseconds it isn’t bad. But if the application multiplies that overhead by hundreds or thousands of calls, it starts to add up and reduce application and even system performance.

To demonstrates cost of overhead, I wrote the following code:

void TestDriverShell()

{

HANDLE hDS;

DWORD Count;

DWORD StartTime;

DWORD EndTime;

hDS = CreateFile( TEXT("XXX1:"), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, 0 );

if( hDS != INVALID_HANDLE_VALUE )

{

StartTime = GetTickCount();

ReadFile( hDS, &Count, 4, &Count, 0 );

EndTime = GetTickCount();

RETAILMSG( 1, (TEXT("ReadFile time: %d\n"), EndTime - StartTime ));

StartTime = GetTickCount();

for( Count = 0; Count < 512; Count++ )

ReadFile( hDS, &Count, 4, &EndTime, 0 );

EndTime = GetTickCount();

RETAILMSG( 1, (TEXT("ReadFile time: %d\n"), EndTime - StartTime ));

}

This code call ReadFile() on the DriverShell example driver that included in another post Windows CE: A Stream Interface Driver Shell. DriverShell doesn’t do anything but return when ReadFile() is called. This means that the time it takes to call ReadFile() is all overhead.

On the system that I tested this, the first call reports that the time to call ReadFile() is 0 milliseconds, but the time to call it 512 times is 26 milliseconds. What is 26 milliseconds? Really not much, but imagine that the driver spends more time actually reading data from hardware. That could start to really add up, and in some cases, like reading from a serial port, the application could start to lose data.

Calling WriteFile with small amounts of data can have the same effect.

For more see Windows CE: Device Driver Summary

Tags: Drivers, ReadFile

Platform Builder: Changing Memory Divisions Causes the System to Fail

Bruce Eitman — Thu, 24 Jul 2008 00:23:07 GMT

So your system is running well, and then someone decides to change the memory divisions using the System Control Panel Applet and BAM something in your system data aborts. This can be difficult to solve, if you don’t have any idea what the cause is, and a challenge even if you do know. If you follow the data abort back to the code that failed (Windows CE: Finding the cause of a Data Abort) you will find yourself confused because the code there most likely did not cause the data abort, but instead the data abort is a symptom of a bigger problem in your system.

The root cause of the problem is likely that you have an overlap of the systems RAM with some RAM that you have allocated for something else, like the display controller’s frames. To figure this out, you might need to do some digging to figure out your total system memory map of RAM.

Summary of When Things Go Wrong

Tags: Data Abort

Platform Builder: IXP46x, 128 MB RAM, CEBaseCESysgen.bat - System Fails to Boot

Bruce Eitman — Wed, 23 Jul 2008 01:02:07 GMT

It has been another one of those days. Several months ago we discovered that one of our systems wouldn’t boot if we did a clean build, but if we built the kernel separately and then ran makeimg the system would boot. What a mystery, that just didn’t make any sense.

So, the engineer on the project modified PreMakeimg.bat to rebuild the kernel when we run makeimg. All was good, but this increases build time. If you have followed this blog, you know that I don’t like long build times but we all tolerated this because there were more important issues for us to address.

Then a second platform had the same problem. I made it a priority to figure out what was wrong this time though because now we were losing time in trouble shooting and in building.

Since the first build is a full build of the system, there is a build.log file in %_WINCEROOT% that I could look at for clues. And the second build was just in the kernel folder, so there was a build.log there as well. I diffed the two files. At first didn’t seem too interesting. That is until I scrolled to the right and could see the command line calls to compile and assemble. At that point it became clear that during the full build of the system that the RAM size was being defined twice, once setting the RAM size to 64MB and the other to 128MB. Our boards have 128MB of RAM so clearly the setting of 64MB was wrong, and of course having both sizes set was a problem.

Now to figure out why this was happening only when we did the full build. I will leave out the details of my search other than that I ruled out sources.cmn, sources, platform.bat and the project’s PBXML file as the cause. Then I noticed that in the root of the platform there was a CEBaseCESysgen.bat file. I had previously ignored it, but now it was beginning to make me curious.

When I opened CEBaseCESysgen.bat, what I found was that it set the RAM size to 64 MB if it is not set to 16, 64 or 256MB, but these platforms are 128MB. Apparently, someone had added support for 128MB, but not noticed this file.

So, what is happening with this CEBaseCESysgen.bat? During the sysgen of the project the platform is built with CEBuild.bat, which runs CESysgen.bat during the build of the platform. CESysgen.bat in turn runs CEBaseCESysgen.bat during the different phases of the build process. But, when we build the kernel separately we just run build.exe which doesn’t run CEBaseCESysgen.bat.

Summary of When Things Go Wrong

Tags: Build.exe, CEBuild.bat