posts - 20 , comments - 57 , trackbacks - 0

APIC for x86 BSP (how to build it for WCE8)

As promised I will talk about how to replace the “old” PIC (Peripheral Interrupt Controller) with the “new” APIC (Advanced Peripheral Interrupt Controller) in a CEPC (x86) BSP. I will refer to the “MyBSP” BSP in my explanation, your clone of the CEPC BSP. As APIC is mostly only available for (Intel) x86 platforms, this talk will only be valid for x86, not ARM.

The Windows CE Boot to Kernel startup phase

There are a few good MSDN links that explain quite a bit about the Windows CE startup phase. This link, for instance, explains the Kernel Startup Sequence in Windows CE 5, but most of it (as many older Windows CE articles) is still valid for Windows CE 8. I summarize briefly what we need to know:

Location BSP (common) Kernel BSP (common) BSP (MyBSP) Remarks
WINCE800\platform\common\src\x86\common\startup\startup.asmStartup()
WINCE800\private\winceos\COREOS\nk\ldr\x86\x86start.asm KernelStart()
WINCE800\private\winceos\COREOS\nk\ldr\x86\x86init.c X86Init()
WINCE800\private\winceos\COREOS\nk\kernel\x86\sysinit.c NKStartup()
DoNKStartup()
WINCE800\platform\common\src\x86\common\other\debug.c OEMInitDebugSerial()
WINCE800\platform\common\src\x86\common\startup\oeminit.c OEMInit()[1]
WINCE800\platform\MyBSP\src\x86\common\startup\oeminit.c OEMInit()[1]
WINCE800\private\winceos\COREOS\nk\kernel\nkinit.c KernelInit()[2]
... ...

KernelInit() [2] is an interesting function to examine. It calls many other functions that completely initialize the kernel. At the end of this function you find

#ifdef DEBUG
    g_pNKGlobal->pfnWriteDebugString (TEXT("Scheduling the first thread.\r\n"));
#endif

When you see the “Scheduling the first thread” debug message in PlatformBuilder’s Output window, you know you have reached the full kernel initialization stage. This is usually a good sign; messing up things in your OemInit() [1] MyBSP modifications, tend to lead to a system reboot or hangup (do I hear a sigh?). Setting a PlatformBuilder debug breakpoint also doesn’t work always before this point (reboot or hangup), after this point it will work ( except for some OEM low level interrupt code in your MyBSP) and you can start debugging, even the kernel itself. OemInit() [1] and functions called from within, is the place where we will do most of our APIC modifications. So let’s have a better look at it.

First of all you need to move this code from the platform\common folder to the platform\MyBSP folder as we will make changes to it. I suppose you know how to do this, or have a look at the source code that accompanies this blog.
Now, OEMInit() itself does not contain much code, it calls other init functions. The 2 functions we are interested in are OALIntrInit() and x86InitPICs()

void OEMInit()
{
    ...
    // initialize interrupts
    OALIntrInit ();
    
    // initialize PIC
    x86InitPICs();
    ...
}

Integrating APIC in MyBSP

You can find an example MyBSP here. It is based on a CEPC BSP that ships with Windows Embedded Compact Edition 2013 Update 5 (2014 release for Visual Studio 2013 compatibility)

  1. Clone CEPC BSP to MyBsp. Create SampleOSDesign based on MyBSP. (You need to restart VS2013 after you have cloned CEPC in order to make VS2013 aware of the new MyBSP). From the Catalog select the features you wish to add to your OSDesign. If you add LAN Networking drivers and ATAPI PCI drivers, you can test immediately if your APIC changes are OK.
  2. Add the ACPICA library to your MyBSP (as we discussed in a previous post)
  3. [MyBSP]\src\acpica
  4. Copy from WINCE800\platform\common the following files to WINCE800\platform\MyBSP
  5. [common -> MyBSP]\src\inc

    [common -> MyBSP]\src\x86
    [common -> MyBSP]\src\x86\COMMON
    [common -> MyBSP]\src\x86\COMMON\intr
    [common -> MyBSP]\src\x86\COMMON\io
    [common -> MyBSP]\src\x86\COMMON\kitl
    [common -> MyBSP]\src\x86\COMMON\startup
    [common -> MyBSP]\src\x86\COMMON\startup\newtable
    [common -> MyBSP]\src\x86\COMMON\timer

    [common -> MyBSP]\src\common
    [common -> MyBSP]\src\common\intr
    [common -> MyBSP]\src\common\intr\base
    [common -> MyBSP]\src\common\intr\base_refcount
    [common -> MyBSP]\src\common\intr\common

    We do this because a Windows CE BSP for x86 exists of a MyBSP folder + a common folder. However the APIC changes we will make are mainly located in the common folder. As we don’t want to overwrite/modify the common folder, we copy the files we will modify to our own MyBSP folder where we have all freedom to make changes without touching the original source code. I tried to make the modifications such that you can choose between PIC or APIC compilation by defining BSP_APIC (or not) in the project’s Environment Property page. BSP_APIC will #define APIC in the sources file.
    Most of the changes in these folders use the APIC define to switch between the old PIC and new APIC.
  6. Add APIC folder which contains most of the new APIC code
  7. [common -> MyBSP]\src\common\apic
    This new code will call code located in
    [MyBSP]\src\acpica
    that we added in step 2.
  8. Edit "WINCE800\platform\common\src\soc\x86_ms_v1\inc\pc.h" such that the INTR_xxx defines reflect your (first) APIC IRQ0-IRQ15 situation. Depending on your hardware motherboard, certain “old PIC” IRQs might be mapped differently. This change is unfortunate (as we are modifying a common header file); you can implement a different mechanism to obtain the INTR_xxx defines (I know a few places where these defines are used, but maybe not all), but this was by far the easiest and safest implementation.
  9. Change Platform.reg to tell the Device Resource Manager we have more IRQ’s available
  10. [HKEY_LOCAL_MACHINE\Drivers\Resources\IRQ]
    "Identifier"=dword:1
    "Minimum"=dword:1
    "Space"=dword:F
    "Ranges"="1,3-7,9-0xF"
    "Shared"="1,3-7,9-0xF"
    IF BSP_APIC
    "Space"=dword:17
    "Ranges"="1,3-7,9-0x17"
    "Shared"="1,3-7,9-0x17"
    ENDIF BSP_APIC
  11. Set KITL to polling mode. Why? As we are changing PIC to APIC and in case we make a mistake, KITL in interrupt mode most likely will not work. As long as we are debugging, it is best to switch KITL to polling mode.
    Edit “WINCE800\platform\MyBSP\src\x86\common\kitl\kitl_x86.c” function OALKitlStart() as follows:
  12. switch (g_pX86Info->KitlTransport & ~KTS_PASSIVE_MODE)
    {
    ...
    case KTS_ETHER:
    case KTS_DEFAULT:
    fRet = InitKitlEtherArgs (&kitlArgs);

    // MyBSP start
    // Override to set poll mode
    kitlArgs.flags |= OAL_KITL_FLAGS_POLL;
    g_pX86Info->ucKitlIrq = OAL_INTR_IRQ_UNDEFINED;
    // MyBSP end
    break;
    default:
    break;
    }
  13. Modify OemInit() in “WINCE800\platform\MyBSP\src\x86\common\startup\oeminit.c”
  14. void OEMInit()
    {
    // Set up the debug zones according to the fix-up variable initialOALLogZones
    OALLogSetZones(initialOALLogZones);

    OALMSG(OAL_FUNC, (L"+OEMInit\r\n"));

    // initialize interrupts
    OALIntrInit ();

    #ifndef APIC
    // initialize PIC
    x86InitPICs();
    #endif

    // Initialize PCI bus information
    PCIInitBusInfo ();

    // starts KITL (will be a no-op if KITLDLL doesn't exist)
    KITLIoctl (IOCTL_KITL_STARTUP, NULL, 0, NULL, 0, NULL);

    #ifdef DEBUG
    // Instead of calling OEMWriteDebugString directly, call through exported
    // function pointer. This will allow these messages to be seen if debug
    // message output is redirected to Ethernet or the parallel port. Otherwise,
    // lpWriteDebugStringFunc == OEMWriteDebugString.
    NKOutputDebugString (TEXT("CEPC Firmware Init\r\n"));

    #endif

    // sets the global platform manufacturer name and platform name
    g_oalIoCtlPlatformManufacturer = g_pPlatformManufacturer;
    g_oalIoCtlPlatformName = g_pPlatformName;

    OEMPowerManagerInit();

    #ifndef APIC
    // initialize clock
    InitClock();
    #endif

    // initialize memory (detect extra ram, MTRR/PAT etc.)
    x86InitMemory ();

    // Reserve 128kB memory for Watson Dumps
    dwNKDrWatsonSize = 0;
    if (dwOEMDrWatsonSize != DR_WATSON_SIZE_NOT_FIXEDUP)
    {
    dwNKDrWatsonSize = dwOEMDrWatsonSize;
    }

    x86RebootInit();
    x86InitRomChain();

    OALMpInit ();

    g_pOemGlobal->pfnIsProcessorFeaturePresent = x86ProcessorFeaturePresent;

    #ifdef APIC
    // this assembly code disables any old PIC interrupt
    __asm
    {
    mov al, 0xff
    out 0xa1, al
    out 0x21, al
    }
    // start APIC
    x86InitAPICs();

    // initialize clock
    InitClock();
    #endif

    #ifdef DEBUG
    NKOutputDebugString (TEXT("Firmware Init Done.\r\n"));
    #endif

    OALMSG(OAL_FUNC, (L"-OEMInit\r\n"));
    }

That’ s about it. Compile and debug.

New APIC code

In step 4 we briefly mentioned that we need to add the new APIC code

[common -> MyBSP]\src\common\apic

Let’s see what that means. This folder contains source file apic.cpp with function x86InitAPICs() with the real meat. This function goes through all steps required to setup the APIC chipset and hook the interrupts to the correct vectors.

  1. x86InitACPICA(). Remember this function from my previous blog? It initializes the ACPICA library that we need in a moment to query ACPI for APIC information.
  2. ApicGlobalSystemInterruptInfo() will init a global data structure that will contain all APIC chipset information and what IRQ source is connected to what APIC chip INT line
  3. SetApicEnable() will search for APIC chipsets (via the LowPinCount (LPC) Intel PCI device) and enable it. Over the years Intel devised a few mechanisms how to do this, I suggest you read the source code to learn how to. Newer chipsets might require adaptation to this code.
  4. ApicIrqRoutingMapInfo() uses the ACPICA library to learn from the _PRT (PCI Routing tables) tables in ACPI how the PCI interrupts are routed to your APIC chipset(s). I only encountered Intel chipsets with only 1 APIC chip, but motherboards can have multiple APICs. Although the code is prepared for it, I could not test boards with multiple APICs.
  5. pci_IRQ_mapping() ties the PCI interrupt info from the _PRT tables to the real PCI devices in your motherboard. _PRT in ACPI tells you that *if* you encounter a PCI device (behind a PCI bridge) to what APIC interrupt it is routed, but you need to scan your PCI bus yourself to find the currently present devices.
  6. Last but not least, you need to HookInterrupt() the APIC IRQs to the interrupt vectors. E.g. a 24x IRQ APIC will let you hook 24 vectors.
  7. x86UninitACPICA(). Clean up (memory) resources used by the ACPICA library.

  8. OALIntrRequestIrqs(). This method will be called by PciBus.dll when setting up drivers. In the old PIC mode this method interrogates the BIOS (and queries the PCI configuration registers) how the PCI interrupts are mapped to the 8259’s. In the new APIC mode this information comes from ACPICA. This method was originally implemented in WINCE800\platform\common\src\x86\common\io\route.c where I surrounded it with the #ifndef APIC define.

Each individual step might require more in-depth explanation. You can find many useful comments in the source code or I refer to the Intel datasheets. An interesting source for APIC programming can be found here http://www.intel.com/content/dam/doc/specification-update/64-architecture-x2apic-specification.pdf

Tips and Tricks.
  • How do you debug an interrupt service routine (ISR) without KITL? As KITL itself might use interrupts or is not yet available in an early bootup sequence.
    • Set KITL to polling mode or
    • Use OALMSGS macro. This will send your trace message to the (first) serial port without using interrupts, instead of the normal OALMSG that will send your traces to KITL once it is up and running. Be sure you disable the OALMSGS macros in your release build as they are pretty slow and will make your ISR unnecessary slow.
  • To control the OALMSG debug zones, set the default value in config.bib
      nk.exe:initialOALLogZones 00000000 0x0000000B FIXUPVAR
    • See also “WINCE800\platform\common\src\inc\oal_log.h”
  • Some notes on the ATAPI driver. This driver is responsible for controlling the IDE chipsets that drive your hard disks. The ATAPI driver can work in
    • Legacy mode = PIC mode => fixed IRQ 14 and 15
    • Native mode = APIC mode => variable IRQ according to board manufacturer.
  • By default the ATAPI driver is set to Legacy mode and works with fixed IRQ 14 and 15. However you can switch the driver and hardware easily to Native mode via the Registry. Add the following registry keys to the end of Platform.reg
  • IF BSP_APIC

    ; @CESYSGEN IF CE_MODULES_ATAPI
    ; @XIPREGION IF PACKAGE_OEMXIPKERNEL
    ; HIVE BOOT SECTION
    IF BSP_NOIDE !

    ; @CESYSGEN IF ATAPI_ATAPI_PCIO
    [$(PCI_BUS_ROOT)\Template\I82371]
    “ProgIF”=dword:8F
    “LegacyIRQ”=- ; The primary legacy IRQ, remove for native mode
    “ConfigEntry”=”NativeConfig” ; PCI configuration entry point
    ; @CESYSGEN ENDIF ATAPI_ATAPI_PCIO

    ; @CESYSGEN IF ATAPI_ATAPI_PCIO
    [$(PCI_BUS_ROOT)\Template\GenericIDE]
    “ProgIF”=dword:8F
    “LegacyIRQ”=- ; The primary legacy IRQ, remove for native mode
    “ConfigEntry”=”NativeConfig” ; PCI configuration entry point
    ; @CESYSGEN ENDIF ATAPI_ATAPI_PCIO

    ENDIF BSP_NOIDE !
    ; END HIVE BOOT SECTION
    ; @XIPREGION ENDIF PACKAGE_OEMXIPKERNEL
    ; @CESYSGEN ENDIF CE_MODULES_ATAPI

    ENDIF BSP_APIC

    The ATAPI driver has code to switch automatically from legacy mode to native mode when loaded. To see how it works, check out (set a breakpoint) in NativeConfig() method in WINCE800\public\common\oak\drivers\block\atapi\pcicfg.cpp. This piece of code detects legacy mode and if the right conditions are met, it disables this PCI device so that in a later stage of the PciBus enumeration, this device is “placed” again in native mode. Hint: place also a breakpoint in OALIntrRequestIrqs() method in WINCE800\platform\MyBSP\src\x86\common\apic\apic.cpp to see how the APIC irq is requested.

Here you can find the MyBsp WCE8 Board Support Package that includes all the source code required to build a working image based on Intel chipsets with APIC. I have tested it successfully with IDE, USB, NIC. Copy the files in WINCE800\platform\MyBsp and create an OSDesign with it. Remember:

  • Mind the OALMSGS() in APIC_ISR() in WINCE800\platform\MyBSP\src\x86\common\apic\apic.cpp.
  • Mind the nk.exe:initialOALLogZones 00000000 0x0000410F FIXUPVAR in WINCE800\platform\MyBSP\FILES\config.bib
  • Mind WINCE800\platform\common\src\soc\x86_ms_v1\inc\pc.h

I tested most of the code on Intel ICH4, ICH7 and NM10 chipsets with 1 APIC on board. The code might not work for other chipsets/motherboards combinations, if so feel free to let me know.

Good Luck!

Useful references:

Print | posted on Monday, June 23, 2014 8:39 PM | Filed Under [ Windows CE Windows Embedded Compact Embedded RTOS Microsoft APIC ACPI ACPICA BSP Visual Studio 2013 ]

Feedback

Gravatar

# re: APIC for x86 BSP (how to build it for WCE8)

Great blog! In previous post you have written "I needed a timer. A timer with a granularity of < 1msec and not tied to the Windows CE 1msec kernel tick. I wanted to program the timer hardware, raise an interrupt and signal an event to trigger specific code in my application."
After replaycing PIC with APIC it can by achived, would be nice if you could write how to run that timer. ? :)
6/24/2014 10:57 AM | bataliero
Gravatar

# re: APIC for x86 BSP (how to build it for WCE8)

It is on the planning :-). A little more patient...
6/24/2014 8:00 PM | Werner
Gravatar

# re: APIC for x86 BSP (how to build it for WCE8)

Then I look forward to see next posts : )
6/27/2014 3:17 PM | bataliero
Gravatar

# re: APIC for x86 BSP (how to build it for WCE8)

Thanks for the instructions on Win CE. I tried to set up an apic timer, but i got stuck on the code of step 4. Where do I get it? Or do I have to code it myself?

(Do you know if this works for WinCE 7.0 ?)
12/16/2015 8:59 AM | Andy
Gravatar

# re: APIC for x86 BSP (how to build it for WCE8)

Hi Andy,
I think you refer to the ACPICA source code. You can download it from here
https://acpica.org/downloads/windows-source
I hope this is what you are missing.

Yes, I originally started integrating ACPICA in WinCE 7, buit moved lateron to WinCE 8. You might need to tweek a bit, but the general idea is the same.

Good luck!
12/16/2015 10:07 PM | Werner
Gravatar

# APIC for Windows CE 6.0

Hi, great post

I'm interested in changing interrupt mode in Windows CE 6.0 from PIC to APIC.
I guess there are differences, but the general idea behind would be the same in this case too?

thanks!
11/23/2016 10:33 AM | ibai
Gravatar

# re: APIC for x86 BSP (how to build it for WCE8)

yes, the steps to follow are similar
11/25/2016 6:49 PM | werner
Post A Comment
Title:
Name:
Email:
Comment:
Verification:
 

Powered by: