July 2011 Entries
Virtualized data center - Part Two: Hardware
Instead of making a list here of what requirements or needs for hardware hosts exist, I’ll make it plain and simple: CPU, memory and network links. Those are the most important things in a virtualization host and is all well documented in build guides across the internet.
What I would to talk about is the following: Whitebox vs. Branded servers!
Throughout the years I’ve been pretty much brainwashed (no kidding) that in a professional, production environment you only use “real” server hardware. And while I agree in every way on this statement for the single server appliances I have been thinking plenty about using whitebox machines as server hardware in virtualization hosts.
Now if you are using VMware this is mostly a mute point. There are very little boxes that you can build which are totally compatible with ESX so you are probably better off using branded servers.
But what if you decided to use Microsoft Hyper-V? Wouldn’t this allow you to build cheaper, bigger bang for the buck, virtualization hosts? Sure enough, you are not going to build a Quad socket, 256 Gb Ram machine on retailed hardware, but then again, why would you?
Unless you plan to deploy a virtualized data center in a large environment I don’t see the point of having a monster server like that sitting in your rack.
I have been researching the possibilities of using “cheap” hardware in a cluster environment to run as virtualization hosts. This particular approach requires more hosts in your environment to match up with larger branded offerings, but it also gives you a more granular control on what machine runs where and , very likely, a faster hardware procurement process.
My point here is that , taking Dell (R610) as an example a branded server will cost you about 3085 euro (incl taxes), whilst a white box configuration, built to a similar spec would cost you around 1200 euro (incl taxes) [As reference I’m talking about a single Quadcore processor, 24 GB or RAM, a sata drive, no OS licence or additional network cards...]
Now, if you take that cost difference you could deploy a cluster of 5 whitebox servers in comparison to a 2 server branded cluster. Yes, generally speaking branded servers give you better support options but these have NOT been selected in the webshop. If you could strike up a deal with a local hardware vendor I’m sure you get the boxes delivered to you manufactured. The only down side I can see is that you would not have a DRAC/iLO solution built in, but then again, you need a license for that as well J.
I’m sure I’m missing some things here so feel free to point them out to me in the given context of this article, namely virtualized data centers!

 

One Comment Filed Under [ Platforms ]
The virtualized data center - Part One: Overview
There used to be a time where the whole concept of a data center was limited to (very) large companies due to the cost and housing, not even to mention the cooling and electricity required, of all those machines humming along in giant, underground lair. It was a huge undertaking that got a bit simpler with the rise of VMware and their virtualization techniques, but still required a massive amount of resources.
Now that different players have come to the market, all with their own possibilities, things are looking up for smaller companies which want to save costs on various fronts or are simply looking into building their own underground lair of a data center.
But if we look at building this, how would we go to realise such an undertaking technically? Not to mention business wise?
For now I’ll leave the business concept out of it. There’s a lot of information on the interwebz about this and Microsoft itself has a number of processes available on the subject (and many more):
·         http://www.technet.com
When we plan to build a data center we have to think about a number of things:
·         Hardware for the host servers
·         Networking requirements
·         Storage requirements
·         Fault tolerance
·         Software infrastructure
·         Management approach
And of course: what is this going to cost?
Out of all the different ways of designing and implementing a (virtual) data center the budget you have available will determine most of the route you can follow in relation to the route you would want to follow. I can imagine that we all would want to setup a fully redundant, highly performing, playground solution  to our business needs, but sometimes we have to deploy a less than perfect infrastructure to be able to get this sold to the business.
Add Comment Filed Under [ Platforms ]
A guideline to migrations: “All hands on deck!”
Welcome to this second instalment of the series! This series of articles is aimed at giving you a hold and a sort of “Best practices” to starting any kind of migrations. Whilst each technology or project has its quirks and traps most of the process you’ll need to go through is, roughly, the same.
The second article will describe a so called “test run” or, as I like to call it, practicing with duds... Seeing as running a test system in the ground is not quite as disruptive as destroying an actual production system I favour having a test run first J. Additionally a number of tips for your actual migration are given.
Step One: “Load the cannons!”
Well you should be prepared at this point to perform a test run of the migration process you have outlined. Whilst I know that it is not in the capabilities of a large number of engineers, this is the moment to put your testing environment to good use.
As you have undoubtedly figured out I want you to make a test run, make it 3 if you can... If you have no issues during these runs I salute you! But it will be more likely  you have either missed a step or have a non-identical environment in your test beds in regards to your production environment.
If you do run into problems (hooray!) this is your opportunity to document them and look at the best course of actions to solve. Remember, a prepared engineer is a ninja!
Step Two: “Tactical retreat”
Make sure that you have that rollback plan tested as well. Remember that, if things go wrong, you want to be able to avoid having shit hit the fan... Especially if you are in the path of said fan! I won’t disagree with you that you’ll be spending a lot more time preparing for migrations then actually performing them, a major downside of working in the field you’re in I suppose...
Make sure you have some backups ready (and tested :@) juuuuuusst in case that roll back plan failed to work. I never needed to use them, but they present another safe net in case the first one breaks...
Last but not least, since you informed everyone there is going to be a migration you should have a phone list handy of the people you “might” need to reach in case of trouble... You did inform everyone, didn’t you?
Step Tree: “Ready! Steady!”
Once you had your experimental test runs in which you hopefully ran into some issues (you know, to keep it interesting [and be prepared of course!]) you can look at doing some first migration actions.
Now while at this point you could just go ahead with the migration you might want to take a step back and decide whether or not you are ready to torment your production environment with making that big move. Review the issues you had and if you were able to resolve them properly.
Nobody likes to pass up an opportunity to torment some users though keep in mind that they, in most cases, will go annoy your manager (and as you undoubtedly know, shit goes downhill...), not  to mention you might lose a lot of game/movie/chill-out time if you don’t conclude that migration in your allotted timeframe!
Step Four: “Fire!”
Now the big moment arrives, you are taking the plunge in the deep and doing the move. Stick to the plan, if possible, seclude yourself from the rest of the world so you can focus on the work at hand. Remember it is important to have food at hand (large pizzas) and loads of fluids (stay away from alcohol, keep that until you are done), some mood music might help you along...
Being mentally prepared before you start is another important aspect. No heavy drinking or partying the night before I’m afraid. Your “little grey cells” need their beauty sleep to function at their peak performance!
Step Four: “I just can’t do it C’ptain!”
If you are running into problems you did not encounter during the test run or even expected (some pesky developer application interfacing with active directory and annoying the crap out of you) keep in mind that stress is your fuel. But don’t be afraid to stop for 5 minutes and take a small break.
Determine where the problem lies and contact the application owner responsible for your woes (revenge is best taken when they least expect it). Solve the problem together and proceed with the plan, but as always, document what happened and how you solved it (and who was responsible for that particular mess).
Step Five: “The aftermath”
Once you were able to run the gauntlet and reach that glorious finish line, having your servers humming along perfectly in sync you should try and work out the documentation you made. Knowing me, that is a couple of hours work as I use a notebook with a not so nice handwriting to keep track of things. Working this out is purely for esthetical reasons. Namely to present a nice report to your manager! We have to try and keep them involved now don’t we?
For the next week or two, try to keep some pro-active monitoring on the migrated systems, just to keep sure everything is working as it is supposed to. Life is hard enough as an IT professional as it is without some service deciding to go AWOL and screw up your life...
Step 6: “Bring forth the rum!”
This is where you can actually enjoy the victory. If all went well you should have a bunch of happy users and a happy manager. Don’t expect praise from anyone but your peers (well, maybe your manager... Maybe!) on your exceptional handling of the situation.
 
 
A guideline to migrations: “The Beginning”
Welcome to this first instalment of the series! This series of articles is aimed at giving you a hold and a sort of “Best practices” to starting any kind of migrations. Whilst each technology or project has its quirks and traps most of the process you’ll need to go through is, roughly, the same.
This first article will describe what’s called the preparation process. Although most  IT professionals have the tendency to just into thing (or “forget about the manual, I’ll make it up as I go along!) it pays to actually prepare for your migrations, transitions or even deployments of new systems, decreasing headaches and issues during the process as well as getting less trouble from the management (I had to learn this the hard way J).
Step One: “Why?”
The most common thing you’ll run into is the question “Why do we need to do this?” so make sure you have your arguments ready for this. Think in terms of the benefits for the business as that always goes better then “Because this version is waaayyy cooler”. This, generally, only applies to people who are working internally and have not been contracted or tasked with performing the migration.
Step Two: “Know Thy System!”
I can’t stress this enough! Knowledge is power! So if you want to make the world a better place make sure you know the ins and outs of the system you are planning on upgrading! Document it if it hasn’t been done because you’ll see that you’re going to forget that one little “this doesn’t matter” setting you’ll need to set on your new system...
You can use the “Best Practice Analyzer” toolsets from Microsoft to help you collect your data and see what warnings or even errors the system you’re looking at is throwing... Available for download here:
http://www.microsoft.com/download/en/search.aspx?q=Best%20Practice%20analyzer
Step Three: “Does this work?”
Determining if your current system is currently up to date with software updates or even just running well is an important consideration as at one point you’ll have to know whether or not you can use it as a fall-back plan in case something goes wrong. Obviously we’d rather avoid having to do that but you would want a backup parachute when jumping out of an aircraft no?
Tackle the issues you can and make sure your system(s) are up to date
Step Four: “Plausible deniability”
Decide on a timeframe where you can migrate the system, limiting impact on the business (so no user or data migration during business hours, thank you very much!) and then communicate this to your management and users! Extensively! Repeat if a number of times so nobody actually “forgets” that it is going to happen... Try to have the first communication about it go out to the users around 3 weeks before it happens. Then, at minimum, send it again a week before and a day before the migration happen. Make sure your users are aware of the possible issues they could run into during the time you are performing the migration.
Sure, this is a huge hassle and effort for something that should be transparent to your users BUT you would not want to run into a situation where an issue arises and causes a productivity loss in, oooh let’s say, the closing of the books...
Step Five: “You know what you’re doing, right?”
Determine if you are comfortable enough with the new technology and the migration process to actually perform the “operation”. After all, you can always hire a consultant to help you plan or, god forbid, even help you during the migration! If you have never ever performed a migration before you are in for a treat because they never go exactly as planned and you’ll have to be able to think fast and make the right choice in order to successfully overcome the obstacles presented...
This involves a lot of reading and, if you’re lucky enough, to test the migration in a lab environment. Not a lot of small (or even large) IT environments actually have a lab environment, much to my frustration, but if you do capitalise on it! Being able to perform a migration in a lab environment at least once (preferably 3 times) will add to your confidence and will show you what exactly you’ll encounter during the migration itself.
Use Microsoft Technet (http://technet.microsoft.com/) to prepare and find the guides on how to perform the migration your technology. Do note that Microsoft often uses the word “transition” instead of ‘”migrate” in its documentation, so if you can’t find your document right away, check your search parameters!
Conclusion
This concluded the first instalment of the series. Covering your bases is important as an IT professional and even more so if you’re about to embark on the wonderful voyage of migration (or transitioning) from an obsolete technology to the crispy, shiny newest and brightest to make your business more efficient and a nicer place to work.
Your comments are welcome and most appreciated!
/upgradecms failed!
An error that happens frequently when trying to upgrade your 2007
CCR or other form of the cluster is represented by the following error prompts:

Move-ClusteredMailboxServer: Database Recovery Storage Group/Mailbox Database is not in the final expected state: final state is Failed, expected Online.

Cluster Common Failure Exception: The cluster resource coudl not be brought online by the resource monitor. 0x8007139A

setup.com /upgradecms failing with error code : "0xc00713a0" "1002"
-------------------------------------------------------------------------------------------------

[TIMESTAMP] [2] ScSetNetworkNameOnResource (f:\08.02.0176\sources\dev\admin\src\libs\exsetup\cluresources.cxx:552)

Error code 0XC00713A0 (5024): The properties were stored but not all changes
will take effect until the next time the resource is brought online.

[TIMESTAMP] [2] Leaving ScSetNetworkNameOnResource

[TIMESTAMP]] [2] ScSetNetworkNameOnResources
(f:\08.02.0176\sources\dev\admin\src\libs\exsetup\cluresources.cxx:1864)

Error code 0XC00713A0 (5024): The properties were stored but not all changes
will take effect until the next time the resource is brought online.

[TIMESTAMP] [2] Leaving ScSetNetworkNameOnResources

[TIMESTAMP] [2] ScUpdateExchangeClusterResources
(f:\08.02.0176\sources\dev\admin\src\libs\exsetup\cluresources.cxx:2032)

Error code 0XC00713A0 (5024): The properties were stored but not all changes
will take effect until the next time the resource is brought online.

[TIMESTAMP] [2] Leaving ScUpdateExchangeClusterResources

[TIMESTAMP]] [2] ScSetupExchangeVirtualServer (f:\08.02.0176\sources\dev\admin\src\udog\exsetdata\exsetds.cxx:2013)

Error code 0XC00713A0 (5024): The properties were stored but not all changes
will take effect until the next time the resource is brought online.

[TIMESTAMP] [2] Leaving ScSetupExchangeVirtualServer

[TIMESTAMP] [2] [ERROR] Unexpected Error

[TIMESTAMP]] [2] [ERROR] Error The properties were stored but not all changes
will take effect until the next time the resource is brought online occured
while performing exsetdata operation; the original error code was 0xc00713a0.

[TIMESTAMP] [2] Ending processing.

[TIMESTAMP] [1] The following 1 error(s) occurred during task execution:

[TIMESTAMP] [1] 0. ErrorRecord: Error The properties were stored but not
all changes will take effect until the next time the resource is brought online
occured while performing exsetdata operation; the original error code was
0xc00713a0.

[TIMESTAMP]] [1] 0. ErrorRecord: Microsoft.Exchange.Management.Tasks.ExsetdataKnownException:
Error The properties were stored but not all changes will take effect until the
next time the resource is brought online occured while performing exsetdata
operation; the original error code was 0xc00713a0.
 
[TIMESTAMP] [1] [ERROR] Error The properties were stored but not
all changes will take effect until the next time the resource is brought online
occured while performing exsetdata operation; the original error code was
0xc00713a0.

[TIMESTAMP] [1] Setup is halting task execution because of one or more errors
in a critical task.

[TIMESTAMP] [1] Finished executing component tasks.

[TIMESTAMP] [1] Ending processing.

[TIMESTAMP] [0] The Exchange Server Setup operation did not complete.
For more information, visit
http://support.microsoft.com and enter
the Error ID.

[TIMESTAMP] [0] End of Setup

------------------------------------------------------------------------------------------------
In order to solve this error the following actions should be
taken:
 
1.    Remove the RSG from the cluster
2.    Check for the watermark keys in HKLM\SOFTWARE\Microsoft\Exchange\v8.0\AdminTools and delete them after taking a backup of the register
3.    Reboot each node in the cluster
4.    Run the setup.com /upgradecms command
 
Add Comment Filed Under [ Exchange ]
Article: Recreating the public folder hierarchy

When the EMC (exchange management console) is launched, it returns the following error:

--------------------------------------------------------
Microsoft Exchange Warning
--------------------------------------------------------
The following warning(s) were reported while loading topology information:

get-PublicFolderDatabase
Completed

Warning:
Object AAA-EX01\Second Storage Group\Public Folder Database has been corrupted and it is in an inconsistent state. The following validation errors have occurred:

Warning:
PublicFolderHierarchy is mandatory.

Warning:
PublicFolderHierarchy is mandatory.


--------------------------------------------------------
OK
--------------------------------------------------------

Trying to mount the Public folder database gets you the following eror:

--------------------------------------------------------
Microsoft Exchange Error
--------------------------------------------------------
Failed to mount database 'Public Folder Database'.

Public Folder Database
Failed
Error:
Exchange is unable to mount the database that you specified. Specified database: UDC-EX01\Second Storage Group\Public Folder Database; Error code: MapiExceptionADPropertyError: Unable to mount database. (hr=0x80004005, ec=2418)

--------------------------------------------------------
OK
--------------------------------------------------------

When trying to delete the public folder database:

--------------------------------------------------------
Microsoft Exchange Error
--------------------------------------------------------
The public folder database 'Public Folder Database' cannot be deleted.

Public Folder Database
Failed
Error:
The critical property 'PublicFolderHierarchy' is missing in the PublicFolderDatabase object 'UDC-EX01\Second Storage Group\Public Folder Database'.

--------------------------------------------------------
OK
--------------------------------------------------------

This all happens after you deleted the "First administrative group" in ADSIedit (which you should not have done!) after a migration to Exchange 2007. If these are the only problems you are facing be happy. To resolve it perform the following actions:

1. Open ADSIedit.exe.

2. Right click on Exchange Administrative Group (FYDIBOHF23SPDLT)
3. Select New Object.
4. Select msExchPublicFolderTreeContainer for the class and click Next
5. Enter the following for the value: Folder Hierarchies, click Next
6. Click Finish

Create Public Folder Tree Object

1. Right click CN=Folder Hierarchies -> New Object
2. Selected msExchPFTree for the class
3. For the value we entered, "Public Folders" and clicked next
4. Clicked on the "More Attributes" button, selected msExchPFTreeType and set the value to 1.


Note: This is very important that this value is set to a value of 1 as this tells Exchange that this is a MAPI Tree
5. Click Ok and then finish


Populate msExchOwningPFTreeBL attribute object of the PF Stores in the organization

1. Get properties of the newly created "Public Folders" Tree object in ADSIEdit.
2. Copy the distinguishedname value to the clipboard and then click cancel.
3. Browse to CN=Public Folders Database,CN=Second Storage Group,CN=InformationStore CN="Sever Name:,CN=Servers,CN=Exchange Administative Group,CN=Administrative Groups,CN=ORG Name,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=Domain,DC=COM
4. In the properties of the "Public Folder Database", Edit the msExchOwningPFtree attribute and paste the content copied to the clipboard in Step2
5. Restart the Information Store Service.

 

*Update*

If you read the comments you'll see that Tim had a bit of an issue that was very similar to what was described here but not quite like it. Martina on the technet social helped him to resolve his issue by suggesting to check the msExchOwningPFTree attribute in ADSIedit! Have a look at the thread for the full details

 

http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/ac69d2f2-b8c4-40ab-9466-c6727da87fc7

36 Comments Filed Under [ Exchange ]
Article: ADAM Troubleshooting

Initial troubleshooting

As always, one of the first things to check is the event viewer to see if an event was generated detailing the error. Additionally check the %windir%\debug for the adamsetup.log and adamuninstall.log (this last one is only created during the uninstall process). These two logs will tell you where the setup is failing and what should be checked.

It also pays to know that setup errors are written to the registry. If you cannot find the following key there was no failure as the keys are only generated if there was a failure and they are removed after a successful installation.

Registry keys:

HKLM\Software\Microsoft\Windows\CurrentVersion\ADAM_Installer_Results

Specific troubleshooting

ERROR_LOGON_FAILURE When Trying to Bind

 If the computer is a member of a workgroup and not a domain, verify that the following registry value is 0 and reboot the machine before attempting to run setup again.

HKLM\Control\CurrentControlSet\Control\LSA\forceguest

Error: 0x800706fd The trust relationship between this workstation and the primary domain failed

When you are installing ADAM when not connected to the domain, check if you are trying to install the ADAM service with the Network Service (NetworkService account). If so you will need to connect to the domain to allow this account to resolve or choose a local account for the ADAM service account.

Error:  ADAM Setup could not complete because shortcuts could not be added to the start menu

Delete the following registry key:

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\ADAM_Shared

Error: The name referenced is invalid

When you add an ADAM user to the administrator group of the schema or configuration container you get the error "The name referenced is invalid." This error is by design. An ADAM user cannot be an administrator of the whole instance. Users are not allowed in the Configuration container and groups cannot have cross-NC membership.

Troubleshooting the Addition of Partitions, Users, Groups, and OUs

Unable to Create a Partition

This can be done during setup, but if it was not done at that time, you will have to create the partition via DSMGMT or LDP. You must be logged on with the credentials that were used to create the ADAM instance. This account became the ADAM administrator when the instance was created. Below is an example of how to do this in DSMGMT.

1. Open the ADAM command prompt.
2. Type dsmgmt.
3. Type partition management.
4. Type connections.
5. Type connect to server Where servername is the name or IP address of the server and the port of the ADAM instance.
6. Type quit.

7. Type list to list the existing partitions. Partitions cannot have the same names even if the DN type is different. The following DN types are supported C,CN,DC,L,O,OU.
8. To create a new application partition type in create NC %1 %2 %3  where %1 is the DN of the partition, %2 is the objectclass , %3 is the server:port number or type in NULL for the currently selected instance.

Cannot Add Replica Partition

This can be done with Dsmgmt also. Do this on the machine that you want to hold the new replica partition. Follow Steps 1 through 6 above for adding a partition, then for Step 7 run the following command:

Add NC replica 

Error:  AD/AM create application directory failed with error 64 (Naming Violation)

This can happen if you have the objectclass of container with a DC=domain,DC=com  style partition. This objectclass is domainDNS.

Error:  AD/AM create application directory failed with error 53 (Unwilling To Perform)

One possible cause is if the objectclass is domain instead of domainDNS.

Error:  AD/AM create application directory failed with error 16 (No Such Attribute)

This can happen if you choose an objectclass that does not exist. Here is a list of the types of objects and the objectclass. The first name in the DN is the objectclass that you use

DC = domainDNS

O  = Organization

CN = Container

C   =  Country

L   = Locality

OU= OrganizationalUnit


Error: ldap_addW failed with 0x33(51 (Busy) Ldap extended error message is 0000200E: SvcErr: DSID-0206013A, problem 5001 (BUSY), data -1605 Win32 error returned is 0x200e(The directory service is busy.)

If you get this error exit out of Dsmgmt and go back in. This can occur after you try to create a partition and it fails.

Error: ldap_addW failed with 0x44(68 (Already Exists) Ldap extended error message is 00002071: UpdErr: DSID-0315232B, problem 6005 (ENTRY_EXISTS), data 0 Win32 error returned is 0x2071(An attempt was made to add an object to the directory with a name that is already in use.)

You cannot create a partition with the same name but a different types. This is not allowed.

Error:  I cannot add any users to my ADAM instance 

If the schema extensions were not added during setup you will need to add these with ldifde before you can add users to your ADAM instance. These are stored in the %WinDir%\ADAM folder by default.

Error:  I cannot add ADAM users to the admins group for the ADAM instance "the name reference is invalid"

This is by design. ADAM users cannot be administrators of the instance and they cannot be added to the configuration container. Only the ADAM administrators can do this.

Error:  When I try and add a group to ADAM it is asking me for Value? 

For this you must enter 2147483650 for global group or 2147483656 for universal Group. Since ADAM does not have a global catalog or domains, it does not matter which type is used.

Error:  The option to Add an OU is not there?

OUs can only be created under the following type of namespaces by default DC, O, C, and OU. If you want to change this behavior you will have to add the container that you want to the possSuperiors attribute of the organizational unit in the schema.

Error:  On Windows XP ADSI code to retrieve ntSecurityDescriptor results in an ERROR_NONE_MAPPED: No mapping between account names and security IDs was done.

This issue is resolved with the following hotfix:

817583 Active Directory Services does not request secure authorization over an SSL connection

Troubleshooting Replication

 ADAM not Replicating

Since ADAM is based on the active directory basic troubleshooting is the same. In order for the directory to replicate we must have name resolution, physical connectivity and the correct credentials to authenticate to the machine ADAM is running on.

Troubleshooting steps

1. Look at the Event log for that instance, look for replication or KCC errors.
2. Is the machine and its replication partners in a domain, workgroup, separate forests.
3 .If the machine is XP and it is in a workgroup, the following registry key must be changed to zero and the machine rebooted

HKLM\Control\CurrentControlSet\Control\LSA\forceguest

4.Use ADAM Adsiedit to connect to see which value is set for the attribute msDS-ReplAuthenticationMode in the root of the Configuration container:

A - ADAM Service accounts must be using the same name and password. Machines in a workgroup must use this value for replication to work.

B - Kerberos with failover to NTLM. This is the default setting if the machine ADAM is installed on is a domain member.

C - Kerberos only, no failover to NTLM.

As name resolution is required for replication to work DNS, NETBIOS, WINS, network broadcasts or correct entries in the HOST file are needed. Note that only host records in the DNS service are used.

Network connectivity

Required ports:

1. 389 TCP (LDAP) or TCP 686 (LDAPS) (these can vary if you are using a different port number for your ADAM instance)
2. 88 TCP/UDP (Kerberos)
3. 53 TCP/UDP (DNS)
4. 445 TCP/UDP (SMB over IP traffic)

Service Principal Names

SPNs are generated when ADAM is installed and updated, when the service starts and are created as an attribute on the User account that is running the ADAM service. If it is running under network service they get created as an attribute of the computer object.  If they are not created you will receive an Event ID 2516. This event will tell you what object it tried to create them under and why it failed. You will also get an Event ID 2519 that will give you a script and its location. This script will be using repadmin /writespn to manually add the SPNs.

Check for repadmin errors by running:

1. repadmin /showrepl server:port
2. repadmin /showutdvec (shows end to end replication from the perspective of a single DSA)
3. dsdiag /v /s:server:port

ADAM Service Discovery

Service Connection Points (SCP) objects are created under the machine that hosts the ADAM service. They are created or updated when the service starts and require the ADAM service account to have Create Child rights on the computer object. If the SCP cannot be created you will receive an Event ID 2537 that will describe why it could not be created.

Note that SCPs are not required and the creation of these can be disabled.

Troubleshooting Authentication Security and Certificates

Application Unable to Authenticate with ADAM

1. Verify a user can authenticate to ADAM via LDP using the server name and port number.

2. If ADAM is running on Windows XP, verify the following registry value is set to 0:

HKLM\System\CCS\Control\LSA\forceguest

3. By default anonymous binds are disabled, so an application attempting them will fail. To enable anonymous LDAP operations in ADAM, you must set the seventh character of the dsHeuristics value to 2.

4. Verify the ADAM service is running and check the event log for errors.

5. Verify what type of user is involved - ADAM User, proxy User, local user, or Windows security principal.

6. If a proxy user or Windows security principal is being used, verify that a domain is available. Verify there is a valid secure channel with the domain for the ADAM server. Verify network access, name resolution, DNS to a domain controller. Is there a domain controller available? Can the user logon to a workstation? Is replication both ADAM and AD working (repadmin). Basic workstation/logon troubleshooting applies here.

7. If the user is an ADAM user, a simple bind is used and must be done over SSL, since the password is sent in plain text.

8. Is the ADAM user account locked out or disabled: Check the attribute on the user object msDs-userpassworexpired, msDS-UserAccountAutoLocked or msDS-UserAccountDisabled. This will default to true if you have a password policy enabled and the password is blank or does not meet the password policy requirements.

9. Are we connecting over SSL? If so can you connect over normal LDAP? Check the certificates (see the next issue).

Cannot Bind to ADAM over SSL

1. By default password changes in ADAM must be over SSL, but to do SSL we need a certificate From a Certificate Server CA, or a third party Certificate.

2. Request a server certificate for the Windows machine hosting the ADAM instance. Use the FQDN of the machine for the name of the certificate.  Make sure to check the box to allow it to be exportable to the machine store.

3. Check to ensure the certificate was properly installed.  Via the Certificates MMC snap-in for the computer account.

4. Allowing ADAM to use the server certificate, by adding it to the ADAM service "My store" or place it in the machine personal store and change permissions so that the ADAM service can read it. To give the ADAM service account permission to the machine certificate. Read and execute must be given to the file with the latest time stamp in the following location:

Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

5. Set up the client to trust the rootCA and certificate path of the CA that issued the server certificate. Do this through the CA website.  Export the CA certificate and certification path.  Import these into the Trusted Root Store in the Certificates MMC snap-in.

SASL Bind for ADAM Security Principal

Simple LDAP binds are sent in plain text, which is why SSL should be used for security. Simple binds are the only way to bind to ADAM for an ADAM security principal. SASL binds (using Kerberos, NTLM, or Negotiate) are used by local or domain Windows security principals. Bind redirection for ADAM proxy objects use simple LDAP binds to ADAM and then a SASL bind to Active Directory to authenticate the user.

Unable to See Objects after Binding to ADAM

Is the ADAM user a member of the Readers Built in Group? By default ADAM users are placed in the Users Group which does not have any read permissions to the partition.

Unable to Bind to ADAM with an Active Directory Account or Bind Redirection with LDP

1. On the Connection menu, click Connect, and then connect to your ADAM instance on a new connection.

2. On the Options menu, click Connection Options.

3. In Option Name, in Value click LDAP_OPT_SIGN (enables/disables Kerberos signing prior to binding using the LDAP_AUTH_NEGOTIATE flag), type 1, and then click Set.

4. In Option Name, in Value click LDAP_OPT_ENCRYPT (enables/disables Kerberos encryption prior to binding using the LDAP_AUTH_NEGOTIATE flag) type 1, click Set, and then click Close. Note this does not work on Windows XP.

5. Bind to your ADAM instance with LDP by clicking Bind on the Connection menu.

6. In User, type in the distinguished name (DN) of the proxy object.

7. Make sure the Domain option is not selected.

8. In Password, type the password that is associated with the Active Directory user you specified.

Using a Different Security Principal Other Than User, Person or inetOrgPerson

Any object can be a security principal by adding the msDS-bindableobject auxiliary class and the unicodePwd attribute to the schema definition of the object class in the ADAM schema.

Using Network Load Balancing with ADAM

Follow the steps above and ensure that LDAPS is working by by binding to LDP using SSL. If this works, proceed with binding a wildcard certificate.

Unable to Use Basic Authentication with IIS to Authenticate ADAM users

By default IIS cannot use ADAM as its primary authentication for ASP.NET pages. A forms authentication mechanism that uses the ADAM instance for user verification must be used.

Outlook or Windows Address Book Failure to Logon to ADAM with error: "The specified directory service has denied access. Check the Properties for this directory service and verify that your Authentication Type settings and parameters are correct."

The client software is configured to logon with the simple name not the distinguished name.

No Security Tab in ADAM Adsiedit

All security setting within ADAM must be done through DSACLS, LDP, or using a script.

Storing Application Policies for Authorization Manager with ADAM

For this to work you must first install the AZMAN schema extension then use a tool such as ADAM-ADSIedit to create a container to hold the application policy store.

1. In AZMAN, right-click the root Authorization Manager node in the tree view and select New Authorization Store.

2. Select Active Directory as the store type and specify the LDAP distinguished name (DN) of the store object to be created or managed specifying the ADAM server name and LDAP port as follows:

servername:/cn=,CN=

Obtaining an Object Identifier

http://msdn.microsoft.com/en-us/library/ms677621(VS.85).aspx

Using the Unique GUID for an ADAM Instance to Modify the Schema or Configuration Container

It is not necessary to use the unique GUID for an ADAM instance to modify the schema or configuration container. The ADAM version of Ldifde allows you to use the #schemaNamingContext and #configurationNamingContext variables for this purpose.

Error Importing LDIF File: Add error on line 1: No Such Attribute The server side error is "The parameter is incorrect." 0 entries modified successfully. An error has occurred in the program

Make sure you are using the ADAM version of LDIFDE, which is located in %windir%\ADAM by default.

Error Importing Users: Add error on line 2: Unwilling To Perform The server side error is "The modification was not permitted for security reasons."

 

One Comment Filed Under [ Platforms ]
Article: Troubleshooting the information store service

Alot of calls I get are related to the information store service of the exchange server not starting. Since troubleshooting why this services does not start is often related to alot of stress (after all, your users and managers will be pounding your door as they cannot access their email) it is always handy to have some reference as to where you can start and what direction it can take.

This is my basic modus operandi so feel free to give suggestions on how to improve it :).

First of all you need to know the 3 major reasons what can cause the IS service to be down:
+ Database problems
+ Active Directory problems
+ The antivirus software is acting up.

I start out simple, I try to eliminate one of the causes.

1. Open Exchange Management Console (or the Exchange System Manager if you're on 2003)
2. Expand untill you reach the database
3. Open the properties of the database
4. Check "Do not mount this database on startup"
5. Click ok
6. Open services.msc

1. Open Start, run
2. Type in services.msc, click "OK"
3. Scroll down untill you reach the Microsoft Exchange Information Store Service.
4. Right click the IS service and try to start it.
5. Does it start?



If the IS service mounts at this point you're most likely going to have a corrupt database. open a command prompt and run the "ESEUTIL /mh priv1.edb" command. Scroll down untill you see the "State" and "Log required" Field:

     
335454
     

If you have the database state on "Dirty Shutdown" you'll need to run the following commands on the database:

1. Eseutil /p
2. Eseutil /d
3. ISInteg -s "servername" -test alltest -fix

Follow the on screen instructions for the ISInteg and repeat ISInteg untill all errors have been corrected. This is extremely important as ISInteg fixes the database tables and will either fix or get rid of corrupt items.

Note: ISINTEG is currently not available for Exchange 2010. You can skip step 3 there as 2010 has a self-healing system built in. SP1 for 2010 is expected to have a new ISInteg version.

Depending on how big your database is it might take a while to complete the database recovery. If you need to get your users back online fast you can use the Dialtone recovery method. This means you'll move all the files in the physical location of the database where you can perform the recovery and mount the database in ESM or EMC. It will tell you that it could not find a database and ask you if it can create a new (blank) database. If you confirm a new database can be mounted and users can access new emails that are recieved if they are in online mode and access their old mails only if they have the cached mode enabled.

More information on Dialtone recovery:
MSExchange.org Part 1
MSExchange.org Part 2
MSExchange.org Part 3

Now, in case the above did not get your service to start up you have reached a pickle. We need to find out if it's an AD or AV issue!

1. Open Start, run
2. Type in services.msc, click "OK"
3. Scroll down untill you reach the Microsoft Exchange services.
4. Note what services are down. Is only the IS service not functional or is the transport service down as well?

Note:
If you're transport service is down as well the likelyhood of it being an Active Directory issue increases!

Try starting the IS Service, it will error out but what is important is that you will now have some events logged in the application log. In most cases these events will be ID 5000 and 1121.

Going into the event log:

1. Open start, run
2. Type in eventvwr, click "OK"
3. expand untill you hit the "Application log"
4. Identify the recent events from source "MSExchangeIS"
5. Also have a look at the events from source "ADAcces"

If events 5000 and 1121 are logged they should point you in the right direction for what is wrong with the AD. Usually it's Exchange that cannot contact the GC. In that case there's a quick and dirty workaround. Note that you should only do this to restore functionality for your environment and it is a temporary measure. After you repair the AD issues you are highly advised to let Exchange choose it's DC/GC!

+ For Exchange 2003:
1. Open the Exchange System Manager
2. Expand untill you hit your exchange server
3. Open the properties of the exchange server
4. Switch to the "Directory Access" tab
5. Select "Domain Controllers" in the drop down list
6. Select a working DC
7. Deselect "Automatically discover Servers"

Note: If your Exchange server is installed on a Dc it will always contact that DC, no matter what you set in this feel.

For Exchange 2007:

1. Open the exchange management shell (powershell for exchange)
2. Use the Set-Exchangeserver -StaticConfigDomainController -StaticDomainController -StaticGlobalCatalog command

Note:
If your Exchange server is installed on a Dc it will always contact that DC, no matter what you set in this field. For Exchange 2010 you can use the same command as for Exchange 2007.

In case there are no events 5000 & 1121 you'll most likely have events 9565 & 9564 logged. These are caused by the antivirus program being broken. You'll want to disable the antivirus key in the registry:

1.Click Start, and then click Run.
2.In the Open box, type regedit, and then click OK.
3.In Registry Editor, locate the following subkey in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\VirusScan
4.In the right pane, double-click Enabled.
5.Click Decimal, type 0 in the Value data box, and then click OK.
6.On the File menu, click Exit to quit Registry Editor.
7.Start the Information Store.
http://support.microsoft.com/kb/323664

Deinstall your AV in a service window and reboot the server.

Add Comment Filed Under [ Exchange ]
Disclaimer

The opinions or ramblings in this blog are my own and represent, in no way, those of my employer or any other person. All content is written by myself unless otherwise specified. Common sense applies when performing actions recommended on this blog. I take no liability for any sort of data loss or injuries sustained during IT work.

Do not use microwave to dry pooch. Do not spray directly in eyes. Do not take orally. Banging your head in to the wall may cause head trauma. Ninja’s might follow you after reading this blog.