Cajun MCSE

Windows 2008 R2 and Windows 7 SP1 released to manufacturers

Ryan Roussel — Fri, 11 Feb 2011 09:15:00 GMT

SP1 went RTM today which means it will be widely available soon. According to the Microsoft server division blog, the service pack will be available to Microsoft's TechNet and MSDN subscribers, as well as to Microsoft volume licensing customers, on Feb. 16, 2011. It will be generally available via the Microsoft Download Center and Windows Update on Feb. 22

You can see the blog and news here:
http://blogs.technet.com/b/windowsserver/archive/2011/02/08/windows-server-2008-r2-and-windows-7-sp1-releases-to-manufacturing-today.aspx

New features in SP1 include dynamic memory for Hyper-V VMs and RemoteFX which enables rich desktop content for Virtual Desktop Infrastructure.

Another Exchange 2003 to Exchange 2010 mail flow issue

Ryan Roussel — Wed, 22 Dec 2010 22:34:00 GMT

During a migration recently, we came across another internal mail routing issue. The symptoms were identical to my previous post about Exchange internal mail routing. Mail was flowing from 2010 to 2003, from 2010 to the internet, but not from 2003 to 2010. I went through the normal check list looking at permissions, DNS, and the routing group connectors. I verified that both servers listed in the routing group connectors were the routing master in their respective routing groups through the 2003 ESM. I also verified that inheritable permissions were enabled for the Exchange 2003 server object in the schema. No luck with either.

For my previous post about this issue in which inheritable permissions were the culprit:
Exchange 2010, Exchange 2003 Mail Flow issue

And for Routing Group issues:
Exchange 2007 Routing Group Connector Mayhem

I finally enabled logging on the SMTP virtual server on Exchange 2003 and the Default Receive Connector on 2010 and sent a few test e-mails where I found 2003 was having issues authenticating to 2010. By default 2003 uses Exchange Server Authentication to communicate to 2010. The exact error was:

4.7.0 Temporary Authentication Failure

which was found in the SMTP logs on the Exchange 2003 side

After scouring based on this error, I found the solution: The Access this computer from the network user rights in the local computer policy on the Exchange 2010 server were changed from the default. The network administrator had modified the Default Domain policy and changed this user right assignment to only list Domain Users. The fix was to clear this setting in the Default Domain policy, force gpupdate to refresh the group policy settings, then ensure the appropriate users and groups were listed.

This immediately fixed the problem and the Exchange 2003 server was able to route mail to the Exchange 2010 mailboxes.

The default user rights assignments for Access this computer from the network

On Workstations and Servers:

Administrators

Backup Operators

Power Users

Users

Everyone

On Domain Controllers:

Administrators

Authenticated Users

Everyone

More can be found here: http://technet.microsoft.com/en-us/library/cc740196(WS.10).aspx

Invalid attributes while converting mail-enabled groups

Ryan Roussel — Thu, 25 Nov 2010 01:34:07 GMT

During a recent Exchange 2010 migration, we encountered issues with mail-enabled groups. At first we thought it was a SMTP related issue as no 2010 mailbox could send mail to any of the groups. The mail would just disappear. No NDR was delivered, it wasn’t in the queue, it simply vanished.

Because of this issue, we decided to go ahead and convert all the groups to Universal Mail-Enabled groups through the Exchange 2010 EMC. This is when we encountered a really strange error. The groups would not convert stating that the object had invalid characters in its SMTP address. After checking all the SMTP addresses, no invalid characters or malformed addresses could be found. It was actually stating a physical address was set as a SMTP address.

We opened ADSIEdit on the 2003 server and started scanning the attributes of the group object since this object was still a 2003 distribution list, again nothing could be found. We then thought about using the 2010 server for the ADSIedit since it was running on 2008 R2 and would show the new attributes added. Sure enough there was the attribute targetAddress populated with these physical addresses.

The fix was to clear this attribute across every mail-enabled group, then convert the group to an universal mail-enabled group. This also fixed the strange SMTP issue that we were initially experiencing.

NOTE: You can use a tool like ADModify.NET to perform bulk object attribute changes.

Download here: http://www.computerperformance.co.uk/w2k3/utilities/admodify.htm

Proxying OWA between CAS Arrays with Exchange 2007

Ryan Roussel — Wed, 24 Nov 2010 23:10:22 GMT

Recently for a customer with a rather large exchange environment, we implemented multiple CAS Arrays across various sites in the network. The customer decided that all external access to OWA would come into once Internet entry point and that Array would proxy OWA request to the other CAS Arrays to retrieve the user mailbox.

We found out quickly that this does not work straight off. When you create a new CAS array in PowerShell, it repopulates all the local URLs for the web services, autodiscover, and RPC client access point with the new CAS Array name. Normally this is ideal as you want all connections to use the virtual array name and IP for redundancy purposes. However for CAS proxying, the local URL field for OWA on the remote CAS array servers cannot be populated with the array name, and instead must be populated with the internal FQDN for the individual servers. The external URL fields were empty as required.

FYI: For proxying to work, the external URL for OWA must be left blank otherwise Exchange will offer the user another URL to try rather than proxy the request.

I went ahead and changed these fields back to the server FQDNs and proxying started working. The reason the server FQDN is required is because Exchange uses Kerberos to pass authentication from one CAS to another. This field is primarily to let other Exchange servers know how to access this server through the internal network. The server will still answer request for OWA on other names, so the array name does still work from a web browser if an internal users opens a browser and browses to the array name for OWA. Exchange 2010 overcomes this by using the InternalNLBBypass attribute on the Client Access Server.

NOTE: For further instruction on setting up Proxying between CAS servers:

Proxying OWA across CAS Servers in different Active Directory Sites

Exchange 2010 OWA and Public Folders

Ryan Roussel — Wed, 20 Oct 2010 00:40:13 GMT

I ran into a real head scratcher this week. While migrating a very public folder dependant organization, I couldn’t access public folders through OWA. I had full public folder functionality in the Outlook client, only OWA was affected. The environment was still in a hybrid state with production Exchange 2003 servers in the organization as well as DR servers which were using a third party replication package. The exact error I was getting looked like this:

I had already established all 2010 and 2003 mailbox servers as replicas of both the system folders and public folder hierarchy. I was also able to confirm replication was working by using:

[PS] C:\>PublicFolderStatistics –server servername

All my folders showed the correct number of items, so my content was replicating.

I also verified that my 2010 databases were pointed to the correct public folder database on the Client tab of the database properties.

NOTE: Exchange 2010 Outlook Web App cannot proxy to a 2003 public folder database and needs a replica on a 2010 server.

I also came across this article that had me check the replication limits set against the public folder databases on 2003 side. Again everything configured correctly.

After scouring the blogsphere for the better part of the day, I finally gave in, swallowed my pride, and called MS support. The MS engineer had me again check all the above. He also had me start checking the MsExchOwningPFTree attribute in ADSIEdit on the public folder databases on the 2003 side. The databases on my two production 2003 servers had this attribute set correctly. On a whim, I decided to check this attribute on the 2 DR servers as well. These servers aren’t really in production but are mimics of the production servers. However they do each show an unmounted public folder databases as far as Exchange is concerned. Sure enough, the DR Mimics’ public folder databases did not have the attribute set. After setting this attribute and forcing active directory replication everything functioned correctly.

The value of MsExchOwningPFTree should be set to the public folder hierarchy:

CN=Public Folders,CN=Folder Hierarchies,CN=Administrative Group,CN=Administrative Groups,CN=Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=local

Where Administrative Group and Organization are the names of your 2003 Administrative Group and your Exchange organization.

Exchange and Small Business Server

Ryan Roussel — Fri, 10 Sep 2010 04:57:21 GMT

We recently switched over a client’s small infrastructure to Small Business Server 2008. SBS 2008 has SQL 2008 and Exchange 2007 imbedded. The customer wanted to keep the same internal and external FQDN names used previously with Exchange 2003.

I had them purchase a SAN certificate with all the standard names in an Exchange 2007 or 2010 environment. We installed the certificate, made sure everything was set in both internal and external DNS, and started setting up clients.

Webmail and Active-Sync were fine after I went through and defined all the URLs in the CAS area under server configuration, however all the Outlook clients constantly popped certificate errors. This caught me off guard at first considering the server had a SAN certificate installed that contained the server NetBIOS name and internal FQDN in the certificate. Then I remembered the Autodiscover internal URL.

2007 and 2010 use the Autodiscover web services for many functions including client auto-configuration, free-busy data, and offline address book downloads. SBS by default uses remote.domain.com for a good bit of its “remote” functionally. Sure enough, when looking at the attributes of the CAS, the autodiscover service uri was set to remote.domain.com as well. The Exchange Web Service was set this way also. Both can only be changed through the Exchange Management Shell.

To Change the Autodiscover service URLs:

[PS] c:\>Set-ClientAccessServer –Identity servername
-AutoDiscoverServiceInternalUri https://server/autodiscover/autodiscover.xml

To Change the Autodiscover virtual directory URLs:

[PS] c:\>Set-AutodiscoverVirtualDirectory –Identity "Servername\autodiscover(Default Web Site)” –internalurl https://server/autodiscover/autodiscover.xml –externalurl https://webmail.domain.com/autodiscover/autodiscover.xml

To Change the Exchange Web Services virtual directory URLs:

[PS] c:\>Set-WebServicesVirtualDirectory –Identity “Servername\EWS (Default Web Site)” –InternalUrl https://server/EWS/Exchange.asmx –ExternalUrl https://webmail.domain.com/EWS/Exchange.asmx

The rest of the Web Directory URLs can be changed from inside the Exchange Management Console.

Windows 2008, Exchange, and TCP/IP Auto Tuning

Ryan Roussel — Fri, 20 Aug 2010 06:56:00 GMT

Microsoft introduced TCP/IP Auto tuning with Vista and Windows 2008 server. Its also a part of Windows 7 and 2008 R2. This feature resizes IP packets which can improve network performance. Older network devices, however, like Cisco PIX firewalls do not support Auto tuning and can cause connection issues.

I had a client that was getting email undeliverable errors from a certain domain while successfully sending to various others. After investigating OBL listings and various other possible culprits, the tech for the problematic domain suggested looking at this. My client has an Exchange 2007 environment installed on Windows 2008 SP2 servers. Sure enough after disabling this feature on all the hub transport servers, the SMTP traffic was no longer timing out to this domain and mail flow was established.

To disable TCP/IP Auto Tuning, run the following from an elevated command prompt:

netsh interface tcp set global autotuning=disabled

For more information on TCP/IP Auto Tuning, Visit:

http://blogs.msdn.com/b/wndp/archive/2006/05/05/winhec-blog-tcpip-2.aspx

Hyper-V Live Migration across Sites!

Ryan Roussel — Fri, 11 Jun 2010 14:09:08 GMT

One of the great sessions I sat in on at Tech Ed this week was stretching a Windows 2008 R2 Hyper-V Failover Cluster across sites. With this ability, you could actually implement a Hyper-V cluster where you could migrate or even Live Migrate VMs across sites. With this area’s propensity for Hurricanes, this will be a very popular topic for me over the next few months. While this technology is possible today, it’s also very complicated and can be very expensive to implement.

First your WAN connection has to support the ability to trunk your VLAN across both sites in order to Live Migrate. This means you can’t use a Layer 3 routed connection like MPLS. It has to be a Metro Ethernet connection or "Dark Fiber”. Dark Fiber is unused Fiber already in the ground that can be leased from various providers. Both of these connections would allow you to trunk layer 2 across your WAN. Cisco does have the ability to trunk layer 2 across a routed connection by muxing the traffic but this is only available in their Nexus product line which has a very steep price tag.

If you are stuck with MPLS or the like and Nexus switching is not a realistic possibility, you will have to implement a multi-subnet cluster in which case Live Migration won’t be possible. However you can still failover VMs to the remote site with some planning and manual intervention. The consideration here is that the VMs will be on a different subnet once migrated, so you will have to change the IP addressing of your VMs. This also has ramifications with DNS and Name resolution to control your down time. DHCP with Reservations for your VMs is the preferred method to achieve the IP changes as this will automate that part of the process.

Secondly, you will have to have a mechanism to replicate your storage across both sites. Many SAN vendors natively support hardware based synchronous and asynchronous replication. Some even support cluster shared volumes which were introduced in 2008 R2. If your SANs do not support this natively, there are alternative file based replication products either software based like Double Take or hardware appliance like EMC. Be sure to check with your vendor on the support of Disk majority if you’re replicating your quorum disk between SANs.

The last consideration is the ability to maintain quorum for your cluster. If your replication provider does not support Disk Majority through replication, you will have to explore Node Majority with File Share Witness. This will affect your design as a 3 node cluster with 1 node at the remote site and FSW at the production site would not have the ability to maintain quorum if the production site was lost. MS best practice for this would be to implement an even node cluster with 2 nodes at each site and the FSW at a third site.

And there you have it. While some considerations and research goes into implementing this solution, even a multi-subnet solution would be invaluable to organizations in the implementations of “warm” DR sites.

Mailbox move issue from Exchange 2003 to Exchange 2010

Ryan Roussel — Wed, 02 Jun 2010 06:24:00 GMT

Today while moving mailboxes between Exchange 2003 and Exchange 2010, I hit an issue with a couple of mailboxes. These mailboxes all popped access denied errors or more exactly: Insufficient Access Rights to perform the operation.

The cause was similar to the mail flow issue in that inheritable permissions were not turned on for the user object in Active Directory. This also presented it’s own unique problem in that since the initial move request failed because of permissions, it had to be cleared before a new move request could be created. On top of that, the request did not show up in the EMC. I used the following process to clear the request, assign permission, then create a new request:

1. First you need to know the ExchangeGUID of the mailbox for the remove-moverequest command. To quickly get the GUID for a mailbox simply run:

2. Next we need to clear out the move request using PowerShell by running:

[PS] c:\>Remove-moverequest -moverequestqueue "mailbox database 1030639620"
-mailboxguid 8525686f-d4d3-42b7-92f1-46d77ea841a3

3. Then to re-establish inheritable permissions. This can be done by using AD Users and Computers, switching to View Advanced Features, then under the Security tab of the object. Click Advanced, then check “allow inheritable permissions of parent to propagate to this object”

4. Once the Inheritable permissions are restored, we need to create a new move request:

NOTE: The EMC can also be used to initiate the Move Request once the permissions are corrected.

[PS] c:\>New-moverequest –identity jyoung -baditemlimit 100
-targetdatabase "mailbox database 1030639620"

And that’s it. The mailbox should move over smoothly with no access denied error.

Exchange 2010, Exchange 2003 Mail Flow issue

Ryan Roussel — Mon, 05 Apr 2010 15:34:36 GMT

While performing the initial Exchange 2010 deployment for a customer migrating from Exchange 2003, I ran into an issue with mail flow between the two environments. The Exchange 2003 mailboxes could send to Exchange 2010, as well as to and from the internet. Exchange 2010 mailboxes could send and receive to the internet, however they could not send to Exchange 2003 mailboxes.

After scouring the internet for a solution, it seemed quite a few people were experiencing this issue with no resolution to be found, or at least not easily. After many attempts of manually deleting and recreating the routing group connectors, I finally lucked onto the answer in an obscure comment left to another blogger. If inheritable permissions are not allowed on the Exchange 2003 object in the Active Directory schema, exchange server authentication cannot be achieved between the servers.

It seems when Blackberry Enterprise Server gets added to 2003 environments, a lot of Admins get tricky and add the BES Admin user explicitly to the server object to allow inheritance down from there to all mailboxes. The problem is they also coincidently turn off inheritance to the server object itself from its parent containers. You can re-establish inheritance without overwriting the existing ACL however so that the BES Admin can remain in the server object ACL.

By re-establishing inheritance to the 2003 server object, mail flow was instantly restored between the servers.

To re-establish inheritance:

1. Open ASDIedit by adding the snap-in to a MMC (should be included on your 2008 server where Exchange 2010 is installed)

2. Navigate to Configuration > Services > Microsoft Exchange > Exchange Organization > Administrative Groups > First Administrative Group > Servers

3. In the right pane, right click on the CN=Server Name of your Exchange 2003 Server, select properties

4. Navigate to the Security tab, hit advanced toward the bottom.

5. Check the checkbox that reads “include inheritable permissions” toward the bottom of the dialogue box.