Appdynamics Please Delete the Associated Nodes and Try Again.

This blog post is a role of a 5 post series consisting of:

• Exchange Hybrid Migrations: More than But a Pretty Confront (Part 1)
• Earthworks Into Hybrid Migration Move Study Data (Role 2)
• Troubleshooting Failed Migrations (Part three - you are hither)
• Troubleshooting Dull Migrations (Part iv)
• What to do if a migration is Completed With Warnings (Role 5)

Standing the blog post serial, we arrived at troubleshooting failed migrations.

A 'failed migration' is when the condition of the move request shows as 'failed', and nosotros take one or more failures logged in the motility study. The move is stopped and needs the administrator's attention to investigate the reason of failure. Sometimes, resuming of the motility can assistance, especially if there were some temporary bug on the Commutation Online side that were addressed.

Before getting into troubleshooting, I recommend you check the post-obit 'Minimum Requirements'; those are the things nosotros know volition suspension migrations (and we see them do so):

• MRSProxy needs to be enabled and running on Commutation on-premises.
• Substitution Online requires Negotiate (NTLM) authentication for MRSProxy.
• Brand sure your migration users are synchronized with AADconnect tool and corresponding mail users are provisioned correctly on the Commutation Online side for corresponding on-bounds mailboxes (ExchangeGuid present, alias, recipient blazon correct, accepted domains for the electronic mail addresses and secondary smtp address user@tenant.mail.onmicrosoft.com).
• Maximum storage quota in Exchange Online for both primary and annal mailbox is 100GB each. Even if you take auto-expanding enabled in Exchange Online, every bit of this writing the maximum mailbox size for migration of main archive is still 100GB – reference here.
• Maximum number of items per regular mailbox folder is i million and 3 meg for the dumpster, reference hither
• On-premises migration admin needs to have the minimum required permissions and valid credentials.
• You cannot offboard to Exchange 2010 an Exchange Online Mailbox that has any hold other than Litigation Hold. This is considering Exchange 2010 doesn't know nigh in-place hold which was introduced in Substitution 2013 or virtually organization-wide holds in Function 365. More info on the types of holds can be found hither.
• Y'all cannot offboard an Commutation Online archive mailbox which has auto-expanding enabled in Office 365
• Y'all cannot offboard a mailbox and a primary archive to Exchange 2007
• Y'all cannot offboard a remote mailbox without ExchangeGuid assault it
• Any network load-balancing for Exchange 2010 MRSProxy servers requires IP persistence (affinity).
• SSL offloading is not supported for MRSProxy.
• For archetype hybrid – where we require inbound connectivity from Exchange Online to on-bounds Exchange, allow all Substitution Online IP addresses to connect to on-premises EWS / Autodiscover.
• For archetype hybrid, pre-hallmark for EWS / Autodiscover virtual directories is not supported.(*)
• For archetype hybrid, a valid 3^rd party certificate is required for EWS / IIS. Also encounter this.
• TLS1.ii should be enabled in the on-premises infrastructure.
• If you have an Exchange organization with Exchange 2013/2016 in coexistence with legacy Exchange servers (Exchange 2010), then yous need to point the MRSproxy namespace to the newer Commutation version in your environment. This is required because, for example, Commutation 2010 cannot proxy to Exchange 2016 in order to motility an Exchange 2016 mailbox to or from Exchange Online through an Substitution 2010 MRSProxy endpoint.

(*) Strictly speaking, for hybrid migrations and EWS/mrsproxy.svc endpoint, information technology would work if the device supported NTLM pre-authentication. It would not work for other hybrid scenarios similar Free/Busy, which cannot work with pre-authentication on EWS and Autodiscover.

We too recommend that y'all bypass the network devices such every bit firewalls and reverse proxies during migrations in social club to reduce source network latency and avoid frequent communication transient errors that would upshot in mailbox locks and slow migrations.

Ofttimes when troubleshooting Office 365 migrations, the Exchange Admin Center GUI is helpful and quite verbose regarding the reason of failure, and it many times includes a link to the corresponding documentation page for more information on specific consequence.

Let me briefly evidence you lot some useful info that we can see in the (Archetype) Exchange Admin Eye. Equally a note, at the time of writing this commodity, the New Exchange Admin Center doesn't currently show all of this information.

The following tin be seen in the above screenshot:

• Nosotros take one migration batch called "Test Hybrid Migration" of type Exchange Remote Move
• Management of the move is Onboarding (from on-premises to the cloud)
• The current status Syncing (things are going well so far)
• There is just one migration user in the batch (looking at the Total column)
• The user is non Synced (hasn't reached the Incremental Sync at 95%), non Finalized (hasn't reached the 100% completion) and not Failed (didn't run across a fatal failure)

After clicking on View details, nosotros also encounter:

• Who created the batch (crystal@mytenant.onmicrosoft.com),
• When it was created and started (New-MigrationBatch -AutoStart),
• When information technology should complete (subsequently the initial sync will exist done)
• At that place is no final synced time considering the condition is syncing and no initial sync has been done
• Also, the associated endpoint is the name of my migration endpoint (Get-MigrationEndpoint) through which I am running the batch.

Later on a picayune while, the user failed because of the ExchangeGuid missing on the mail user object in Substitution Online:

In such situation, the migration service failed to inject the move request because the user failed validation. This ways that we don't accept a movement request for this user and therefore will have no move written report.
If you were to click on 'Download the report for this user', yous would get an empty .txt file.
Permit me show yous how this failure looks similar in PowerShell and what objects are created and available for u.s. to check in that location.
With Get-MigrationBatch command, we can encounter the proper name of the batch, the status, the type and how many users are independent in the batch:

To see all backdrop, run Get-MigrationBatch |FL.

Another attributes values that you saw in the Substitution Admin Middle GUI, for this batch were:

CreationDateTime           : half dozen/i/2020 eight:23:35 AM
StartDateTime              : half dozen/one/2020 viii:23:34 AM
LastSyncedDateTime         :
SubmittedByUser            : crystal@<mytenant>.onmicrosoft.com
BatchDirection             : Onboarding
SourceEndpoint             : Hybrid Exchange Miry

If I had multiple batches and I was interested in seeing this detail 1, I would run: Get-MigrationBatch "Test Hybrid Migration" or if I wanted to see all batches that are failed, I would run: Get-MigrationBatch -Status SyncedWithErrors

Going further with PowerShell, if I want to see the migration user contained in that batch, I would practise information technology like this: Get-MigrationUser -BatchId "Examination Hybrid Migration". To see all the details on the migration user, I would again append |FL

This error is self-explanatory, ExchangeGuid is missing on the user and I can likewise run across it with Become-MailUser command for this migration user:

From the Get-MigrationUser output, I can also run into the RequestGuid is empty, and so this also tells me that in that location is no move asking / movement report for this migration user. I tin can run Get-MoveRequest <user> or Get-MigrationUser -BatchId "Examination Hybrid Migration" | Get-MoveRequest to confirm this.

In cases where the error message on the migration user is not so obvious and you still don't have a move request created for information technology, yous can check Get-MigrationUserStatistics with DiagnosticInfo verbose switch: Get-MigrationUserStatistics <user identity> -DiagnosticInfo verbose |FL and see if any more details constitute.
I will at present become through some more control examples if y'all want to play around and bank check elementary or more than complicated stuff in PowerShell. Also, some things tin can be just checked from PowerShell and if you have a motion asking created and this is failed or is progressing slow, y'all can see more than on analyzing motion reports with PowerShell in later part of this blog series.

To get an overview of migration statistics:

Get-MigrationStatistics

To get all migration users, their status and corresponding batches:

Get-MigrationUser

To get a specific migration user:

Get-MigrationUser <email address>

To cheque the mistake on a specific migration user:

Get-MigrationUser <email address> |FL errorsummary
Get-MigrationUser <e-mail accost> |FL

To get all failed migration users:

Get-MigrationUser -Status Failed

To go all failed migration users and their errors:

Go-MigrationUser -Status Failed | FT identity , errorsummary
Get-MigrationUser -Status Failed | FL identity , errorsummary

To become migration users from a particular batch:

Go-MigrationUser -Batch "Batch Name"

To get all migration batches:

Get-MigrationBatch

To get a particular batch:

Get-MigrationBatch "Batch Name"

Checking movement requests (specific for hybrid remote moves)

To get all existing motion requests:

Get-MoveRequest

To get move request statistics for a specific motility asking:

Get-MoveRequestStatistics "User"

Know that there are 2 primary types of failures:

• Transient Exceptions, instance DataExportTransientException
• Permanent Exceptions, example StoragePermanentException

Note: For a motion asking to be in a Failed land, we would need to have a permanent failure. Too many transient failures (ordinarily more 60) will eventually cause a permanent failure. Also many transient failures can also slow down your migration considerably.

To see the failures (transient or permanent), you would run commands similar to these or consign the statistics to an XML file (discussed in the later role of this weblog series)

To store the motion report in a variable:

$stats = Go-MoveRequestStatistics "Affected User" -IncludeReport

To check all failures and their count:

$stats.report.Failures | grouping failuretype | Format-Table -AutoSize

To cheque total details of the last failure:

$stats.study.Failures[-1]

To check the last 2 failures:

$stats.Report.Failures | select -last 2

To check the offset failure:

$stats.report.Failures[0]

To check the kickoff 3 failures:

$stats.Report.Failures | select -first 3

If at that place are a lot of failures, you can create a listing of the failures with the PowerShell Index number associated with each failure by running the following:

$i=0;$stats.written report.Failures | % { $_ | Select-Object @{name="alphabetize";expression={$i}},timestamp,failurecode,failuretype,failureside;$i++} | ft

Using this output, you tin and then hands place the index number you want to focus on by enclosing the failure index number in [brackets], example:

$stats.report.Failures[4]

To get failed motility requests:

Get-MoveRequest -MoveStatus Failed

Most frequent failures

Here is a list of most ofttimes seen failures in hybrid migrations (and when I say 'most frequent' I hateful 'most frequent' bug that we come across in support, not that you will see those errors in every migration). Note that not all are permanent failures, meaning non all these will crusade your migrations to neglect.

• "User is already being moved" – reference here
• "You can't use the domain because it's not an accustomed domain for your system" – reference hither
• "Target mailbox doesn't have an smtp proxy matching '.mail service.onmicrosoft.com'" – reference hither
• "MigrationPermanentException: Cannot find a recipient that has mailbox GUID" – reference here. Note that another possible scenario for this fault is when we cannot find a ComponentShared Mailbox by its GUID on the Exchange Online side. A ComponentShared mailbox is used to host information from other Role 365 workloads similar Teams, OneDrive for Business and SharePoint. You would cheque (in Commutation Online PowerShell) these mailbox GUIDs with the command: Get-MailboxLocation -User <SMTP>. If the mailbox GUID in the fault belongs to a component shared mailbox, please log a instance with Microsoft Support.
• "Y'all must specify the PrimaryOnly parameter" – reference here
• "The remote server returned an Error 404" or "HTTP request has exceeded the allotted timeout" – reference here
• "The remote server returned an error: (403) Forbidden" – reference here
• "Access is denied" – reference here
• "Couldn't switch the mailbox into Sync Source mode" – reference hither
• "CommunicationErrorTransientException - The remote endpoint no longer recognizes this sequence. This is almost likely due to an arrest on the remote endpoint. The value of wsrm:Identifier is non a known Sequence identifier. The reliable session was faulted." – reference here
• "The server was unable to process the request due to an internal error.  For more information about the error, either plow on IncludeExceptionDetailInFaults ..." – references here and here
• "TooManyBadItemsPermanentException" - Failed to notice a principal from the source forest or target woods – references here and hither
• "The data consistency score (Investigate) for this request is too low" – reference here. Annotation that we will have more than on Data Consistency Score afterward in the blog post series.
• "Exception has been thrown past the target of an invocation." – reference here
• "Transient error CommunicationErrorTransientException has occurred. The system will retry" – reference here
• "The Mailbox '<username>@contoso.com' isn't enabled for unified messaging." – reference hither
• "Failed to convert the source mailbox 'Main (00000000-0000-0000-0000-000000000000)' to mail-enabled user after the move." or "Unable to update Active Directory information for the source mailbox at the end of the movement." – reference hither
• "Target user <User> already has a master mailbox". Annotation: pay special attending to the scenario, it matters if you get this fault in onboarding (move to Substitution Online) or offboarding (movement from Commutation Online). For onboarding moves, please run into this, and for offboarding see this. Note on scenario 1 stride vii in that article: it is non supported to remote restore a disconnected mailbox from Exchange 2010 on-premises source server version, it needs to be minimum Exchange 2013 version.
• "StalledDueTo_Target*" when you lot move mailboxes to Exchange Online – reference hither. More on this when we will be discussing slow migrations in adjacent part of this blog mail service series.
• "MapiExceptionTooComplex: Unable to query table rows. (hr=0x80040117, ec=-2147221225)" – reference here.
• "Mailbox Replication Proxy Service can't process this asking because it has reached the maximum number of active MRS connections allowed" – reference here.

Migration failures due to Substitution Online mailbox folder limits

I desire to talk over two common migration errors related to mailbox folder quota limits in Exchange Online when onboarding a mailbox to Office 365:

• MapiExceptionFolderHierarchyChildrenCountQuotaExceeded
• MapiExceptionFolderHierarchyDepthQuotaExceeded

First i, FolderHierarchyChildrenCount refers to Maximum number of subfolders per mailbox folder (x,000 subfolders per folder). MapiExceptionFolderHierarchyChildrenCountQuotaExceeded is thrown when a binder has more than than 10,000 same level subfolders in it on the source mailbox.
2d one, FolderHierarchyDepth refers to maximum folder hierarchy depth (300 folders 'deep').
MapiExceptionFolderHierarchyDepthQuotaExceeded is thrown when the source mailbox has a folder hierarchy with more than than 300 folder levels depth.
Here is a visual of these 2:

The mailbox folder limits are documented in the above and you can read them with PowerShell (Get-MailboxStatistics) ran against an Substitution Online Mailbox. These particular two folder limits are:

• FolderHierarchyChildrenCountReceiveQuota
• FolderHierarchyDepthReceiveQuota

You can export the mailbox folders to CSV, on the source side with the post-obit commands:

Get-MailboxFolderStatistics <user> -FolderScope NonIpmroot | Consign-Csv -NoTypeInformation PrimaryFolderStats.csv
Become-MailboxFolderStatistics <user> -Annal -FolderScope NonIpmroot | Export-Csv -NoTypeInformation ArchiveFolderStats.csv

Annotation that NonIpmRoot Folder scope is not available on Exchange 2010 Servers, you will need to utilize MFCMAPI to see system folders.

The solution is to merge affected folders or movement them across the folder hierarchy. You will then be able to drift the mailbox.
However, to avoid the warnings that the end-user volition receive after migration, you accept to make sure yous are below warning quotas (that is ix,000 for child folders and 250 for folder depth).

Here is an example of such warning for binder hierarchy children count:

There is another folder limit (maximum number of messages in a mailbox folder) that is quite oft seen in hybrid migrations: one one thousand thousand items per folder limit and more rarely the 3 meg items per Dumpster (Recoverable Items binder).
These limits are also documented hither and you lot can encounter read them using Get-MailboxStatistics:

The failure is: MapiExceptionMessagePerFolderCountQuotaExceeded. You tin export the folders items to CSV using the following commands:

Get-MailboxFolderStatistics <user> -FolderScope NonIpmroot | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation PrimaryFolderItems.csv
Become-MailboxFolderStatistics <user> -Archive -FolderScope NonIpmroot | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation ArchiveFolderItems.csv
Get-MailboxFolderStatistics <user> -FolderScope RecoverableItems | sort itemsinfolder -Descending | select folderpath, itemsinfolder | Export-Csv -NoTypeInformation PrimaryDumpsterFolderItems.csv
Get-MailboxFolderStatistics <user> -Archive -FolderScope RecoverableItems | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation ArchiveDumpsterFolderItems.csv

Solution is to reduce the number of items in the folder (by deleting or moving them to a different folder when possible).
Y'all volition need to recreate migration (remove batch or afflicted migration user from migration batch and create new batch with the user) in order to successfully fix this failure.

A few more troubleshooting tips

MoveOptions Parameter

Oftentimes mailbox moves neglect because of corrupt items or elements in a mailbox. These mailbox move failures can be avoided by excluding those (often corrupt) elements from existence migrated.

The MoveOptions parameter (previously known every bit the SkipMoving parameter which is being deprecated) can be added to the onboard or offboard request from PowerShell with the values of:

'SkipFolderRules, SkipFolderACLs, SkipFolderPromotedProperties, SkipFolderViews, SkipFolderRestrictions, SkipContentVerification, SkipPerObjectIndex'.

This will tell the migration to skip these elements when performing the move. Nosotros recommend yous perform these skips under the guidance of Microsoft Support.

You can review a move written report from a previously failed move endeavor and become some clues on what exclusions you should consider making.

For example, this failure beneath means that we have a search binder on the source mailbox where the query (restriction) is too complex and cannot be created on the target.

Sometimes the failure identifies the bodily problematic source folder so yous can look more at the DataContext content. You tin so either delete the query on the source mailbox or merely skip the migration of the queries (search folders) so that you tin can consummate the migration:

Set-MoveRequest user@contoso.com -MoveOptions @{add="SkipFolderRestrictions"}

Mailbox Integrity checks

If y'all migrate a mailbox (primary mailbox or archive) to Commutation Online and the size is bigger than 10GB, this is considered a large mailbox and the MRS will perform an ISinteg task to ensure integrity of the mailbox that is beingness moved.

If yous suspect that your move is stuck on ISinteg chore, y'all tin check the move report in EXO PowerShell and search for all strings containing isinteg keyword:

$stats = Get-MoveRequestStatistics <user> -IncludeReport
$stats.report.Entries | where { [string] $_ -like "*IsInteg*" } | % {[string] $_}

If that shows completed, this means there are no bug. Otherwise, you tin try running the aforementioned command MRS is using on your Exchange on-premises environment, in Ems:

New-MailboxRepairRequest <migration user identity> -CorruptionType MessageId

For more than info on the New-MailboxRepairRequest cmdlet, y'all can check hither.

Depending on the Substitution Server Version y'all can check so the status of the repair request.

For Exchange 2013 and later, use this cmdlet:

Get-MailboxRepairRequest -Mailbox <user identity>

For Substitution 2010 version, you lot would need to look in Upshot Viewer for the post-obit events:

• Event 10047 when the repair request is started
• Event 10062 when a abuse is detected and repaired
• Outcome 10048 when the repair completes successfully

You can likewise try to move a mailbox locally from one server to another, remove the local motility request and and so retry migration of the mailbox to Exchange Online.

Testing MRS service

One utility that can be used for troubleshooting the mailbox move performance is the Test-MRSHealth cmdlet.  One thing to realize is that information technology cannot be tested from Part 365 side since the cmdlet is non bachelor to a tenant administrators. However, at to the lowest degree from my experience, I have never encountered a situation where MRS service would be stopped on the Office 365 side (and was non automatically recovered within seconds). We can use this utility to test the mailbox replication service health on-premises. Also on-premises, y'all can check if the MRSProxy is enabled on the EWS virtual directories and if EWS awarding pool is started in IIS manager.

Issue Viewer Diagnostic logging

When performing a mailbox move, you can turn up diagnostic logging on the mailbox replication service or other component like asp.net to get ameliorate, more granular events in the result log on-premises.

In most situations, you lot don't actually get useful events in the on-bounds event viewer when troubleshooting an Commutation Online remote move due to the fact that those events would exist written in the datacenter. The default event logging can provide y'all with plenty information on what the issue would be, take for example event 1309 from ASP.Cyberspace where the description is cocky-explanatory: MRSproxy service existence disabled.

If you exercise notice a relevant event log for the affected Commutation Online remote move in the upshot viewer and this is related to MRS, you can turn up diagnostic logging for the MRS service with the post-obit cmdlet:

Become-EventLogLevel 'MSExchange Mailbox Replication*' | Set-EventLogLevel -Level Proficient

Then reproduce the event or wait for it to be reproduced once more and so check in the Effect Viewer logs for any relevant events.

Tracking incoming failed requests from EXO

Especially useful in communication or timeout failures, there are 3 main logs to track the MRS requests on the Exchange on-premises servers in order for you to understand if an MRS Exchange Online request reached your Commutation server, or not. These oft aid us narrow down the issue to a about probable network device (in front of Substitution Server) that could cease the connection and not laissez passer information technology to Exchange Servers. Or if the request reaches the Commutation servers, nosotros can see where this is stuck and get a improve understanding on what's the problem on the Exchange server on-premises.

Exchange on-bounds server logs to rails an EXO Incoming MRS request:

• HTTPerr logs: %SystemRoot%\System32\LogFiles\HTTPERR
• IIS logs for Default Web Site (DWS): %SystemDrive%\inetpub\logs\LogFiles\W3SVC1 – UTC Timezone

The proper noun of the IIS logs contains the date of the log, for example u_ex190930.log is from Sept 30, 2019.

• HTTPProxy logs for EWS (available in Exchange 2013 or after): %ExchangeInstallPath%Logging\HttpProxy\Ews

The name of the HTTPProxy logs contains the date and hr starting to log, for example HttpProxy_2019093014-10.LOG (ten^th log from Sept thirty, 2019, starting 60 minutes 14:00 UTC)

Few things to mention here:

• Always correlate the timestamp of a failure HH:MM:SS in move report with these logs (IIS and HTTPProxy are in UTC timezone)
• A failed request will never have 200 Condition code (if you lot meet it with 200 in logs, it means you are not looking at the failed one). Note that for a request that times out, you might nevertheless exist able to see information technology hither with 200 status code and possibly a higher fourth dimension-taken
• If yous see the failed request  in HTTPerr logs, this won't probably be present in IIS logs or HTTPProxy logs – it is stuck in front end of IIS, check the particular reason in HTTPerr logs and check for IIS misconfiguration
• If you see the failed requests in IIS logs , so you tin do IIS failed request tracing on that status code and check further the detailed error in HttpProxy logs

This concludes Part 3 of these blog series. We volition exist talking about troubleshooting slow migrations next!

I would similar to thank the following persons for contributing to this blog and for their time and patience to read this: Angus Leeming, William Rall, Brad Hughes, Chris Boonham, Ben Winzenz,  Cristian Dimofte, Nicu Simion, Nino Bilic,  Timothy Heeney

taylorsmill1968.blogspot.com

Source: https://techcommunity.microsoft.com/t5/exchange-team-blog/troubleshooting-failed-migrations/ba-p/1746234

Share this post