For no apparent reason, on one 2x1x1 SharePoint 2007 (build 12.0.0.6548) I started getting crawl errors on an entire web application. Error message reads something like this:
The start address <http://<yourWebApp.domain.com>> cannot be crawled.
Context: Application 'yourSSPName', Catalog 'Portal_Content'
Details: The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.. (0x80041206).
I tried all kind of things, including, disabling the firewall, restarting the admin and timer services, checking the hosts file for SharePoint created entries. Not to mention, making sure accounts used by SSP and Web Apps have required permissions, and there are no DCOM errors. I also moved around the search query role but not getting any results. Crawl was just failing when attempting to reach the web app. interestingly other Web Apps like My Site were just fine and could not found any pointer on Google.
Finally, case was cracked. The farm is configured using Kerberos and there were some Kerberos audit errors. Nothing very unusual and I was not paying much attention to them when working on this issue. The environment has a domain controller running on Windows 2003 OS (for Exchange 2007 compatibility) and that DC frequently goofs up on Kerberos (still have to figure out why) and the SharePoint Servers were using that DC for authentication and logon server.
Just to eliminate that cause, I stopped that DC and recycled the SharePoint Servers and started a new crawl and Bingo – here you go.
So the lesson learned for crawl to be successful on Kerberos configured web application, it should not only be configured correctly (Web App URL that configured in Content Source configured as SPN) but Kerberos should be working as well. Kerberos falls back to NTLM but crawl seems to be very sensitive.
There is an excellent article that worth reading. Check out this TechNet link for more details http://technet.microsoft.com/en-us/library/cc298559.aspx
I am sure if Kerberos problem on Windows 2003 is not solved, the crawl issue will return that day when SharePoint Servers start using that faulty domain controller again, but that’s for some other day to solve.
Kerberos errors may not be even getting logged in security event viewer. You may have to turn on the auditing for both success and failure. On Index Server you will see something of this effect:
Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 1/17/2011 10:45:55 PM
Event ID: 4625
Task Category: Logon
Level: Information
Keywords: Audit Failure
User: N/A
Computer: <indexServer>.agileconcepts.com
Description:
An account failed to log on.
Subject:
Security ID: DOMAINNAME\FarmDbAppPoolID
Account Name: FarmAppPoolID
Account Domain: <DOMAINNAME>
Logon ID: 0x6a4d53
Logon Type: 5
Account For Which Logon Failed:
Security ID: NULL SID
Account Name: <Default_Crawl_Account>
Account Domain: <DOMAINNAME>
Failure Information:
Failure Reason: An Error occured during Logon.
Status: 0xc000018b
Sub Status: 0x0
Process Information:
Caller Process Name: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\OWSTIMER.EXE
Network Information:
Workstation Name: <IndexServerName>
Source Network Address: -
Source Port: -
Detailed Authentication Information:
Logon Process: Advapi
Authentication Package: Negotiate
Transited Services: -
Package Name (NTLM only): -
Key Length: 0
On Index Server specific info logged into Event Log is below:
Log Name: Application
Source: Office Server Search
Date: 1/18/2011 2:24:57 PM
Event ID: 2436
Task Category: Gatherer
Level: Warning
Keywords: Classic
User: N/A
Computer: agile03.agileconcepts.com
Description:
The start address <http://webApp.domain.com> cannot be crawled.
Context: Application 'SSPName', Catalog 'Portal_Content'
Details:
The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.. (0x80041206)
Event Xml: <System>
<Provider Name="Office Server Search" />
<EventID Qualifiers="32768">2436</EventID>
<Level>3</Level>
<Task>3</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2011-01-18T19:24:57.000000000Z" />
<EventRecordID>323102</EventRecordID>
<Channel>Application</Channel>
<Computer>IndexServer.agileconcepts.com</Computer>
<Security />
</System>
<EventData>
<Data>
Context: Application 'SSPName', Catalog 'Portal_Content'
Details:
The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.. (0x80041206)</Data>
<Data>http://webApp.domain.com</Data>
</EventData>
</Event>