While trying to establish the Search Crawl on a UAT server, I was having a recurring error :

The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly

This led me to check a number of things :

• Default Search Crawl Account – check – it HAS access to the server
• Can connect using a browser (with the account) – check – that works – while LOCAL logged on (RDP)
• Dump index – and re-crawl – no, didn’t fix it.
• Try crawling another site collection (eg. Central Admin) – ok – that works !!

I was beginning to run out of ideas, when I suddenly clicked on the page url that was shown in the error log.   And, I was shown an ACCESS DENIED page (proper ASP.NET stack trace) – for IUSR.

Hang on a sec – that’s the ANON account (!!!)

Yes – my site (intranet) has been extended to also allow anonymous users.

And – here’s the clincher – the page that was the HOME page was not anonymously accessible…

My home page was set to be “/SitePages/Home.aspx”.

I changed this to be “/Pages/Home.aspx” – with a new, blank page – checked-in and published – and then tried to re-crawl the content.

SUCCESS !!

The simple reasoning is (as far as I can tell) :

• Search will try to hit the home page in ANON mode first – when crawling.  
• If this fails (denied) – then crawling STOPS.
• If it succeeds, it flips to AUTH mode, and crawls all content happily.
• So – make sure your home page is ANON accessible – only if you need ANON access (of course).
• If you don’t have it turned on, this is a non-issue.

It was a wierd error for sure – I’ll probably never encounter it again – but hope this helps someone !