robots.txt « Support Forum | Kriesi.at - Wordpress Themes and HTML Templates

Support Forum » Choices

robots.txt

4 posts from 3 voices

Started 5 months ago by marko7
Latest reply from Devin
This topic is not a support question

Tags:

robots.txt

marko7
Member

Hi. I have just received this:

http://www.accommodationinapollobay.com.au/: Googlebot can't access your site
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

The robots.txt appears to be:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Should I change this and if so where can I access the file?

Thanks.
Mark.

Posted 5 months ago #
Ismael
Moderator
Hi,

First, try to disable each plugins see if that is the culprit. Check if your Dashboard > Privacy > "My blog is visible to anyone" is enabled.

Are you using Google XML Sitemap generator?

Edit sitemap-core.php file for this plugin.

Find this code around line 3151

echo "\nSitemap: " . $smUrl . "\n";

Replace it with
```
echo  "\nUser-agent: *";
echo  "\nAllow: /\n";
echo  "\nSitemap: " . $smUrl . "\n";
```
Regards,
Ismael
Posted 5 months ago #
marko7
Member

Hi Ismael.
I am using Yoast Wordpress SEO and I have checked the box to enable sitemap functionality.
My Privacy settings in the dashboard enable access by search engines.
If I disable plugins how will I know that Googlebot can then access the site?

Posted 5 months ago #
Devin
Moderator

Hi Mark,

After disabling your plugins you should be able to tell by if you get the same message you received to first let you know about the issue.

Just looking at your robots.txt file as is, I don't see why you should have any issues.

Regards,

Devin

Posted 5 months ago #