Viewing 3 posts - 1 through 3 (of 3 total)
  • Author
    Posts
  • #479879

    I received a notice from Google stating that they could not access js and css files. I have had a disallow wp-includes in the robots.txt file for years, without problem; but, since the js and css files are in wp-includes, it appears now to be a problem. Apparently, Google requires access to the js and css files. I removed the disallow wp-includes, which should solve that problem, but opens up a lot of other files for Google to index which may not be so desirable. Are you aware of this issue??

    And, would it suffice to do something like this:
    User Agent*
    Disallow/ wp-includes/
    Allow/ wp-includes/js
    Allow/ wp-includes/css

    Thanks for your help…..

    #480292

    Hi kenwae!

    It doesn’t sound like a theme issue. Try deleting your robots.txt file and then have Google crawl your site again in webmaster tools.

    Also be sure to deactivate all plugins while testing.

    Cheers!
    Elliott

    #481018

    Thanks for the response; no, it is not a theme issue; from what I am reading on the net, this is a fairly upcoming issue across the board will all WordPress sites; whereas disallowing /wp-content in robots.txt has been a common practice, it appears now to be considered a bad idea. Same with disallowing /wp-admin. So, I’ve removed those disallows and we’ll see what happens in terms of Googlebot’s indexing of irrelevant stuff, which tends to contribute to crawl errors. But, in a general sense, perhaps users should know that disallowing /wp-content is now considered bad practice….

    And, you can close this now….

    • This reply was modified 8 years, 8 months ago by kenwae.
Viewing 3 posts - 1 through 3 (of 3 total)
  • The topic ‘Problems with Googlebot and robots.txt disallow/ wp-includes’ is closed to new replies.