This is the question I was asking myself a couple of days ago. I have a WordPress responsive website, and I know that it works well on mobile devices. So why would Google's mobile friendly tester show my website as NOT friendly? Firstly let's explore why you need a mobile friendly website.
Google's upcoming algorithm change
Well according to the Google Webmaster blog in a post titled "Finding more mobile friendly search results", Google will very soon be making a change to the search results displayed on a mobile device.From the article:"Starting April 21, we will be expanding our use of mobile-friendliness as a ranking signal. This change will affect mobile searches in all languages worldwide and will have a significant impact in our search results. Consequently, users will find it easier to get relevant, high quality search results that are optimized for their devices"The key words here are "... will affect mobile searches ... worldwide ... will have a significant impact in our search results ..."I think you get the picture that if you want to appear when someone is doing a Google search on a mobile device after 21st April 2015, then you need to be mobile friendly.
How to be mobile friendly
Well the preferred way to be mobile, according to Google is to have just one version of your website that is a Responsive Website Design.What reasons does Google give for this (source: why responsive design):
[cposhort_list icon="check" style="square"]easier for people to share a single link to a page on your website, rather than multiple links to the same content[/cposhort_list][cposhort_list icon="check" style="square"]easier for Google to index a single url than a separate desktop and mobile url[/cposhort_list][cposhort_list icon="check" style="square"]only have one page for the webmaster to maintain[/cposhort_list][cposhort_list icon="check" style="square"]saves on load time when redirection is required based on device type[/cposhort_list][cposhort_list icon="check" style="square"]improves Googlebot's crawling efficiency as only one version of a page to be crawled and indexed so "can indirectly help Google index more of your site's content and keep it appropriately fresh"[/cposhort_list]
So if you are running a WordPress website, then simply choose a responsive theme. You can also use a mobile plugin that creates a mobile version of your website, however given the reasons listed above you would be better off using a responsive theme. There are plenty to choose from and sometimes the theme you are using may actually have a setting to turn on responsive design, so if you are not sure then check the documentation that comes with your theme, or alternatively contact your theme provider.
How to test if your website is now mobile friendly?
This step is nice and easy, since Google have provided us with a really easy tool to use. Simply visit test here and enter your home page url for Google to test.Another way to test is by setting up your website with Google Webmaster Tools, and view the "mobile usability" section for any errors that Google has found.
What if I have a responsive website and Google is showing it as "not friendly"?
Well this is what I found out the other day. Even though my website is mobile friendly through testing on a range of mobiles and tablets, Google seems to be of the opinion that it is not.Why is it so?So I checked using the Chrome developer tools, and switched on the device mobile emulator to verify that the website was still appearing correctly on mobile devices. Yes that checked out ok. Then I considered why the mobile testing tool might be showing a different result to Google. Reading on the page I realised that Google is trying to view the website exactly as a typical mobile user might see the page. So that means it needs access to all the stylesheets and javascript, images and other resources that might effect how a page looks.Then it made sense. Based on the way that used to be recommended for WordPress websites, we use the robots.txt file to help prevent crawling and indexing of content that isn't relevant for search results. In other words, we simply wanted Google and other search engines to crawl the text content of our websites, not all the other files. So the Googlebot was being blocked from crawling all the stylesheets and javascript, so it cannot therefore see the website exactly how it actually appears on a mobile device.
So even though the website is fully responsive, the reason why Google believes it is not is because the robots.txt file blocks the files that Google now wants to be able to view.
So what to do? Do I simply remove the robots.txt file and allow all crawling of the website? Do I amend the file and stop blocking all the files?
A new robots.txt file for Googlebot
I decided that the reason for blocking certain resources on the website is still relevant at this stage, though in the future this may change again. So the best way to fix the issue with Googlebot is to allow it to crawl the javascript, css, images and other important files. However I still may have some files loaded in my website that I don't want crawled.Thankfully we can target Googlebot in our robots.txt using "User-agent: Googlebot". Also although the directive "Allow" is not recognised across all crawlers (so it shows as an error in robots txt testing tools), Google does actually recognise it, provided that it is "the URL path in of a subdirectory, within a blocked parent directory, that you want to unblock" (source: Webmaster tools help).This is my revised robots.txt tool with an explanation below:User-agent: Googlebot
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/plugins/
Allow: /wp-content/plugins/*.css
Allow: /wp-content/plugins/*.js
Disallow: /wp-content/mu-plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Allow: /wp-content/themes/*.css
Allow: /wp-content/themes/*.js
Allow: /wp-content/themes/*.gif
Allow: /wp-content/themes/*.png
Allow: /wp-content/themes/*.jpg
Disallow: /*.php$
Disallow: /*.inc$
Disallow: /*?
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/mu-plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*?
Sitemap: https://www.davidtiong.com/sitemap.xml
Explanation:
- Lines 1-17 are for Googlebot
- Lines 19-31 are for all other bots
- Line 33 is to inform search engines where my sitemap is located
- Line 4 blocks the directory "plugins", but lines 5-6 allow Googlebot to access css and js files in there
- Line 9 blocks the directory "themes", but lines 10-14 allow Googlebot to access css, js and image files in there
You can add other allow or disallow rules in there as needed using the same pattern, however remember that your robots.txt file is publicly available so be careful what you put in there, eg: don't put important pathways in there to hidden files.
Submitting the new robots.txt file to Google and testing
Once you have uploaded your new robots.txt file to your website, you want to do the following:
1. Clear any caches to ensure your new robots.txt file is available2. Purge any old copy of your robots.txt file from your CDN or Cloudflare if applicable3. Test your robots.txt file for errors - try tool.motoricerca.info/robots-checker.phtml (remember that the allow directive will show as errors)4. Tell Google about your new robots.txt file by going into your Google Webmaster Tools account, then navigate to Crawl > Robots.txt tester. Then click on the submit button, then next to 3 ask Google to update - click submit (see images below).
5. Check that css, js and image files are not being blocked by Googlebot anymore by using the Robots.txt tester, view the images below as an example.
6. Check that your website is now mobile friendly using the mobile friendly tester - www.google.com/webmasters/tools/mobile-friendly/.7. Keep an eye on your Google webmaster tools account over the next couple of weeks to make sure that no new errors are showing up.
How did you go? If you need to print out this page to refer to, then click the button below.[print-me-button buttontext="Print a copy"]
Comments: