.Google's John Mueller answered an inquiry about why Google.com marks webpages that are forbidden from crawling through robots.txt and also why the it is actually safe to disregard the related Explore Console files about those crawls.Bot Web Traffic To Question Guideline URLs.The individual asking the concern recorded that bots were developing web links to non-existent concern guideline URLs (? q= xyz) to webpages with noindex meta tags that are also shut out in robots.txt. What motivated the inquiry is that Google is crawling the links to those pages, acquiring blocked out by robots.txt (without noticing a noindex robotics meta tag) at that point receiving turned up in Google Explore Console as "Indexed, though shut out by robots.txt.".The person asked the complying with inquiry:." However listed below is actually the large concern: why would Google index pages when they can't even see the web content? What's the benefit in that?".Google.com's John Mueller affirmed that if they can not crawl the webpage they can't find the noindex meta tag. He likewise produces a fascinating reference of the website: search driver, recommending to dismiss the results due to the fact that the "average" users won't view those results.He wrote:." Yes, you're right: if our team can't crawl the web page, our experts can't see the noindex. That said, if our team can not crawl the webpages, at that point there's certainly not a whole lot for our team to index. Therefore while you may find a few of those web pages with a targeted internet site:- concern, the common user will not see them, so I wouldn't bother it. Noindex is actually likewise great (without robots.txt disallow), it only suggests the Links are going to find yourself being actually crept (and end up in the Explore Console file for crawled/not listed-- neither of these standings cause problems to the rest of the website). The fundamental part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's response verifies the restrictions in operation the Site: search progressed hunt operator for analysis reasons. One of those causes is since it's not attached to the regular search mark, it is actually a separate factor completely.Google's John Mueller commented on the site hunt driver in 2021:." The short response is actually that an internet site: query is actually not suggested to become total, neither used for diagnostics objectives.An internet site inquiry is a specific sort of hunt that confines the end results to a specific site. It is actually primarily merely words website, a digestive tract, and after that the web site's domain name.This concern confines the results to a particular internet site. It is actually certainly not meant to become a comprehensive selection of all the web pages from that site.".2. Noindex tag without making use of a robots.txt is alright for these type of conditions where a bot is connecting to non-existent webpages that are receiving discovered by Googlebot.3. URLs along with the noindex tag will certainly generate a "crawled/not catalogued" item in Browse Console and also those won't have a bad effect on the remainder of the web site.Read the concern as well as respond to on LinkedIn:.Why will Google index webpages when they can not even find the material?Featured Picture by Shutterstock/Krakenimages. com.