.Google's John Mueller responded to an inquiry regarding why Google.com indexes web pages that are refused from creeping by robots.txt as well as why the it is actually risk-free to overlook the relevant Look Console records concerning those creeps.Crawler Visitor Traffic To Inquiry Parameter URLs.The person inquiring the question documented that robots were actually making hyperlinks to non-existent concern guideline Links (? q= xyz) to pages with noindex meta tags that are additionally obstructed in robots.txt. What caused the concern is actually that Google is actually creeping the hyperlinks to those webpages, getting obstructed by robots.txt (without noticing a noindex robots meta tag) then getting turned up in Google Search Console as "Indexed, though blocked out through robots.txt.".The person asked the observing concern:." Yet here's the major question: why would Google.com index web pages when they can't even see the information? What is actually the advantage because?".Google's John Mueller affirmed that if they can not crawl the web page they can not view the noindex meta tag. He also creates a fascinating reference of the web site: search driver, urging to neglect the results because the "typical" individuals will not observe those results.He created:." Yes, you're appropriate: if we can not creep the webpage, our team can not find the noindex. That pointed out, if our company can not crawl the web pages, at that point there's not a whole lot for us to mark. Therefore while you could find some of those pages with a targeted website:- inquiry, the common individual will not view them, so I would not bother it. Noindex is actually also great (without robots.txt disallow), it simply implies the Links will definitely find yourself being actually crept (as well as end up in the Browse Console report for crawled/not indexed-- neither of these standings result in issues to the rest of the site). The essential part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the restrictions in using the Site: search progressed search driver for analysis main reasons. Among those factors is given that it's certainly not hooked up to the regular search index, it is actually a distinct factor entirely.Google.com's John Mueller talked about the web site search driver in 2021:." The short answer is actually that a site: query is actually certainly not implied to become total, nor utilized for diagnostics objectives.A website query is actually a specific type of search that limits the end results to a certain site. It is actually essentially only words website, a digestive tract, and then the internet site's domain.This concern confines the results to a particular site. It is actually certainly not meant to be an extensive selection of all the web pages coming from that site.".2. Noindex tag without making use of a robots.txt is great for these kinds of situations where a bot is actually connecting to non-existent web pages that are actually acquiring found out by Googlebot.3. URLs along with the noindex tag will produce a "crawled/not indexed" entry in Explore Console and that those won't possess an adverse effect on the rest of the site.Read through the inquiry and answer on LinkedIn:.Why would certainly Google.com mark web pages when they can't also find the web content?Included Image through Shutterstock/Krakenimages. com.