For your page to appear in a search engine, two separate steps must succeed: crawling (the search engine visits and reads the page) and indexing (it places the page in its collection so it can appear in results). Here you have two goals: to ensure the search engine quickly discovers and indexes your important pages, and at the same time to keep unimportant or private pages out. The first is handled by the sitemap, links, and a fast server; the second by robots.txt and the noindex tag - two tools you must not confuse.
Crawling versus indexing
The robots.txt file governs crawling: it tells the search engine which addresses not to visit. The noindex tag (in the page's head) governs indexing: it tells the search engine not to place the page in the results. Robots.txt therefore does not remove a page from the results - only noindex can do that.
How to speed up the bot's visits and indexing
The search engine discovers new pages in three ways: through the sitemap (XML sitemap) you submit, through internal links it follows from already-known pages, and through backlinks from other websites. For fast discovery:
- Submit the sitemap in Google Search Console and make sure it updates on the fly with fresh dates (lastmod) - a static sitemap that never changes loses its purpose.
- Link internally strongly: more internal links from established pages to new content means faster discovery.
- Request indexing of an individual important page through the URL Inspection tool in Search Console (only about 10 to 12 requests per day are available, so reserve them for the most important pages).
- Use the IndexNow protocol, which notifies search engines (Bing and others) immediately on publishing that content is new or changed.
- A fast, reliable server: a fast response and good Core Web Vitals let the search engine visit more of your pages in the same time.
Keep crawl budget in mind: new stores and pages with little authority get fewer visits from the search engine, so their discovery is slower.
Backlinks speed up discovery
Backlinks (links from other websites to your page) are one of the three discovery paths - and a very effective one. The search engine assigns higher crawl priority to addresses that trusted sources point to, and at the same time such links build your site's authority and thus a larger crawl budget. The difference is noticeable: sites with strong authority are often indexed within a few days, while weaker ones can take several weeks. Backlinks are therefore not only a ranking signal but also an accelerator of discovery and indexing. (A separate step is devoted to building authority with links.)
The most common mistake: robots.txt is not for hiding from results
If you disallow an address in robots.txt, the search engine does not visit it, but it can still place it in the results - without content, only with the address - if links point to it. From this follows the rule: if you want to remove a page from the results, give it a noindex tag and let the search engine crawl it. An address that should be deindexed must NOT also be disallowed in robots.txt, otherwise the search engine never sees the noindex tag and the page stays in the index. Allow crawling, then apply noindex.
What to hide from the search engine in an online store
In robots.txt, disallow the crawling of addresses with no search value that only waste crawl budget: checkout (/checkout/), the cart (/cart/), and the customer account (/customer/), internal search (/catalogsearch/result/) and product comparison, and session and store parameters (?SID=, ?___store=). Also include in robots.txt a link to the sitemap, so the search engine finds it easily.
How to set it up in Magento
Magento has built-in editing of the robots.txt file: in Content > Design > Configuration (per store) in the Search Engine Robots section, set the default value of the robots tag (e.g. INDEX, FOLLOW), add your own Disallow rules and a link to the sitemap. Magento generates the sitemap automatically (Stores > Configuration > Catalog > XML Sitemap). For precise control of the noindex tag per individual page type, a dedicated SEO module is often required, since core Magento does not offer this level of control.
How to measure whether it works
In Google Search Console use URL Inspection (whether a page is indexed and why not) and the page indexing report. Track the time from publishing to indexing of new pages - if it shortens, your acceleration measures are working. Pay particular attention to the warning "Indexed, though blocked by robots.txt", which is a sign of misuse. The goal is twofold: important pages indexed quickly and reliably, and unimportant and private ones kept out.
