Crawlable Links: What Are They And Why Are They Important?

Whether or not you understand how the internet works (and whether or not you even want to), becoming familiar with the concept of crawlability can help you ensure that your website is properly indexed by search engines. 

Since indexing is the necessary prerequisite to receiving organic traffic, you can already gauge the importance of ensuring your links can be crawled. 

Here’s what you need to know about crawlable links and their importance: 

What Are Crawlable Links?

A crawlable link is a link that can be followed by search engine crawlers. 

Links that are not crawlable can, for example, be put to use by the JavaScript code on a page, but they can’t be used by crawlers. 

Build A Powerful Internal Link Strategy Today

Enter your email & we'll send you 8 tips to build an internal link strategy.

And 3 things you should avoid doing with internal links.

    We won't send you spam. Unsubscribe at any time.

    Crawlability, then, is the ability of said crawlers to access your website and all of its resources. It’s not to be confused with indexability, which is the ability of a search engine to analyze your pages and add them to its index

    In order for your links to first be crawlable and then indexable, you need to use the <a> tag with the href attribute, followed by a real, live URL. 

    What Does a Crawlable Link Look Like? 

    Here’s an example of a crawlable link: 

    <a href=””> 

    Note the space between the opening of the <a> tag, and the lack of space between the href, the equals sign and the open quotation marks. Note the same lack of space between the closed quotation marks and the closing of the tag <a>.

    Only links in this format will work and be crawlable. If you add or omit a space, they won’t. 

    Here’s what Google says they can crawl: 

    screenshot from google guidelines on crawlable links

    What Does a Non-Crawlable Link Look Like?

    A non-crawlable link can look like anything other than the above. 

    For example, all of these links are non crawlable:

    <a href=”> 

    <a href=”> 

    <a =””> 


    Google themselves list these examples: 

    screenshot from google guidelines on non-crawlable links

    Why Is Link Crawlability Important?

    In order for a website’s pages to be properly indexed and ranked, they first need to be crawled. If your links aren’t crawlable, search engines won’t be able to analyze their content, add them to their index, and determine how high they should rank. 

    There are very rare cases where Google can index a URL that has not been crawled. They will use the URL text and the anchor text of its backlinks to do it, but the page’s title and description will not show up in search results. 

    Note that crawlability impacts not just Google’s crawlers, but the crawlers of all search engines. 

    SEO tools like Ahrefs, SEMRush and Screaming Frog also use crawlers to scan and analyze websites, and they won’t have access to your non-crawlable links either. This can impact the results of their crawls and analyses. 

    Factors That Can Affect Crawlability 

    Certain factors can directly influence the crawlability of a link. They include:

    • Discoverability: Pages that can’t be discovered can’t be crawled. If a page has no internal links and is therefore an orphan page, and if it can’t be found in the sitemap, it won’t be crawled, and consequently indexed or ranked. 
    • Nofollow attribute: Links that are marked as “rel=nofollow” are not followed by Google’s crawlers. Well, at least they shouldn’t be. Google themselves say they treat this attribute as a hint, so they may or may not choose to follow nofollow links. However, in theory, if  a page only has one nofollow incoming link, it might not get crawled. 
    • screenshot from google guidelines on nofollow links
      Robots.txt file: If you have disallowed a part of your website in your robots.txt file, Google’s crawlers won’t access it, so they will not be able to crawl it. 

    How Do You Know If a Link is Crawlable?

    Google themselves give you two ways to check if a link is crawlable

    If you have Search Console access to the website whose crawlability you want to check, sign in and add the URL you want to test to the search bar at the top.

    The results will tell you if the page was crawled and whether it is indexed, and a bunch of related information. 

    Google Search Console URL Inspection


    If you don’t have Search Console access, or would simply like to do another test, you can use Google’s Rich Results Test tool. Just pop the URL in and wait for the results to appear (it may take quite a while). 


    rich results screenshot
    You can also try using the site: command in Google search to see if a page has been indexed. If it has, it has also successfully been crawled.

    site: command in google screenshot

    You can also use SEO tools such as Ahrefs and Screaming Frog to check the crawlability of a website’s link. And of course, our own Link Whisper will help you stay on top of all of your internal and external links. 

    How To Make Your Links Crawlable

    These are the boxes you need to tick to make sure you links are crawlable:

    • Use the href attribute correctly
    • Add all of your pages to the correct sitemap
    • Make sure every page on your website has incoming internal links and that you use the correct anchor text for them 
    • Keep all of your internal links dofollow
    • Make sure the page is not disallowed in robots.txt
    • Reduce the number of clicks it takes to get from the homepage to other pages  

    Note that while you should allow Google and other search engine crawlers access to your website, you can block other bots from crawling your pages, if you so choose. This can help you save bandwidth and prevent others from accessing information you would rather keep private, like your website’s structure. 

    How Link Whisper Can Help Your Website’s Crawlability 

    Now that you understand the basics of crawlability, you know just how important it is to help search engines out by establishing a clear page hierarchy and providing plenty of internal links to your most important pages. 

    Use Link Whisper to easily manage your internal links, their anchor text and check for any broken links or 404 pages. It will save you not only a lot of time, but all the annoyance that usually comes with a broken or poorly chosen internal link. 

    Build A Powerful Internal Link Strategy Today

    Enter your email & we'll send you 8 tips to build an internal link strategy.

    And 3 things you should avoid doing with internal links.

      We won't send you spam. Unsubscribe at any time.

      4 Ways To Check Outbound Links And Why It’s Important

      Contents1 Why Is Checking Outbound Links Important?2 Use Link Whisper to Check Outbound Links3 Check Outbound Links With an SEO Spider4 Use an SEO Tool to Find Outbound Links5 Check Outbound Links Manually6 The Best Way to Check Outbound Links …

      Read More

      How to Perform an Internal Link Audit: Complete Internal Link Analysis

      Contents1 Why Internal Link Audits Are Important for SEO1.1 User Experience1.2 Page Rank2 How to Complete an Internal Link Audit: Step-by-Step2.1 1. Develop a Website Structure for Internal Linking2.2 2. Get Your Link Juice in the Right Places2.3 3. Fix…

      Read More

      Get Started with LinkWhisper

      Speed Up the Process of Internal Linking and Help You Rank Better in Google

      Get LinkWhisper Now