Crawlable Links: What Are They And Why Are They Important?

Whether or not you understand how the internet works (and whether or not you even want to), becoming familiar with the concept of crawlability can help you ensure that your website is properly indexed by search engines. 

Since indexing is the necessary prerequisite to receiving organic traffic, you can already gauge the importance of ensuring your links can be crawled. 

Here’s what you need to know about crawlable links and their importance: 

What Are Crawlable Links?

A crawlable link is a link that can be followed by search engine crawlers. 

Links that are not crawlable can, for example, be put to use by the JavaScript code on a page, but they can’t be used by crawlers. 

Crawlability, then, is the ability of said crawlers to access your website and all of its resources. It’s not to be confused with indexability, which is the ability of a search engine to analyze your pages and add them to its index

In order for your links to first be crawlable and then indexable, you need to use the <a> tag with the href attribute, followed by a real, live URL. 

What Does a Crawlable Link Look Like? 

Here’s an example of a crawlable link: 

<a href=”https://linkwhisper.com”> 

Note the space between the opening of the <a> tag, and the lack of space between the href, the equals sign and the open quotation marks. Note the same lack of space between the closed quotation marks and the closing of the tag <a>.

Only links in this format will work and be crawlable. If you add or omit a space, they won’t. 

Here’s what Google says they can crawl: 

screenshot from google guidelines on crawlable links

What Does a Non-Crawlable Link Look Like?

A non-crawlable link can look like anything other than the above. 

For example, all of these links are non crawlable:

<a href=https://linkwhisper.com”> 

<a href=”https://linkwhisper.com> 

<a =”https://linkwhisper.com”> 

<a https://linkwhisper.com>

Google themselves list these examples: 

screenshot from google guidelines on non-crawlable links

Why Is Link Crawlability Important?

In order for a website’s pages to be properly indexed and ranked, they first need to be crawled. If your links aren’t crawlable, search engines won’t be able to analyze their content, add them to their index, and determine how high they should rank. 

There are very rare cases where Google can index a URL that has not been crawled. They will use the URL text and the anchor text of its backlinks to do it, but the page’s title and description will not show up in search results. 

Note that crawlability impacts not just Google’s crawlers, but the crawlers of all search engines. 

SEO tools like Ahrefs, SEMRush and Screaming Frog also use crawlers to scan and analyze websites, and they won’t have access to your non-crawlable links either. This can impact the results of their crawls and analyses. 

Factors That Can Affect Crawlability 

Certain factors can directly influence the crawlability of a link. They include:

  • Discoverability: Pages that can’t be discovered can’t be crawled. If a page has no internal links and is therefore an orphan page, and if it can’t be found in the sitemap, it won’t be crawled, and consequently indexed or ranked. 
  • Nofollow attribute: Links that are marked as “rel=nofollow” are not followed by Google’s crawlers. Well, at least they shouldn’t be. Google themselves say they treat this attribute as a hint, so they may or may not choose to follow nofollow links. However, in theory, if  a page only has one nofollow incoming link, it might not get crawled. 
  • screenshot from google guidelines on nofollow links
    Robots.txt file: If you have disallowed a part of your website in your robots.txt file, Google’s crawlers won’t access it, so they will not be able to crawl it. 

How Do You Know If a Link is Crawlable?

Google themselves give you two ways to check if a link is crawlable

If you have Search Console access to the website whose crawlability you want to check, sign in and add the URL you want to test to the search bar at the top.

The results will tell you if the page was crawled and whether it is indexed, and a bunch of related information. 

Google Search Console URL Inspection

 

If you don’t have Search Console access, or would simply like to do another test, you can use Google’s Rich Results Test tool. Just pop the URL in and wait for the results to appear (it may take quite a while). 

 

rich results screenshot
You can also try using the site: command in Google search to see if a page has been indexed. If it has, it has also successfully been crawled.

site: command in google screenshot

You can also use SEO tools such as Ahrefs and Screaming Frog to check the crawlability of a website’s link. And of course, our own Link Whisper will help you stay on top of all of your internal and external links. 

How To Make Your Links Crawlable

These are the boxes you need to tick to make sure you links are crawlable:

  • Use the href attribute correctly
  • Add all of your pages to the correct sitemap
  • Make sure every page on your website has incoming internal links and that you use the correct anchor text for them 
  • Keep all of your internal links dofollow
  • Make sure the page is not disallowed in robots.txt
  • Reduce the number of clicks it takes to get from the homepage to other pages  

Note that while you should allow Google and other search engine crawlers access to your website, you can block other bots from crawling your pages, if you so choose. This can help you save bandwidth and prevent others from accessing information you would rather keep private, like your website’s structure. 

How Link Whisper Can Help Your Website’s Crawlability 

Now that you understand the basics of crawlability, you know just how important it is to help search engines out by establishing a clear page hierarchy and providing plenty of internal links to your most important pages. 

Use Link Whisper to easily manage your internal links, their anchor text and check for any broken links or 404 pages. It will save you not only a lot of time, but all the annoyance that usually comes with a broken or poorly chosen internal link. 

Build A Powerful Internal Link Strategy Today

Enter your email & we'll send you 8 tips to build an internal link strategy.

And 3 things you should avoid doing with internal links.

    We won't send you spam. Unsubscribe at any time.

    17 of the Best SEO Chrome Extensions to Help You Master SEO

    Contents1 The Best SEO Chrome Extensions1.1 MozBar1.2 GrowthBar SEO Chrome Extension1.3 Meta SEO Inspector1.4 Mangools1.5 Hreflang Tag Checker1.6 SEOquake1.7 Rank Up Content1.8 AIPRM for ChatGPT1.9 SEO Pro by Marketing Syrup1.10 SEO Meta in 1 Click1.11 SEO Analysis and Website Review…

    Read More

    What Is Pogo Sticking?

    Contents1 What Does Pogo Sticking Say About My Site?2 3 Main Causes of Pogo Sticking2.1 Your Website Has Clickbait2.2 Your Content Is Hard to Access2.3 Your Website Has Poor UX Design3 How To Discourage Pogo Sticking3.1 Use a Table of…

    Read More

    Get Started with LinkWhisper

    Speed Up the Process of Internal Linking and Help You Rank Better in Google

    Get LinkWhisper Now