Crawlable Links: What Are They And Why Are They Important?
Contents
Whether or not you understand how the internet works (and whether or not you even want to), becoming familiar with the concept of crawlability can help you ensure that your website is properly indexed by search engines.
Since indexing is the necessary prerequisite to receiving organic traffic, you can already gauge the importance of ensuring your links can be crawled.
Here’s what you need to know about crawlable links and their importance:
What Are Crawlable Links?
A crawlable link is a link that can be followed by search engine crawlers.
Links that are not crawlable can, for example, be put to use by the JavaScript code on a page, but they can’t be used by crawlers.
Crawlability, then, is the ability of said crawlers to access your website and all of its resources. It’s not to be confused with indexability, which is the ability of a search engine to analyze your pages and add them to its index.
In order for your links to first be crawlable and then indexable, you need to use the <a> tag with the href attribute, followed by a real, live URL.
What Does a Crawlable Link Look Like?
Here’s an example of a crawlable link:
<a href=”https://linkwhisper.com”>
Note the space between the opening of the <a> tag, and the lack of space between the href, the equals sign and the open quotation marks. Note the same lack of space between the closed quotation marks and the closing of the tag <a>.
Only links in this format will work and be crawlable. If you add or omit a space, they won’t.
Here’s what Google says they can crawl:
What Does a Non-Crawlable Link Look Like?
A non-crawlable link can look like anything other than the above.
For example, all of these links are non crawlable:
<a href=https://linkwhisper.com”>
<a href=”https://linkwhisper.com>
<a =”https://linkwhisper.com”>
<a https://linkwhisper.com>
Google themselves list these examples:
Why Is Link Crawlability Important?
In order for a website’s pages to be properly indexed and ranked, they first need to be crawled. If your links aren’t crawlable, search engines won’t be able to analyze their content, add them to their index, and determine how high they should rank.
There are very rare cases where Google can index a URL that has not been crawled. They will use the URL text and the anchor text of its backlinks to do it, but the page’s title and description will not show up in search results.
Note that crawlability impacts not just Google’s crawlers, but the crawlers of all search engines.
SEO tools like Ahrefs, SEMRush and Screaming Frog also use crawlers to scan and analyze websites, and they won’t have access to your non-crawlable links either. This can impact the results of their crawls and analyses.
Factors That Can Affect Crawlability
Certain factors can directly influence the crawlability of a link. They include:
- Discoverability: Pages that can’t be discovered can’t be crawled. If a page has no internal links and is therefore an orphan page, and if it can’t be found in the sitemap, it won’t be crawled, and consequently indexed or ranked.
- Nofollow attribute: Links that are marked as “rel=nofollow” are not followed by Google’s crawlers. Well, at least they shouldn’t be. Google themselves say they treat this attribute as a hint, so they may or may not choose to follow nofollow links. However, in theory, if a page only has one nofollow incoming link, it might not get crawled.
Robots.txt file: If you have disallowed a part of your website in your robots.txt file, Google’s crawlers won’t access it, so they will not be able to crawl it.
How Do You Know If a Link is Crawlable?
Google themselves give you two ways to check if a link is crawlable.
If you have Search Console access to the website whose crawlability you want to check, sign in and add the URL you want to test to the search bar at the top.
The results will tell you if the page was crawled and whether it is indexed, and a bunch of related information.
If you don’t have Search Console access, or would simply like to do another test, you can use Google’s Rich Results Test tool. Just pop the URL in and wait for the results to appear (it may take quite a while).
You can also try using the site: command in Google search to see if a page has been indexed. If it has, it has also successfully been crawled.
You can also use SEO tools such as Ahrefs and Screaming Frog to check the crawlability of a website’s link. And of course, our own Link Whisper will help you stay on top of all of your internal and external links.
How To Make Your Links Crawlable
These are the boxes you need to tick to make sure you links are crawlable:
- Use the href attribute correctly
- Add all of your pages to the correct sitemap
- Make sure every page on your website has incoming internal links and that you use the correct anchor text for them
- Keep all of your internal links dofollow
- Make sure the page is not disallowed in robots.txt
- Reduce the number of clicks it takes to get from the homepage to other pages
Note that while you should allow Google and other search engine crawlers access to your website, you can block other bots from crawling your pages, if you so choose. This can help you save bandwidth and prevent others from accessing information you would rather keep private, like your website’s structure.
How Link Whisper Can Help Your Website’s Crawlability
Now that you understand the basics of crawlability, you know just how important it is to help search engines out by establishing a clear page hierarchy and providing plenty of internal links to your most important pages.
Use Link Whisper to easily manage your internal links, their anchor text and check for any broken links or 404 pages. It will save you not only a lot of time, but all the annoyance that usually comes with a broken or poorly chosen internal link.