Tag: Crawl directives

Which pages to noindex or nofollow on your site?

Michiel Heijmans

Some of the pages of your site serve a purpose, but that purpose isn’t ranking in search engines or even getting traffic to your site. These pages need to be there, as glue for other pages or simply because regulations require them to be accessible on your website. If you regularly read our blog, you’ll know how noindex or nofollow can help you deal with these pages. However, if you are new to these terms, please read on and let me explain what they are, and what pages they might apply to!

What is noindex nofollow?

noindex means that a web page shouldn’t be indexed by search engines and therefore shouldn’t be shown on the search engine’s result pages. nofollow means that search engines spiders shouldn’t follow the links on that page. You can add these values to your robots meta tag. The robots meta tag is a piece of code in the head section of a web page. It tells search engines how to crawl and whether to index a page.

Our ultimate guide on the robots meta tag is a great read if you want to take a bit of a deeper dive into this subject.

In short:

  • The robots meta tag looks like this in most cases:
    <meta name="robots" content="[VALUE1,VALUE2]">
  • VALUE1 and VALUE2 are set to index, follow by default, meaning the page at hand can be indexed by search engines and links on that page can be followed to crawl the pages they link to.
  • VALUE1 and VALUE2 can be set to noindex, nofollow or another combination like index, nofollow .

But don’t let this code scare you away. Yoast SEO helps you out! If you want to know how to noindex a post in WordPress, in a super-easy way, you should read this post: How to noindex a post in WordPress: the easy way.

But when should you use which value?

Pages to set to noindex

Author archives on a one-author blog

If you are the only one writing for your blog, your author pages are probably 90% the same as your blog homepage. That’s of no use to Google and can be considered duplicate content. To prevent this kind of duplicate content you can choose to disable the author archive entirely. Here’s how to enable or disable it easily with Yoast SEO. If, for some reason, you’d like to keep it on your site, but out of the search results, you can noindex it. Fortunately, with Yoast SEO, this is not very difficult either; just check how to noindex an author archive.

Certain (custom) post types

Sometimes a plugin or a web developer adds a custom post type that you don’t want to be indexed. At Yoast, for example, we use custom pages for our products, as we are not a typical online store selling physical products. So, we don’t need a product image, filters like dimensions and technical specifications on a tab next to the description. Therefore, we noindex the regular product pages WooCommerce outputs and are using our own pages. Indeed, we noindex the product post type.

Relatedly, we’ve seen eCommerce solutions that added specifications like dimensions and weight as a custom post type as well. These pages are considered to be low-quality content. You’ll understand that these pages have no use for a visitor or Google, so need to be kept out of the search result pages too.

Thank you pages

That page serves no other purpose than to thank your customer/newsletter subscriber/first-time commenter. These pages are usually thin content pages, with upsell and social share options, but no value for someone using Google to find useful information. Therefore, those pages shouldn’t be in the search results pages.

Admin and login pages

Most login pages shouldn’t be in Google. But these are. Keep yours out of the index by adding a noindex to it. Exceptions are the login pages that serve a community, like Dropbox or similar services. Just ask yourself if you would google one of your login pages if you were not in your company. If not, it’s probably safe to say that Google doesn’t need to index these login pages. Luckily, if you are running WordPress, you’re safe as the CMS noindexes your site’s login page automatically.

Internal search results

Internal search results are pretty much the last pages Google would want to send its visitors to. If you want to ruin a search experience, you link to other search pages, instead of an actual result. But the links on a search result page are still very valuable, you definitely want Google to follow them. So, all links should be followed, and the robots meta setting should be:
<meta name="robots" content="noindex, follow">

Yoast SEO makes sure your internal search pages are set to noindex by default. It’s one of Yoast SEO’s hidden features. This is not an editable setting, because it’s simply how it should be done according to the Google Guidelines, and we fully agree with them on this.

For developers only: If you do want to change this, this can be done by using one of our filters. An example can be found here.

Pages to set to nofollow

For all the examples mentioned above, there is no need to nofollow all the links on these pages. You don’t want to show them in the search results, but you do want Google to follow the links on the page. Now, when should you add a nofollow to your robots meta tag?

If you set a page to nofollow with a robots meta tag, none of the links on that page will be followed. Google came up with nofollow to be able to distinguish links to untrusted content (or, later on, paid for, like advertisements). On a regular website, there are probably very few pages you’d want Google not to follow any link.

An example: if you have a page that lists SEO books, with a surplus of Amazon affiliate links, these could be of value to your site for your users. But I’d nofollow that entire page if there’s nothing else that matters on the page. You might have it indexed, though. Just make sure you cloak your links the right way.

Nofollow single links

If you have a post or page with multiple links you might want to help search engines qualify them. Nowadays, you can nofollow a single link to, or even set it to sponsored or UGC. Adding the right rel attributes to your link allows you to do so. For instance, a link to an advertisement would look like this:  <a href="https://www.example.com" rel="nofollow sponsored">example link</a>. With Yoast SEO adjusting these rel attributes is super easy, as you can see in this video:

Conclusion

As we have seen, whether or not to noindex a page or nofollow a link boils down to two questions: do you want this page to show up in the search results pages and should search engines follow the links on this page? For ‘thank you’ pages or login pages, for example, the answer to the first questions is “no”. For a page with loads of affiliate links, the answer to the second question is “no”. Keep the examples from this post in mind and you shouldn’t have any more trouble deciding the answers for your own site!

PS. Did you noindex a post or page while you didn’t mean to? No worries, as you can fix an accidental noindex easily!

Read more: How to noindex a post »

The post Which pages to noindex or nofollow on your site? appeared first on Yoast.

What are sponsored, nofollow and ugc links, and why use them?

Willemien Hallebeek

Links are an important part of SEO. Without links, Google (or other search engines) may not discover your pages, or might not think that they’re important. Sometimes, though, you might want Google not to follow a link. Or you might want to tell them a particular is sponsored, or added to your page by a user. Why’s that? And how do you implement this on your website? Learn all about sponsored, nofollow and ugc links here!

Links and SEO

When you link to another website, search engines may count that as a ‘vote’ for the page you’re linking to. Pages which have many such ‘votes’, from authoritative and trusted websites, may rank higher in the search results as a result (as they, in turn, become more authoritative and trustworthy). That makes links a kind of currency.

That’s why a good SEO strategy should always consider how the types of content, marketing and PR that you do will encourage other websites to link to you. If you’re not already thinking about how your site can earn links from others, our guide to link building tips and tactics is a good place to get inspiration on where to start.

Link building

In the past, but still even today, people try to game the system by buying links. Obviously, that’s not the way to go; Google’s penguins might come after you! That’s why we recommend holistic link building, which boils down to creating great resources for your audience and reaching out to get the word out, eventually leading to more links.

But, what happens if you want to link to a page, without voting for it? And, what stops people from finding ways to cheat the system, such as posting links to their site on your website; on comment forms, forums, or social media profiles?

In these cases, we need to use a special type of link, to tell search engines that it shouldn’t be trusted.

The nofollow attribute

In the early days of SEO, many unscrupulous marketers realised that they could easily get hundreds of links to their pages by leaving spam comments on other blogs, by buying links from webmasters, or from placing links on any site which allowed user-submitted content.

To combat this, in 2005 Google introduced a way to mark a link as untrusted; specifically, a way of saying “don’t follow this link”. By adding a nofollow attribute to your links, they’d no longer count as votes. It also became Google’s policy that any link which is paid for (typically an advert, paid placement, or similar) should use a nofollow attribute to indicate that it shouldn’t affect their ranking calculations.

That’s because paid links are the same as a ‘vote’ for a page. For instance, if someone pays you to put an ad on your website, you might send some visitors to the advertised page or product. Since it’s not a natural endorsement, link value shouldn’t pass on to this particular page; search engines shouldn’t rank it higher because you’ve received some kind of compensation for that link.

This also made it possible to link to a page which you don’t endorse, but you still want to use it as an example in your copy (e.g., “I tried this product, but it was horrible”).

Today, almost all comment systems and social media platforms automatically add a nofollow attribute to user-submitted content.

What does that look like?

Let’s take a closer look at a link. In HTML, a plain link looks like this: <a href="https://www.example.com">example link</a>. You probably use these types of links a lot throughout your content. You use them to point readers to interesting, related content on your own site or someone else’s website.

If you want to indicate that you don’t trust the site you’re linking to, or that it’s a paid placement, including the nofollow attribute would look like this: <a href="https://www.example.com" rel="nofollow">example link</a>.

Internal links

So far, we’ve only considered whether external links should be nofollow’d. In some cases, it might also make sense to mark an internal link with a nofollow attribute. In Yoast SEO, we automatically add a nofollow attribute to internal links which point to your login or registration pages. This prevents Google from wasting resources crawling and evaluating those pages.

Nofollow doesn’t always mean “don’t follow”

It’s important to understand that most search engines treat nofollow as a ‘hint’, and might follow them whilst still ‘devaluing’ them. An announcement from Google in September 2019 clarified this:

Links contain valuable information that can help us improve search, such as how the words within links describe the content they point at. Looking at all the links we encounter can also help us better understand unnatural linking patterns. By shifting to a hint model, we no longer lose this important information, while still allowing site owners to indicate that some links shouldn’t be given the weight of a first-party endorsement.

Danny Sullivan, Google

What are sponsored and ugc links?

In September 2019, Google announced two new types of link attribute. It’s now possible to mark links as sponsored or ugc (short for ‘user-generated content), as well as nofollow. They explained that:

  • The sponsored attribute should be used to identify links which are specifically the result of paid placement; e.g., sponsored placements, advertorials, paid links, and similar.
  • The ugc attribute should be used to identify links which are created by users (e.g., author links in a comment form), which therefore aren’t necessarily trusted or endorsed by the page’s author.

In both cases, these work similarly to the original nofollow attribute – they tell Google note to count the link as a ‘vote’. We don’t know precisely how Google uses this data internally, but they’ve hinted that it’ll help them understand more about the link. That might improve how they count ‘votes’ and evaluate pages.

What does that look like?

That means that we have four different types of HTML markup for links:

  • A normal link, with no rel attribute
  • A nofollow link: <a href="https://www.example.com" rel="nofollow">example link</a>
  • A sponsored link: <a href="https://www.example.com" rel="sponsored">example link</a>
  • A user-generated content link: <a href="https://www.example.com" rel="ugc">example link/a>

Combining attributes

Whilst each of these attributes describe different types of links, it’s possible to combine various rel attributes in one link. For instance, a sponsored and nofollow attribute can exist in one link: <a href="https://www.example.com" rel="nofollow sponsored">example link</a>.

This is useful, because not all search engines support the two new rel attributes, so it’s best practice to use the nofollow attribute along with the sponsored and ugc attribute.

So, now you know what these links and rel attributes look like. But why and when should you use them?

When should you use which attribute?

The sponsored attribute

An advertisement or link you get paid for or in any other way should use the sponsored attribute. The reasoning behind this is that Google sees links to a page as an endorsement; you link to an article because it’s a valuable resource you’d like to point your users to. When you get paid to place a reference to another website your motivation is different. It might be something you wouldn’t link to without compensation. With the sponsored attribute Google can differentiate these “unnatural links” from normal links.

As other search engines won’t recognise this sponsored attribute (yet), we do recommend to add the nofollow attribute to this type of link as well.

The UGC attribute

You should use the ugc attribute whenever users of your website are able to create content or links on it; e.g., in the comment section on your site. If you’re on WordPress, there’s no need to worry about this attribute; WordPress automatically adds a ugc attribute, as well as a nofollow attribute – a specific request from our team – to the links in the comment section on your site.

The nofollow attribute

As not all search engines support the sponsored or ugc attribute (yet) you should still add the nofollow attribute to both these type of links as well.

Creating sponsored or nofollow links in WordPress

While this might sound a bit complicated when you’re not an HTML native, qualifying links is simple with the WordPress block editor and Yoast SEO. Since Yoast SEO 14.4 we’ve added an option to easily add a sponsored or nofollow attribute to a link in your content.

If you want to nofollow a link or qualify it as sponsored (and nofollow at the same time), click on the link icon, paste your link and you’ll see these options:

Adjust a link setting in Yoast SEO: add nofollow or sponsored to you link with a slider

Select the option of your choice by moving the slider and you’re done!

Rather watch a video? Check this out:

You’ll find more about this on our help page on link settings. Good luck!

Read more: Do outbound links matter for SEO? »

The post What are sponsored, nofollow and ugc links, and why use them? appeared first on Yoast.

Yoast SEO & Ryte: Checking your site’s indexability

Edwin Toonen

Your site needs to be up and running if you want to be found in search engines. If you aren’t blocking anything — deliberately or accidentally — search engine spiders can crawl and index it. You probably know that Yoast SEO has lots of options to determine what does and doesn’t need to be indexed, but did you know it also has a tool that monitors your site’s indexability? This is the indexability check, provided by our good friends at Ryte.

Update: In Yoast 13.2, we moved the Ryte indexability check from the Yoast meta box on your Dashboard. You can now find the Ryte indexability check in the WordPress Site Health tool!  In WordPress 5.4 the site health status widget also appears on the Dashboard.

What does it do?

The indexability check checks if your site is indexable and does so on a weekly basis. You can find the Ryte indexability check in the Site Health tool in WordPress. The Site Health tool groups its checks into three sections:

  • critical issues
  • recommended improvements
  • and passed tests.

The Ryte check will be displayed within one of these groups. When your site can be found by search engines, the Ryte check appears in the Passed tests section of the Site Health tool. If something is wrong and it needs your attention, the Ryte check appears with an SEO tag next to the issue. 

The Ryte check in the WordPress Site Health tool

When you see a Ryte check warning among the critical issues, it means that your site cannot be found by search engines. You need to act on this issue quickly if you want your content to appear in the search engines.

In some cases, an error occurs while checking whether your site can be found by search engines. Or Ryte might not be able to determine if your site is findable. When this happens, we recommend you check what is causing those issues.

What do I have to do to get it?

We add this check automatically when you install Yoast SEO. You can find it by going to the WordPress admin menu item Tools > Site Health. In WordPress 5.4 you can also find a Site Health Status widget on the Dashboard.

We advise you to keep the Ryte check turned on. But we do give you the option to turn it off. Learn how to do this by following the steps in this post. 

Error: your site cannot be found by search engines

Your homepage cannot be indexed by search engines. This is very bad for SEO and should be fixed.

Are you getting the error message shown above? When Yoast SEO reports your site as not indexable by search engines, this can be caused by several reasons. However, it is an issue of the highest priority, assuming that you want your site to be indexed by Google.

First of all, test your URL in Google’s mobile-friendly test here. If Google can run the mobile-friendly test for your domain, Google can also crawl the domain. In that case, the Ryte warning is wrong.

If Google cannot run the mobile-friendly test, our advice is to follow this Go to Ryte to analyze your entire site link, create an account on Ryte (formerly OnPage.org) and see what they have to tell you. One possibility is that you have a robots.txt file that’s blocking indexation. Or perhaps your theme has some code in it that’s blocking search engines. Ryte can tell you what the problem is and how to fix it!

Indexability check fails

If your indexability check fails, your server is not able to connect to the ryte.com servers to retrieve the indexability status for your domain. This can happen for several reasons:

  • Your server can’t connect to other servers at all.
  • Your server has a whitelist of servers it’s allowed to connect to. In this case, adding ryte.com would fix that.
  • A myriad of other server configuration reasons.

Usually, your hosting party can tell you the cause of the issues and will also be able to help you fix it. If not, go to Ryte where you can create a free account and have access to all of the benefits without having to change anything on your server.

The Yoast Ryte check

Ryte & Yoast SEO

Ryte offers a free indexability check for Yoast SEO users. This way, you can quickly see whether your site is still reachable for search engines and visitors. If you need help fixing technical SEO issues or if you are in need of a suite of SEO tools to help you fix or improve your rankings, sign up for the free Ryte introductory plan. Just go to Ryte and follow the instructions!

Read more: SEO basics: What is crawlability »

The post Yoast SEO & Ryte: Checking your site’s indexability appeared first on Yoast.

Help, I’ve accidentally noindexed a post. What to do?

Willemien Hallebeek

It can happen to anyone: You’re working on your site, fiddling on some posts here and there, and hit update when you’re done. After a while, you check back on how a post is doing and, to your dismay, it disappeared completely from the search engines! It turns out you’ve accidentally set a post or page to noindex on your site. Here, we’ll share a – pretty funny – story about how it happened to us, and what to do if you’ve made a similar mistake.

How to unintentionally noindex a post

Let’s start with a short story. We have a post called Noindex a post in WordPress, the easy way on yoast.com. In this post, we – surprise, surprise – explain how to noindex a post with Yoast SEO in WordPress. To show how easy that is, we added some screenshots of setting a post to noindex. A picture says more than a thousand words, right?

setting a post to noindex with Yoast SEO
Some of the copy and the screenshot in the ‘How to noindex a post’ post

Now, we’ll reveal a little secret. Oftentimes, when we want to illustrate a certain feature of Yoast SEO, we’ll just take a screenshot of that feature in the post we’re working on. So, in this case, we went to the Advanced tab in the Yoast SEO meta box, clicked No in the dropdown, took a screenshot, and added the screenshot to our post. We checked the copy we’ve written, added images, checked the SEO and readability scores and previewed our post. All looked fine, so we hit publish, shared it on social and in our newsletter and went on with other tasks.

Sometime later, we were checking how our content performed on the query [how to noindex a post] in Google. Surprisingly, we didn’t encounter this article, while we were pretty sure we already had a post like this. We started looking for it in our post overview, and there it was! Waiting in vain for visitors to learn more about this handy feature of Yoast SEO.

So, while we were happily typing away, making sure people understand what this feature is about, we forgot one thing… removing the noindex from this post. Therefore, accidentally and ironically, we’ve set our post about setting posts to noindex to… noindex.

How to reverse noindexing a post

In our case, reverting the noindex on that post wasn’t very difficult. The post, although it describes a nifty feature of Yoast SEO, wasn’t crucial for our business. Therefore, we decided to just remove the noindex and republish and share it again. But there’s more you can do; the options to get your article back in the search engines are listed here below. Depending on the severity of the issue you can choose to follow all steps or select some of them.

1. Remove the noindex tag

This is an essential step. You can easily remove the noindex tag by Google and other search engines in the Advanced tab of the Yoast SEO meta box. Just click on Yes here and you’ve removed the noindex tag:

Remove the noindex tag in the Yoast SEO meta box

In the search appearance section, you can set multiple posts or pages on your site to noindex. If you did that by accident or forgot to reverse that after temporarily setting it to noindex, you can set it to index again there too:

Remove the noindex tag on a post type in the Search appearance section of Yoast SEO

If you’ve added a meta robots tag in the code to noindex your post, please remove it from the code. There’s no need to set it to index though since that is the default value when nothing is set.

2. Google Search Console is your friend

If you’ve accidentally noindexed a valuable post or maybe even an important part of your website, there are some things you can do to make Google retrieve your content faster. Google Search Console can help you do this. So if you didn’t sign up for Google Search Console yet, now’s the time to do it. Yoast SEO will help you to verify your site, as you can read in this guide on how to add your site to Google Search Console.

Request for reindexing of a URL

In the URL Inspection Tool of Google Search Console there’s an option to ask Google to crawl or recrawl a URL. This might speed up the process and allows you to follow the progress. There is a quota for submitting individual URLs with this tool. So, if you’ve noindexed (a part of) your site it might be wise to select the posts or pages that are most crucial for your business and request to index those again.

Resubmit your XML sitemap

Another option is to resubmit your XML sitemap in Google Search Console. If you’re using Yoast SEO you don’t have to worry about this though. In that case, when you publish or update content on your site, Yoast SEO automatically pings Google with your sitemap.

If you didn’t submit your XML sitemap to Google Search Console yet, you’ll find a step-by-step guide to submitting your XML sitemap to GSC here.

3. Republish and share it again

Lastly, you can share the reindexed content in your newsletter, on social and other marketing channels. This way, you’ll generate some traffic and exposure, especially if other people start sharing it too. In case of a blog post, you can republish it on your blog. If it considers important pages of your site quickly thinking up a campaign and publishing new blog posts that link to the reindexed content could also help to get the initial traffic and rankings back.

It’s not the end of the world

Finding out you’ve accidentally set a post or even (parts of) your site might give you a big scare. But, fortunately, it’s not the end of the world and there are various things you can do to get it back in the search engines again. Depending on the size of the issue and the frequency your site gets crawled, it will take some time to recover, but eventually, it probably will.

Now, let’s hope I haven’t accidentally set anything to noindex when creating this post…

Read more: The ultimate guide to the meta robots tag »

The post Help, I’ve accidentally noindexed a post. What to do? appeared first on Yoast.

Robots meta changes for Google

Joost de Valk

Sometimes Google does announcements about new features and we go “huh, why did they do that?” This week we had one of those. Google introduced a new set of robots meta controls, that allows sites to limit the display of their snippets in the search results. There is a reason for that, but they buried that reason far, far away.

With their newly introduced robots meta controls, you can “say” things like: I don’t want you to show more than 200 characters from my site! Or: I don’t want you to show images from my site. None of this made sense to us. Why would anyone who truly wants to optimize for Google’s search results, do any of that?

A French copyright law

Suddenly, we make a dramatic plot twist: France turns out to have introduced a new copyright law, which is the first implementation of Europe’s new copyright law. I am not a lawyer, but as Google explains it in this blog post, it requires for publishers to opt in to a display of their snippets, whereas in the past you had to opt out. Note that that blog post is in French, on their French blog, but there’s an English version in the bottom of that post.

It turns out that you can also use these new robots meta features to opt in to all those displays. Basically, use the meta robots to opt in to have your snippet text, videos and images being displayed at all.

robots meta?

Robots meta is hard enough as it is, read our ultimate guide to understand what you can do with it.

Because this French law implements the new European law, chances are that the other implementations of this copyright law across Europe are going to require similar action. This would mean more countries would need to opt in if they don’t want to loose their snippets. We don’t know that, of course, but of course: better safe than sorry.

A simple change to Yoast SEO

Because this potentially affects so many countries, we’ve decided to make a change for all Yoast SEO users. Every page will have the following robots meta bit added to its robots meta string if that page was not already set to noindex or nosnippet:

max-snippet:-1, max-image-preview:large, max-video-preview:-1

This makes sure that nobody runs into unexpected surprises and we’re ready for implementations of this law around Europe.

I want to change that robots meta value

If you want to change that robots meta value, we have a filter in our code that allows you to programmatically change that. That filter is called wpseo_robots, should you want to play with it.

You are opting in by using Yoast SEO

We realize that this means that we’re opting you in to all of these snippet features in Yoast SEO. I think it’s fair to say that if you use Yoast SEO to optimize your search results, we can assume that that’s what you want. At the same time, you might want somewhat more granular control over these values, if that’s true, please let us know in the comments!

The Yoast SEO release containing this change will be 12.2. This will be released next Tuesday, October 1.

The post Robots meta changes for Google appeared first on Yoast.

What’s technical SEO? 8 technical aspects everyone should know

Willemien Hallebeek

An SEO Basics post about technical SEO might seem like a contradiction in terms. Nevertheless, some basic knowledge about the more technical side of SEO can mean the difference between a high ranking site and a site that doesn’t rank at all. Technical SEO isn’t easy, but here we’ll explain – in layman’s language – which aspects you should (ask your developer to) pay attention to when working on the technical foundation of your website.

What is technical SEO?

Technical SEO refers to improving the technical aspects of a website in order to increase the ranking of its pages in the search engines. Making a website faster, easier to crawl and understandable for search engines are the pillars of technical optimization. Technical SEO is part of on-page SEO, which focuses on improving elements on your website to get higher rankings. It’s the opposite of off-page SEO, which is about generating exposure for a website through other channels.

Why should you optimize your site technically?

Google and other search engines want to present their users with the best possible results for their query. Therefore, Google’s robots crawl and evaluate web pages on a multitude of factors. Some factors are based on the user’s experience, like how fast a page loads. Other factors help search engine robots grasp what your pages are about. This is what, amongst others, structured data does. So, by improving technical aspects you help search engines crawl and understand your site. If you do this well, you might be rewarded with higher rankings or even rich results.

It also works the other way around: if you make serious technical mistakes on your site, they can cost you. You wouldn’t be the first to block search engines entirely from crawling your site by accidentally adding a trailing slash in the wrong place in your robots.txt file.

But it’s a misconception you should focus on technical details of a website just to please search engines. A website should work well – be fast, clear and easy to use – for your users in the first place. Fortunately, creating a strong technical foundation often coincides with a better experience for both users and search engines.

What are the characteristics of a technically optimized website?

A technically sound website is fast for users and easy to crawl for search engine robots. A proper technical setup helps search engines to understand what a site is about and it prevents confusion caused by, for instance, duplicate content. Moreover, it doesn’t send visitors, nor search engines, into dead-end streets by non-working links. Here, we’ll shortly go into some important characteristics of a technically optimized website.

1. It’s fast

Nowadays, web pages need to load fast. People are impatient and don’t want to wait for a page to open. In 2016 already, research showed that 53% of mobile website visitors will leave if a webpage doesn’t open within three seconds. So if your website is slow, people get frustrated and move on to another website, and you’ll miss out on all that traffic.

Google knows slow web pages offer a less than optimal experience. Therefore they prefer web pages that load faster. So, a slow web page also ends up further down the search results than its faster equivalent, resulting in even less traffic.

Wondering if your website is fast enough? Read how to easily test your site speed. Most tests will also give you pointers on what to improve. We’ll guide you through common site speed optimization tips here.

2. It’s crawlable for search engines

Search engines use robots to crawl or spider your website. The robots follow links to discover content on your site. A great internal linking structure will make sure that they’ll understand what the most important content on your site is.

But there are more ways to guide robots. You can, for instance, block them from crawling certain content if you don’t want them to go there. You can also let them crawl a page, but tell them not to show this page in the search results or not to follow the links on that page.

Robots.txt file

You can give robots directions on your site by using the robots.txt file. It’s a powerful tool, which should be handled carefully. As we mentioned in the beginning, a small mistake might prevent robots from crawling (important parts of) your site. Sometimes, people unintentionally block their site’s CSS and JS files in the robot.txt file. These files contain code that tells browsers what your site should look like and how it works. If those files are blocked, search engines can’t find out if your site works properly.

All in all, we recommend to really dive into robots.txt if you want to learn how it works. Or, perhaps even better, let a developer handle it for you!

The meta robots tag

The robots meta tag is a piece of code that you won’t see on the page as a visitor. It’s in the source code in the so-called head section of a page. Robots read this section when finding a page. In it, they’ll find information about what they’ll find on the page or what they need to do with it.

If you want search engine robots to crawl a page, but to keep it out of the search results for some reason, you can tell them with the robots meta tag. With the robots meta tag, you can also instruct them to crawl a page, but not to follow the links on the page. With Yoast SEO it’s easy to noindex or nofollow a post or page. Learn for which pages you’d want to do that.

Read more: https://yoast.com/what-is-crawlability/

3. It doesn’t have (many) dead links

We’ve discussed that slow websites are frustrating. What might be even more annoying for visitors than a slow page, is landing on a page that doesn’t exist at all. If a link leads to a non-existing page on your site, people will encounter a 404 error page. There goes your carefully crafted user experience!

What’s more, search engines don’t like to find these error pages either. And, they tend to find even more dead links than visitors encounter because they follow every link they bump into, even if it’s hidden.

Unfortunately, most sites have (at least) some dead links, because a website is a continuous work in progress: people make things and break things. Fortunately, there are tools that can help you retrieve dead links on your site. Read about those tools and how to solve 404 errors.

To prevent unnecessary dead links, you should always redirect the URL of a page when you delete it or move it. Ideally, you’d redirect it to a page that replaces the old page. With Yoast SEO Premium, you can easily make redirects yourself. No need for a developer!

Read more: https://yoast.com/what-is-a-redirect/

4. It doesn’t confuse search engines with duplicate content

If you have the same content on multiple pages of your site – or even on other sites – search engines might get confused. Because, if these pages show the same content, which one should they rank highest? As a result, they might rank all pages with the same content lower.

Unfortunately, you might have a duplicate content issue without even knowing it. Because of technical reasons, different URLs can show the same content. For a visitor, this doesn’t make any difference, but for a search engine it does; it’ll see the same content on a different URL.

Luckily, there’s a technical solution to this issue. With the so-called, canonical link element you can indicate what the original page – or the page you’d like to rank in the search engines – is. In Yoast SEO you can easily set a canonical URL for a page. And, to make it easy for you, Yoast SEO adds self-referencing canonical links to all your pages. This will help prevent duplicate content issues that you’d might not even be aware of.

5. It’s secure

A technically optimized website is a secure website. Making your website safe for users to guarantee their privacy is a basic requirement nowadays. There are many things you can do to make your (WordPress) website secure, and one of the most crucial things is implementing HTTPS.

HTTPS makes sure that no-one can intercept the data that’s sent over between the browser and the site. So, for instance, if people log in to your site, their credentials are safe. You’ll need a so-called SSL certificate to implement HTTPS on your site. Google acknowledges the importance of security and therefore made HTTPS a ranking signal: secure websites rank higher than unsafe equivalents.

You can easily check if your website is HTTPS in most browsers. On the left hand side of the search bar of your browser, you’ll see a lock if it’s safe. If you see the words “not secure” you (or your developer) have some work to do!

Read more: SEO Basics: What is HTTPS?

6. Plus: it has structured data

Structured data helps search engines understand your website, content or even your business better. With structured data you can tell search engines, what kind of product you sell or which recipes you have on your site. Plus, it will give you the opportunity to provide all kinds of details about those products or recipes.

Because there’s a fixed format (described on Schema.org) in which you should provide this information, search engines can easily find and understand it. It helps them to place your content in a bigger picture. Here, you can read a story about how it works and how Yoast SEO helps you with that.

Implementing structured data can bring you more than just a better understanding by search engines. It also makes your content eligible for rich results; those shiny results with stars or details that stand out in the search results.

7. Plus: It has an XML sitemap

Simply put, an XML sitemap is a list of all pages of your site. It serves as a roadmap for search engines on your site. With it, you’ll make sure search engines won’t miss any important content on your site. The XML sitemap is often categorized in posts, pages, tags or other custom post types and includes the number of images and the last modified date for every page.

Ideally, a website doesn’t need an XML sitemap. If it has an internal linking structure which connects all content nicely, robots won’t need it. However, not all sites have a great structure, and having an XML sitemap won’t do any harm. So we’d always advise having an XML site map on your site.

8. Plus: International websites use hreflang

If your site targets more than one country or countries where the same language is spoken, search engines need a little help to understand which countries or language you’re trying to reach. If you help them, they can show people the right website for their area in the search results.

Hreflang tags help you do just that. You can define for a page which country and language it is meant for. This also solves a possible duplicate content problem: even if your US and UK site show the same content, Google will know it’s written for a different region.

Optimizing international websites is quite a specialism. If you’d like to learn how to make your international sites rank, we’d advise taking a look at our Multilingual SEO training.

Want to learn more about this?

So this is technical SEO in a nutshell. It’s quite a lot already, while we’ve only scratched the surface here. There’s so much more to tell about the technical side of SEO! If you want to take a deep-dive into technical SEO, we’d advise our Technical SEO training or Structured data training. With these courses, you’ll learn how to create a solid technical foundation for your own website.

PS You’re the ambitious type? Get both training courses together and save $59!

Read more: https://yoast.com/wordpress-seo/

The post What’s technical SEO? 8 technical aspects everyone should know appeared first on Yoast.

How to keep your page out of the search results

Michiel Heijmans

If you want to keep your page out of the search results, there are a number of things you can do. Most of ’em are not hard and you can implement these without a ton of technical knowledge. If you can check a box, your content management system will probably have an option for that. Or allows nifty plugins like our own Yoast SEO to help you prevent the page from showing up in search results. In this post, I won’t give you difficult options to go about this. I will simply tell you what steps to take and things to consider.

Why do you want to keep your page out of the search results?

It sounds like a simple question, but it’s not, really. Why do you want to keep your page out of the search results in the first place? If you don’t want that page indexed, perhaps you shouldn’t publish it? There are obvious reasons to keep for instance your internal search result pages out of Google’s search result pages or a “Thank you”-page after an order or newsletter subscription that is of no use for other visitors. But when it comes to your actual, informative pages, there really should be a good reason to block these. Feel free to drop yours in the comments below this post.

If you don’t have a good reason, simply don’t write that page.

Private pages

If your website contains a section that is targeted at, for instance, an internal audience or a, so-called, extranet, you should consider offering that information password-protected. A section of your site that can only be reached after filling out login details won’t be indexed. Search engines simply have no way to log in and visit these pages.

How to keep your page out of the search results

If you are using WordPress, and are planning a section like this on your site, please read Chris Lema’s article about the membership plugins he compared.

Noindex your page

Like that aforementioned “Thank you”-page, there might be more pages like that which you want to block. And you might even have pages left after looking critically if some pages should be on your site anyway. The right way to keep a page out of the search results is to add a robots meta tag. We have written a lengthy article about that robots meta tag before, be sure to read that.

Adding it to your page is simple: you need to add that tag to the <head> section of your page, in the source code. You’ll find examples from the major search engines linked in the robots meta article as well.

Are you using WordPress, TYPO3 or Magento? Things are even easier. Please read on.

Noindex your page with Yoast SEO

The above mentioned content management systems have the option to install our Yoast SEO plugin/extension. In that plugin or extension, you have the option to noindex a page right from your editor.

In this example, I’ll use screenshots from the meta box in Yoast SEO for WordPress. You’ll find it in the post or page editor, below the copy you’ve written. In Magento and TYPO3 you can find it in similar locations.

How to keep your site out of the search results using Yoast SEO

Advanced tab Yoast SEO meta box

Click the Advanced tab in our Yoast SEO meta box. It’s the cog symbol on the left.
Use the selector at “Allow search engines to show this post/page in search results”, simply set that to “No” and you are done.

The second option in the screenshot is about following the links on that page. That allows you to keep your page out of the search results, but follow links on that page as these (internal) links matter for the other pages (again, read the robots meta article for more information). The third option: leave that as is, this is what you have set for the site-wide robots meta settings.

It’s really that simple: select the right value and your page will tell search engines to either keep the page in or out of the search results.

The last thing I want to mention here is: use with care. This robots meta setting will truly prevent a page from being indexed, unlike a robots.txt suggestion to leave a page out of the search result pages. Google might ignore the latter, triggered by a lot of inbound links to the page. 

If you want to read up on how to keep your site from being indexed, please read Preventing your site from being indexed, the right way. Good luck optimizing!

The post How to keep your page out of the search results appeared first on Yoast.

SEO for a new website: the very first things to do

Marieke van de Rakt

How does a new website start ranking? Does it just magically appear in Google after you’ve launched it? What things do you have to do to start ranking in Google and get traffic from the search engines? Here, I explain the first steps you’ll need to take right after the launch of your new website. Learn how to start working on the SEO for a new website!

Optimize your site for search & social media and keep it optimized with Yoast SEO Premium »

Yoast SEO for WordPress pluginBuy now » Info

First: you’ll need to have an external link

One of my closest friends launched a birthday party packages online store last week. It’s all in Dutch and it’s not WordPress (wrong choice of course, but I love her all the same :-)). After my friend launched her website, she celebrated and asked her friends, including me, what they thought of her new site. I love her site, but couldn’t find her in Google, not even if I googled the exact domain name. My first question to my friend was: do you have another site linking to your site? And her answer was ‘no’. I linked to her site from my personal site and after half a day, her website popped up in the search results. The very first step when working on SEO for a new website: getting at least one external link.

Why do you need an external link?

Google is a search engine that follows links. For Google to know about your site, it has to find it by following a link from another site. Google found my friend’s site because I put a link to that site on my personal site. When Google came around to crawl my site after I put the link there, it discovered the existence of my friend’s site. And indexed it. After indexing the site, it started to show the site in the search results.

Read more: ‘What does Google do?’ »

Next step: tweak your settings…

After that first link, your site probably will turn up in the search results. If it doesn’t turn up, it could be that the settings of your site are on noindex or is still blocked by robots.txt. If that’s the case, you’re telling Google not to index your site. Sometimes developers forget to turn either of these off after they finished working on your site.

Some pages are just not the best landing pages. You don’t want people landing on your check out page, for instance. And you don’t want this page to compete with other – useful – content or product pages to show up in the search results. Pages you don’t want to pop up in the search results ever (but there aren’t many of these) should have a noindex.

Yoast SEO can help you to set these pages to noindex. That means Google will not save this page in the index and it’ll not turn op in the search results.

Keep reading: ‘The ultimate guide to the robots meta tag’ »

Important third step: keyword research

My friend’s site now ranks on her domain name. That’s about it. She’s got some work to do to start ranking on other terms as well. When you want to improve the SEO for a new website you have carry out some proper keyword research. So go find out what your audience is searching for! What words do they use?

If you execute your keyword research properly, you’ll end up with a long list of search terms you want to be found for. Make sure to search for those terms in Google yourself. What results are there already? Who will be your online competitors for these search terms? What can you do to stand out from these results?

Read on: ‘Keyword research: the ultimate guide’ »

Learn how to write awesome and SEO friendly articles in our SEO Copywriting training »

SEO copywriting training$ 199 - Buy now » Info

And then: write, write, write

Then you start writing. Write about all those topics that are important to your audience. Use the words you came up with in your keyword research. You need to have content about the topics you want to rank for to start ranking in the search results.

Read more: ‘How to write a high quality and seo-friendly blog post’ »

But also: improve those snippets

Take a look at your results in the search engines once you start ranking (the so called snippets). Are those meta descriptions and the titles of the search results inviting? Are they tempting enough for your audience to click on them? Or should you write better ones?

Yoast SEO helps you to write great titles and meta descriptions. Use our snippet preview to create awesome snippets. That’ll really help in attracting traffic to your site.

Keep reading: ‘The snippet preview: what it means and how to use it?’ »

Think about site structure

Which pages and posts are most important? These should have other pages and posts linking to them. Make sure to link to the most important content. Google will follow your links, the post and pages that have the most internal links will be most likely to rank high in the search engines. Setting up such a structure, is basically telling Google which articles are important and which aren’t. Our brand new text link counter can be a great help to see if you’re linking often enough to your most important content.

Read on: ‘Internal linking for SEO: why and how’ »

Finally: do some link building

Google follows links. Links are important. So get the word out. Reach out to other site owners – preferably of topically related websites – and ask them to write about your new site. If Google follows multiple links to your website, it’ll crawl it more often. This is crucial when you do the SEO for a new website, and will eventually help in your rankings. Don’t go overboard in link building for SEO though, buying links is still a no-go:

Read more: ‘Link building: what not to do?’ »

Preventing your site from being indexed, the right way

Joost de Valk

We’ve said it in 2009, and we’ll say it again: it keeps amazing us that there are still people using just a robots.txt files to prevent indexing of their site in Google or Bing. As a result their site shows up in the search engines anyway. You know why it keeps amazing us? Because robots.txt doesn’t actually do the latter, even though it does prevents indexing of your site. Let me explain how this works in this post.

For more on robots.txt, please read robots.txt: the ultimate guide.

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training$ 199 - Buy now » Info

There is a difference between being indexed and being listed in Google

Before we explain things any further, we need to go over some terms here first:

  • Indexed / Indexing
    The process of downloading a site or a page’s content to the server of the search engine, thereby adding it to its “index”.
  • Ranking / Listing / Showing
    Showing a site in the search result pages (aka SERPs).

So, while the most common process goes from Indexing to Listing, a site doesn’t have to be indexed to be listed. If a link points to a page, domain or wherever, Google follows that link. If the robots.txt on that domain prevents indexing of that page by a search engine, it’ll still show the URL in the results if it can gather from other variables that it might be worth looking at. In the old days, that could have been DMOZ or the Yahoo directory, but I can imagine Google using, for instance, your My Business details these days, or the old data from these projects. There are more sites that summarize your website, right.

Now if the explanation above doesn’t make sense, have a look at this 2009 Matt Cutts video explanation:

If you have reasons to prevent indexing of your website, adding that request to the specific page you want to block like Matt is talking about, is still the right way to go. But you’ll need to inform Google about that meta robots tag.  So, if you want to effectively hide pages from the search engines you need them to index those pages. Even though that might seem contradictory. There are two ways of doing that.

Prevent listing of your page by adding a meta robots tag

The first option to prevent listing of your page is by using robots meta tags. We’ve got an ultimate guide on robots meta tags that’s more extensive, but it basically comes down to adding this tag to your page:

<meta name="robots" content="noindex,nofollow>

The issue with a tag like that is that you have to add it to each and every page.

Or by adding a X-Robots-Tag HTTP header

To make the process of adding the meta robots tag to every single page of your site a bit easier, the search engines came up with the X-Robots-Tag HTTP header. This allows you to specify an HTTP header called X-Robots-Tag and set the value as you would the meta robots tags value. The cool thing about this is that you can do it for an entire site. If your site is running on Apache, and mod_headers is enabled (it usually is), you could add the following single line to your .htaccess file:

Header set X-Robots-Tag "noindex, nofollow"

And this would have the effect that that entire site can be indexed. But would never be shown in the search results.

So, get rid of that robots.txt file with Disallow: / in it. Use the X-Robots-Tag or that meta robots tag instead!

Read more: ‘The ultimate guide to the meta robots tag’ »

Ask Yoast: Block your site’s search results pages?

Joost de Valk

Every website should have a decent internal search functionality that shows the visitors search results that fit their search query. However, those search results pages on your site don’t need to be shown in Google’s search results. In fact, Google advises against this too; it’s not a great user experience to click on a Google search result, just to end up on a search result page of your site. Learn what’s best practice to prevent this from happening!

User experience is not the only reason to prevent Google from including these pages in their search results. Spam domains can also abuse your search results pages, which is what happened to Krunoslav from Croatia. He therefore emailed Ask Yoast:

“Some spam domains were linking to the search results pages on my WordPress site. So what could I do to block Google from accessing my site search results? Is there any code that I could put in robots.txt?”

Check out the video or read the answer below!

Become a technical SEO expert with our Technical SEO 1 training! »

Technical SEO 1 training$ 199 - Buy now » Info

Block your search results pages?

In the video, we explain what you could do to prevent Google from showing your site’s search results:

“Well, to be honest, I don’t think I would block them. What you could do, is try two different things:

1. One is do nothing and run our Yoast SEO plugin. We’ll automatically noindex all the search result pages on your site. But if that leads to weird rankings or to other stuff that is not really working for you, then you could do another thing:

2. The second way is to block them and put a disallow:/?=s* in your robots.txt. This basically means that you’re blocking Google from crawling your entire search query. I don’t know whether that’s the best solution though.

I would try noindex first and see if that does anything. If it doesn’t, then use the method of blocking your search results in your robots.txt.

Good luck!”

Ask Yoast

In the series Ask Yoast we answer SEO questions from followers. Need some advice about SEO? Let us help you out! Send your question to ask@yoast.com.

Read more: ‘Block your site’s search results pages’ »