When you roll out the red carpet on a new website, you might be expecting visitors to start coming eagerly. In reality, before people start visiting your site, search engines need to find, index and rank it. In this article, I share some tips that could help you get your site indexed faster.
A quick note before we go on. Although in the short term, there are some things you can do to get your site indexed faster, you mustn’t forget the long game. A sustainably high-ranking website depends in large part on creating remarkable content. Why? Because search engines want to find the best answer to the queries their users make. The site with the best content wins the race to the top of the search results.
How do search engines work?
To understand how to get your site indexed, it’s useful to know how search engines work. Search engines generate results in three main steps: crawling, indexing, and ranking.
Crawling is the process of discovery done by crawlers, bots, or spiders. A computer program instructs crawlers on what pages to crawl and what to look for. When crawlers land on a page, they gather information and follow links. Whatever they find, they report back to the search engine servers. Then, the search engine tries to make sense of the page in order to index it. It looks at the content and everything it finds, it puts in a giant database; their ‘index’.
Finally, ranking begins when you search for something online. So, the search engine algorithm looks through the index and filters the pages to find the best ones. We do not know the exact mechanics of the algorithm. Still, we know that search engines are especially enthusiastic about high-quality content and user-friendly, up-to-date pages.
So, how can you speed up this process and help search engines find you quicker? An excellent way to start is submitting an XML sitemap in Google Search Console.
An XML sitemap is a file that contains information about your website. In plain language, it is a list of your most important pages. It is a useful tool that helps Google find and explore your site. Yoast SEO can help you to create a sitemap. All you need to do is enable the XML sitemap option and the sitemap will be automatically generated. It’s quite a time saver!
After you’ve created the sitemap, you need to tell Google about it. Google Search Console is a tool that can help you with that. To add your sitemap to the Console, you need to create an account. Yoast SEO can also help you get your site verified with Google Search Console.
In Google Search Console, you will find the XML sitemap tab. There, you can add the sitemap you created, so that Google will know where to find it. If you update content on your site, your XML sitemap will be updated automatically.
And/or, submit your most important individual pages in Google Search Console
On top of this, you can ask Google to crawl individual pages too. In Google Search Console you’ll find the URL inspection tool, where you can ask Google to crawl or recrawl a URL. There’s a quota, so think about which pages are crucial for your business in terms of ranking and submit those here.
Why do you need an XML sitemap?
We mentioned that crawlers discover pages by following links. When you have a new website, you may face at least two issues. First, there are likely not that many external sites that point to your website. Second, you probably still don’t have a lot of content, so your internal linking and your site structure are not (yet) stellar. Without links, how can the crawlers come to your site?
One solution to getting your site indexed is to create a sitemap right from the start and add it to Google Search Console. However, since you might still not have a lot of content, you should be careful with what you include in it. Although you can create sitemaps for videos, images, categories, and tags, it does not mean that you should necessarily do it. For example, you might have already set up some categories. But, for each category, you have just one post. In that case, creating a sitemap for your categories is not so useful, since the content does not give a lot of information, both to visitors and crawlers.
It is important to note that Google may not crawl and index all the items in your sitemap. Still, we encourage you to create one, as we believe that you will benefit from it.
Getting your site indexed beyond Google
We told you how you could submit a sitemap with Google Search Console. But, it’s not the only search engine out there. So, how can you submit your sitemap to other search engines? It is easy with Yoast SEO. Other search engines also have Webmaster tools, where you can submit a sitemap and follow the performance of your site. Currently, you use Yoast SEO to add your site to:
After you’ve created a sitemap and connected it to search engines with Yoast SEO, can you finally sit back, relax, and watch as visitors pour in? Not really. As we said, you will have to continue making high-quality content. Don’t forget that you can also use social media to your advantage and strategically share your content there. Another important thing is getting links from other, preferably high-ranking websites. That means that you will need to work on your link building. Of course, don’t forget to apply holistic SEO strategies to your website to cover all SEO fronts and ensure high rankings.
It can happen to anyone: You’re working on your site, fiddling on some posts here and there, and hit update when you’re done. After a while, you check back on how a post is doing and, to your dismay, it disappeared completely from the search engines! It turns out you’ve accidentally set a post or page to noindex on your site. Here, we’ll share a – pretty funny – story about how it happened to us, and what to do if you’ve made a similar mistake.
How to unintentionally noindex a post
Let’s start with a short story. We have a post called Noindex a post in WordPress, the easy way on yoast.com. In this post, we – surprise, surprise – explain how to noindex a post with Yoast SEO in WordPress. To show how easy that is, we added some screenshots of setting a post to noindex. A picture says more than a thousand words, right?
Some of the copy and the screenshot in the ‘How to noindex a post’ post
Now, we’ll reveal a little secret. Oftentimes, when we want to illustrate a certain feature of Yoast SEO, we’ll just take a screenshot of that feature in the post we’re working on. So, in this case, we went to the Advanced tab in the Yoast SEO meta box, clicked No in the dropdown, took a screenshot, and added the screenshot to our post. We checked the copy we’ve written, added images, checked the SEO and readability scores and previewed our post. All looked fine, so we hit publish, shared it on social and in our newsletter and went on with other tasks.
Sometime later, we were checking how our content performed on the query [how to noindex a post] in Google. Surprisingly, we didn’t encounter this article, while we were pretty sure we already had a post like this. We started looking for it in our post overview, and there it was! Waiting in vain for visitors to learn more about this handy feature of Yoast SEO.
So, while we were happily typing away, making sure people understand what this feature is about, we forgot one thing… removing the noindex from this post. Therefore, accidentally and ironically, we’ve set our post about setting posts to noindex to… noindex.
How to reverse noindexing a post
In our case, reverting the noindex on that post wasn’t very difficult. The post, although it describes a nifty feature of Yoast SEO, wasn’t crucial for our business. Therefore, we decided to just remove the noindex and republish and share it again. But there’s more you can do; the options to get your article back in the search engines are listed here below. Depending on the severity of the issue you can choose to follow all steps or select some of them.
1. Remove the noindex tag
This is an essential step. You can easily remove the noindex tag by Google and other search engines in the Advanced tab of the Yoast SEO meta box. Just click on Yes here and you’ve removed the noindex tag:
Remove the noindex tag in the Yoast SEO meta box
In the search appearance section, you can set multiple posts or pages on your site to noindex. If you did that by accident or forgot to reverse that after temporarily setting it to noindex, you can set it to index again there too:
Remove the noindex tag on a post type in the Search appearance section of Yoast SEO
If you’ve added a meta robots tag in the code to noindex your post, please remove it from the code. There’s no need to set it to index though since that is the default value when nothing is set.
2. Google Search Console is your friend
If you’ve accidentally noindexed a valuable post or maybe even an important part of your website, there are some things you can do to make Google retrieve your content faster. Google Search Console can help you do this. So if you didn’t sign up for Google Search Console yet, now’s the time to do it. Yoast SEO will help you to verify your site, as you can read in this guide on how to add your site to Google Search Console.
Request for reindexing of a URL
In the URL Inspection Tool of Google Search Console there’s an option to ask Google to crawl or recrawl a URL. This might speed up the process and allows you to follow the progress. There is a quota for submitting individual URLs with this tool. So, if you’ve noindexed (a part of) your site it might be wise to select the posts or pages that are most crucial for your business and request to index those again.
Resubmit your XML sitemap
Another option is to resubmit your XML sitemap in Google Search Console. If you’re using Yoast SEO you don’t have to worry about this though. In that case, when you publish or update content on your site, Yoast SEO automatically pings Google with your sitemap.
Lastly, you can share the reindexed content in your newsletter, on social and other marketing channels. This way, you’ll generate some traffic and exposure, especially if other people start sharing it too. In case of a blog post, you can republish it on your blog. If it considers important pages of your site quickly thinking up a campaign and publishing new blog posts that link to the reindexed content could also help to get the initial traffic and rankings back.
It’s not the end of the world
Finding out you’ve accidentally set a post or even (parts of) your site might give you a big scare. But, fortunately, it’s not the end of the world and there are various things you can do to get it back in the search engines again. Depending on the size of the issue and the frequency your site gets crawled, it will take some time to recover, but eventually, it probably will.
Now, let’s hope I haven’t accidentally set anything to noindex when creating this post…
Google is rapidly expanding the capabilities of Search Console — its must-have tool for site owners/managers. We’ve seen al lot of cool structured data reports appear. In this post, we’re examining an enhancement report dedicated to site speed. It’s important to have a fast site with a good user experience and Google’s new tool helps you monitor it and improve it. Here’s is a quick guide to its capabilities.
What is the Core Web Vitals report in Google Search Console?
The Core Web Vitals report gives you an idea of how fast or slow your pages load over any given time. It gives you insights that were almost impossible to get up until now. Running page speed analysis on your complete site is not something the average user can do. Testing a couple of pages in PageSpeed Insights, fine, but 1,000 pages? The new Core Web Vitals report in Google Search Console gives you an idea of how your site loads. It puts all pages in buckets conveniently labeled poor, in need of improvement and good.
The Core Web Vitals report overview in Search Console (desktop view)
As you know, site speed and user experience have been a hot topic for quite a while. Google even declared page speed a ranking factor as well as new ranking factor called page experience. The search engine is rolling out all sorts of initiatives to help visualize site speed and prioritize improvements, like PageSpeed Insights and Lighthouse. Sometimes, Google also tries some outside the box changes like Chromes “speed badge of shame”. It is one of the indicators in the Chrome browser that helps users understand why a site may be loading slower. In reality, this is more a not so subtle jab at site owners to do something about their slow sites.
Chrome’s slow site badge
This focus on site speed is understandable. Site speed is user experience and users expect fast. But in regards to all those pretty numbers and colors, it’s hard to know what to look for. But as our own SEO expert Jono Alderson loves to say: “Don’t optimize for scores — just make it faster.” Scores say a lot, but all that matters is the perception of speed by users. How quickly can you make your page feel ready?
What does the Core Web Vitals report do?
The Core Web Vitals report looks at the pages on your site, checks their scores in the Web Vitals data and puts these into buckets. There are mobile and desktop specific checks and these might differ. Due to hardware and network differences, it is harder to get a good score on mobile than it is on desktop. You’ll notice, though, that the same URLs are often troublesome both on mobile as well as desktop. They might load slightly faster due to changes in test setting, but they are a point of interest nonetheless.
Two specific reports help you analyze the different sources
While not the end-all tool for measuring site speed, the Core Web Vitals report is a valuable addition. It helps you find problematic URLs which you can check in PageSpeed Insights to get a deeper understanding — plus ways of fixing it. This way, you can keep an eye on all speed-related things, spot trends, make improvements and keep track of the results of those changes.
Where does it get its metrics?
The cool thing about the Core Web Vitals report is that it uses data from the Chrome UX Report. The Chrome UX Report is a public data set of real user experience data collected from millions of opted-in users and websites. This way, Google collects loads of data — like connection type, type of device and much more — from real situations and used to give a better understanding of performance in the real world. Google uses this data in several speed-oriented Google tools, like PageSpeed Insights and Lighthouse.
What should I look for?
When looking at site speed tools it is easy to focus on the wrong stuff. Many tools check site speed in particular circumstances, like a set location at one point in time, accessed from a specific device. There’s not enough context to make a decision based on this data. That’s why our advice in this has always been for you to look at a multitude of site speed tools. Combined these will give you a better handle on the problem.
Google built the Search Console Core Web Vitals report around three metrics: FCP, FID and CLS. These three metrics form the Core Web Vitals. Here’s what these metrics mean:
LCP, or Largest Contentful Paint: This metric tells how long it takes for the largest content element you see in the viewport to load.
FID, or First Input Delay: The FID looks at how long it takes for a browser to respond to an interaction first triggered by the user (clicking a button, for instance).
CLS, or Cumulative Layout Shift: The CLS measures how much layout elements move on screen during loading and how often that happens.
Reading the results
The results lead to an overview of pages that have good or poor scores, or are in need of improvement. The score of a URL is the lowest status assigned to it by a specific device. According to Google, the three metrics work together to come to a conclusion about the loading of the URL:
A URL on mobile with Poor FID but Needs improvement LCP is labeled Poor on mobile.
The URLs on mobile with Needs improvement LCP but Good FID are labeled Needs improvement on mobile.
A URL on mobile with Good FID and CLS but no LCP data is considered Good on mobile.
A URL with Good FID, LCP, and CLS on mobile and Needs improvement FID, LCP, and CLS on desktop is Good on mobile and Needs improvement on desktop.
These insights give you a good idea of how your pages are performing. As said before, you probably need to run a couple of more tests to get the full picture.
Further analysis on a per-URL basis in PageSpeed Insights
URL grouping
Instead of showing a gazillion URLs and the corresponding results, Google uses aggregate scores and URL groups to make the results slightly less intimidating. For any issue, you’ll see a number of URLs getting the same score or issue. So it might be that from a specific URL, 70 other URLs suffer from the same performance issues. That makes it easier to uncover issues on a grander scale because all these pages probably have the same problems. Of course, you can do a deep-dive and check individual pages by clicking on the URL list and picking a URL to analyze using PageSpeed Insights.
Grouping URLs with similar perfomance issues makes the report easier to digest
Aggregate scores
The same goes for scoring. Grouping makes it easier to digest the results. The Core Web Vitals report in Search Console focuses mainly on FCP and FID, as mentioned above. It’s a good idea to keep an eye on PageSpeed Insights as well, as this has a multitude of other metrics, graphics of the loading process and suggestions to improve the results.
In the Core Web Vitals report, Google calculates the FCP and FID from all the visits to those particular pages.
Aggregate FCP: The aggregate first content paint is the time it takes for 75% of the visits to a URL in the report to reach FCP.
Aggregate FID: The aggregate first interactive delay is the time it takes for 95% of the visits to that URL to respond to interactions on that page.
CLS agg.: The aggregate cumulative layout shift shown in the report is the lowest common CLS for 75% of visits to a URL in the group.
The calculation of these scores continues to fluctuate due to outside influences. That’s why you might see the trend line go up and down.
Aggregate CLS is the lowest common CLS for 75% of the visits to a URL in that group
Fixing issues and validating fixes
The Core Web Vitals report allows you to monitor your site for speed-related issues. It helps you find problems and prioritize their resolution. Once you or your developer have run through all the suggestions and improvements you can validate the fix. Google will then monitor the pages for 28 days to see if the issue is fixed for these URLs.
Site speed resources
This post is not about telling you how to fix your site speed issues, but rather guiding you through the new Core Web Vitals report that might give you the insights you need. To get practical, you can start here:
Google’s recent run of enhancement reports in Search Console gives you lots of insights into how your site is performing in search. Sometimes, though, it gives you stuff to think about, like errors or improvements to make. For instance, if you run an online store, you’re bound to have come across this structured data error: “Either ‘offers’, ‘review’ or ‘aggregateRating’ should be specified.” There’s a very easy solution for this if you run WooCommerce and Yoast SEO: our WooCommerce SEO add-on.
The “Either ‘offers’, ‘review’ or ‘aggregateRating’ should be specified” error in Google Search Console
The “Either ‘offers’, ‘review’ or ‘aggregateRating’ should be specified” happens for a lot of online stores. It means that Google misses several properties in your product schema implementation. By not offering these, your product listings will not reach their full potential in search. This way, Google has a hard time tying all the product-specific properties together to paint a full picture of your product. In some cases, though, they manage, but why let them figure it out? Fixing this becomes imperative if you want a better chance of standing out.
Who doesn’t want a product listing like the one pictured below?
Valid product schema might lead to eye-popping rich results like this one from Reverb
Oftentimes, however, invalid or incomplete structured data might cripple your perfomance in search. Errors are all too common, like the one in the screenshot from Search Console below.
The “Either ‘offers’, ‘review’ or ‘aggregateRating’ should be specified” error is very common
Help is at hand: Yoast SEO & WooCommerce SEO
WooCommerce is huge in the WordPress world. According to W3Techs, 15% of all WordPress sites run an online shop on the WooCommerce platform. That’s amazing. We have a plugin that helps customers improve their online store: WooCommerce SEO. This addon ties neatly into Yoast SEO, including the big schema graph we build for every site. It also greatly improves the product schema output by WooCommerce.
If your site runs on WooCommerce and Yoast SEO you need WooCommerce SEO. Besides all the cool behind-the-scenes improvements, it fixes that dreaded “Either ‘offers’, ‘review’ or ‘aggregateRating’ should be specified” error for you: automatically! It gives Google everything it needs to figure out your products are products and thus increases your chances of getting those important rich results.
Why you should fix this error
Google is increasingly betting on schema structured data to help understand the world. If your site offers search engines enough context about what’s on it, the rewards could be great: rich results. And for some types, visibility on other devices like smart speakers or visual assistants.
Getting your product schema right, means you can get these types of results. The one earlier in this article is from Reverb and shows a nicely formed product rich results, with breadcrumbs, product information, ratings and reviews, pricing details and an in-stock message. This is all powered by product schema.
Reporting on the performance of products
To help you track how your products are doing, Google recently added a Product enhancement report to Search Console. This report lets you know if your products are correctly structured and, therefore, eligible for rich results. This week, Google also announced that it will allow you to see the performance of your product in the search results. You can now find a new Product line in the Search Appearance section of the Search Performance section.
Search Console now has a product results view in Search Appearance
This report shows exactly how well your products are doing: how many impressions did they have and how many clicks? This is invaluable data to improve your product listings.
Fix the error and check your listings
Seeing the product schema error in Search Console? Using Yoast SEO and WooCommerce? Well, you’re in luck. The WooCommerce SEO add-on is the glue that ties the product schema structured data between those two platforms together. It fixes that dreaded error and gives you a better chance at getting your products noticed in Google!
Yesterday, Google announced a new addition to the rapidly expanding lineup of enhancement reports in Search Console. This time, it’s all about video! You can now not only see how your video performs in search, but also if your videos have valid structured data so they are eligible for rich results. Need help getting your videos noticed in Google? We’ve got just the thing!
What’s this Video enhancement report?
Video is incredibly important. Humans are producing an huge amount of video and surfacing the right one at the right moment is a real struggle for a search engine. Google now gives you two new tools to help you improve the performance of your videos in search. The first one is all about helping search engines figure out if there is a video on a site. The second one is about visualizing your video’s performance in search.
By adding structured data to your page, you can identify the most important elements plus its characteristics. By doing this, you make it crystal clear to search engines what is what. As a result, you might be awarded a rich result if the search engine deems your content the best fit for the query. For videos, this might include information on a video’s duration, upload date, and other metadata, as well as previews.
Google is pushing structured data very hard and this latest report shows how important video has become for them and for you. You can now see how your video performs and if there are any technical errors or warnings that keep it from performing better. From here, it’s an easy fix.
But how can I get this for my videos?
That’s easy: the Video SEO for WordPress add-on for Yoast SEO! As you might know, we launched a massive and innovative Schema structured data implementation in Yoast SEO 11.0. This was the first implementation of its kind — the first to completely tie the content of your site together in code so search engines are eager to gobble it up. Not only that, by connecting everything a search engine now knows what goes where and how it all fits into the bigger picture. This is all available for free in Yoast SEO.
Our add-ons extend this structured data implementation. The Video SEO plugin, for instance, adds specific structured data for videos. And the awesome thing is: you don’t have to do anything to get this on your site. It’s all automatic and it all ties neatly into the rest of the structured data we generate for your site. Just install Video SEO.
Here’s how the new report picks up our structured data, without us having to do any additional work. You can even see the source code by clicking on an example URL.
Click on a post to see the structured data highlighted
So, if you need search engines to understand your videos — who doesn’t? — you need Video SEO for WordPress!
What else can I do in the Google Search Console Video enhancement report?
The Video enhancement report not only shows you if Google has trouble understanding your video content, but also how your videos perform in search. In the Performance tab, you’ll find a new Search Appearance type for video. This overview gives you an idea of how many times your videos appear in search and many people click on it. Of course, you can filter it to your hearts content.
See how your video performs in the search results
Help search engines understand your videos
The new Video reports in Google Search Console go a long way in helping Google identify videos via valid structured data. Not only that, it gives you the insights you need to help identify issues and improve performance. All in all, this is a great addition to the treasure trove that is Google Search Console!
I often get a request from our Blog team about one of their pages. Sometimes they want to know if the page has gotten more pageviews or they notice something weird and they want me to find out what’s going on. And this time they wanted more insight in the performance of one particular page. I want to share with you how I deal with this request.
So the other day I got a request to check the performance of our Blog homepage. We want to optimize that page so it fits better with the need of our audience. If you have an idea about that, you can leave your feedback in the comments section below this post.
Page level analysis
The first thing I had a look at is how ‘popular’ the page is and if it’s worth the effort to spend time and resources on this page. I went to the All pages report in Google Analytics, which you can find under Behavior –> Site content and did a cmd+F or ctrl+F search for the page https://yoast.com/seo-blog/. If you can’t find the page you’re looking for; it’s because it probably isn’t within the first ten results. So you need to expand your table first; you can do this on the right bottom of the table.
I notice it’s at position 32 if you look at the number of pageviews, which is a reasonably high position and thus worthy of further investigation. I also notice that in 6,327 of the cases it’s an entrance page, this means it’s in 35,3% of the cases the first page of the session (6,327 / 17,931 * 100).
You can also search for a specific post in the search bar to see just the metrics of that page but sometimes other pages show up as well.
If the page is in blue letters, it means you can specify even further when clicking on it because it’s a link. Or you can use regex; regex stands for “regular expression”. Lunametrics made this fun regex guide that shows you how you can use regex in Google Analytics. It may sound a bit scary, but if you know the basics, it’s quite doable and will make your Google Analytics life a whole lot easier. Here’s how a regex would look if I just wanted the SEO blog homepage:
^yoast\.com\/seo-blog\/$
But in this case, you can just as well click on the page to see just the metrics of that page. Try to understand what the metrics are saying and how it compares to the site’s average. In this case, for a page that’s built to guide people to blog posts of their interests, a bounce rate of 50.48% is fairly high. That means that in half of the cases, people didn’t do anything on that page! That’s not what the page is designed for.
I was also curious to see if this page gets a lot of mobile traffic, so I added a secondary dimension with the Device category. I then checked what the metrics told me.
About 10% of the page views come from a mobile device. You can see it has a higher bounce rate so checking the mobile experience is a good idea.
Trend analysis
And, I was curious to see how the page developed over time, so I added a wider timeframe to check if I saw something unusual. You can adjust the graph you’re seeing. Perhaps you’re interested if Bounce rate declined or not. You can select this metric and you’ll see the trend of the bounce rate of that page.
Session level analysis
I then looked at this page from a page level. But, I had more questions about this page. If people are entering our site through this page, where are they coming from? So, I had a look from a session level perspective. I went to landing pages and did the same search as in the All pages report.
It’s at position 65 and obviously has 6,327 sessions since we saw that in the All pages report at entrances. I once again looked at the metrics and tried to understand what they’re telling me. The number of pages per session, the bounce rate and the number of ‘new’ users. And I had a look at conversions.
I then dove in further, clicked on the page and added a secondary dimension: medium, so I could quickly see where traffic is coming from. I noticed that we have a lot of traffic that we don’t know the source of. So that’s something to explore further. In second position comes our plugin and third is organic search traffic. Which is interesting to see because I’m curious with what keywords people end up on that page and if we rank properly on that keyword or keyphrase. With that information, we can improve the SEO of that page even further.
Again, I had a look at bounce rates, pages per session, number of new users and possible conversions. Thinking about if the page is doing what it’s supposed to do.
Google Search Console analysis
I needed to go to Google Search Console to find information about the keywords or keyphrases the blog homepage ranks for. You go to your Google Search Console account and click on Search results. Then you set a filter that exactly matches the URL of your page, in our case: https://yoast.com/seo-blog/.
You now see the queries and position of that page. Take a look at the metrics and try to understand what’s going on. It’s especially interesting if you have a lot of impressions but a low clickthrough rate (CTR).
Conclusion
What can we learn from this analysis? For one is that it’s worth the while to put some time and effort into this page. I learned that we can optimize the SEO of that page even further and that we can put some more effort into ranking for the keyword SEO blog.
I also noticed that it’s quite a popular page, but the bounce rate is too high for my taste. Especially when the goal of the page is to guide people to a blog post of their interest. So, there needs to be interaction with this page. We need to find out what people expect to find on this page. So, therefore, extra information is needed. That’s why we added a simple poll on this page, using Hotjar. We also created a heatmap with this tool to get a better understanding of how people behave on the page.
Combining data gives you a far more holistic view and will make sure you can draw more reliable conclusions. Data we can use to optimize the blog homepage even further. The perfect dataset doesn’t exist but we can try to get as far near perfection as possible.
If you’ve noticed that your Yoast SEO Search Console report looks surprisingly empty, it’s because Google have shut down the system which provides data about their crawl errors.
There’s no need to worry, though – we knew this was coming, and we have plans for the future. Your Search Console report won’t show any information for now, but nothing on your site will break, and you don’t need to take any special action.
The good news is that we’re already building our own, brand new systems for error reporting and management. We think it’ll be even better and more useful than the old Google Search Console integration.
What happened?
If you actively manage your website’s SEO, then you’re probably familiar with Google Search Console. It’s a great way to discover and manage any errors which Google may have encountered when crawling and reading your site.
A snapshot from yoast.com’s own Google Search Console account. We should fix those errors!
Until recently, Google provided us with a way to extract information from their systems. That meant that we could show you Google’s crawl errors and issues right inside your site’s admin area, via the Yoast SEO plugin.
Help! What is an API?
An API is a system which allows software to talk other software. The Google Search Console API allows websites to connect to Google’s systems, and to request information about your website. It used to provide information about crawl errors, but no longer does so.
Having those issues listed inside your side admin area made it easy for you to fix those issues ‘on the fly’. You could set up redirects, tweak pages, and mark individual issues as resolved – without ever leaving your website.
Unfortunately, Google have removed the API which powered this system.
That means it’s no longer possible for us – or anyone else – to get information about your crawl errors. The only way to get this information (at the moment) is for you to manually visit your Google Search Console account.
What does this mean for me?
Nothing will break on your website, and you don’t need to take any action. You’ll just find that if you visit your Search Console admin page in your WordPress website, you won’t see any errors. You’ll see something like this empty table:
You won’t find any errors in your Search Console report, as it’s no longer possible to retrieve them from Google.
That empty table doesn’t necessarily mean that you don’t have any errors – it just means that we can’t fetch or show them to you.
We know that many of our premium users rely on our Search Console integration to create redirects for broken pages and URLs.
If you’ve set up redirects through the Search Console section in the past, don’t worry – nothing’s been lost, and your redirects still work.
In the future, you’ll just need to use the ‘Redirects’ section to create them, instead.
You can create redirects for broken pages or URLs in Yoast SEO premium, via the Redirects system
We knew this was coming, but we expected a replacement system
We’ve known that Google planned to remove support for error reporting from their Search Console API for a while. It provided data from the ‘old’ Search Console system, which has been gradually replaced or removed as they moved people to the ‘new’ Search Console system. As such, we knew it had a short lifespan.
We’d hoped and assumed that they’d replace the API, in the same way that they’d moved or replaced other functionality from the old system.
What we didn’t expect was for them to simply ‘turn it off’, without providing an alternative or updated solution. We’re a little disappointed about how this has been handled by Google, but, we have some options.
What happens next?
For now, you don’t need to do anything.
We’re chasing Google for updates, and once we hear more from them about what they have planned – if anything – we’ll update our information to reflect that.
Their documentation and communications have hinted at planned future capabilities and support, but there’s nothing out there yet. So we’re waiting, watching, and asking questions about what’s in the pipeline (we’ve particularly high-hopes for Google’s SiteKit WordPress plugin, which is currently in beta).
In the meantime, we’re considering alternative options, processes, and ways in which we can replace or supplement this data. We already have some exciting plans:
In the short-term: We’ll be building an import tool, which lets you upload error reports which you’ve manually downloaded from Google Search Console. We’ll also support uploads from some other tools, which we’re excited about. We’ll share more news that soon.
In the mid-term: We’ll be looking to partner and integrate with a wide variety of your favourite SEO tools and platforms, so that we can import their data about your website’s errors and crawl issues. That’ll give us much richer, more diverse, and more interesting data than ever.
We’ll keep you updated with our progress, and we’ll let you know if we hear back from Google about their plans.
As we announced at YoastCon, we’re working together with Bing and Google to allow live indexing for everyone who uses Yoast SEO. In an update currently planned for the end of March, we’ll allow users to connect their sites to MyYoast, our customer portal. After that we’ll roll out live indexing, which means every time you publish, update, or delete a post, that will be reflected almost instantly into Bing and Google’s indices.
How does this work?
When you connect your site to MyYoast, you’ll authorize it to send messages back and forth. One of the types of messages that will be sent will be when you publish, update or delete anything on your site. We’ll pass that notification on to Bing, and Bing will check the changes within approximately one minute of you hitting the button.
You will not need to connect your site to Bing Webmaster Tools to do this. It’s all handled automatically. For Google the details on this are not entirely determined yet, but a Google Search Console account will probably be required.
We do, of course, recommend making use of both Bing Webmaster Tools and Google Search Console as they can give good insight into why your site is or is not performing well in the search engines.
If you’re reading this on Thursday (Feb 7, 2019), you might still catch one of Bing’s principal engineers, Arnold Overwijk, discuss this live at YoastCon through our live stream on Friday.
Google Search Console (or ‘GSC’ for short) lets webmasters monitor and manage their websites through an official portal, and is crammed full with useful statistics. Having access to tools and data provided directly by the search engines can make optimizing your website much easier!
It’s a communication channel
Search Console accounts are the main, and official way in which Google communicates with individual site owners. By having a registered account, Google can send webmasters information about site issues, errors, or even penalties. It also provides some limited tools to allow you to contact them about site issues and feature requests.
It’s a control center
If you’re actively optimizing your website, you’ll understand that SEO is never ‘finished’. You need to be continually improving your content, refining your site settings, and minimizing your errors.
Search Console provides tools which help with this day-to-day management. It lets you do things like submit and monitor your XML sitemaps, ask Google to (re)evaluate your errors, or see how Google sees particular pages and URLs on your site.
It’s a performance dashboard
Your GSC account is full of useful information about how your website is shown and performing in search results. From mobile usability reports to visibility and clickthrough tracking, and much more.
If you’re serious about managing and optimizing your website, your GSC account is your nerve center for understanding when, where and how your site is appearing in Google.
It’s a data source
Most of the data in Google Search Console can be extracted and integrated into other systems, like Google Analytics, and Yoast SEO!
That means that, if you’re running a Yoast SEO plugin, you can integrate some of your GSC data directly into your website. This can make it much easier to manage your errors, analysis, and redirects!
Check out our great guide on how to get that hooked up, and how to take advantage of the integration.
Ready to get started?
Anybody who runs or manages a website should be able to access a Google Search Console account, for free.
There are a few different ways to create and authorize your account, but the easiest is to integrate through Yoast SEO – just follow this quick guide to get things running!
Do you have your own website or maintain the website of the company you work for? Of course, to do this right, you need to keep a keen eye on the performance of your website. Google offers several tools to collect and analyze data of your website. You probably have heard of Google Analytics and Google Search Console before. These tools are free to use for everyone maintaining a website and can give you very valuable insights about your website.
Why everyone with a website should use Google Search Console
Google Search Console has been created to easily track the performance of your website. You can get valuable insights out of your Google Search Console account which means that you can see what part of your website needs work. This can be a technical part of your website, such as an increasing number of crawl errors that need to be fixed. This can also be giving a specific keyword more attention because the rankings or impressions are decreasing.
Besides seeing this kind of data, you’ll get mail notifications when new errors are noticed by Google Search Console. Because of these notifications, you’re quickly aware of issues you need to fix.
Setting up an account
To start using Google Search Console, you’ll need to create an account. Within the new Google Search Console, you can click on ‘add a new property’ in the top bar:
Add a new property — or a site — to get started
Clicking on the ‘Add a property’ button, you can insert the website you want to add. If you choose the new Domain option, you only need to add the domain name — so without www or subdomains. This option tracks everything connected to that domain. With the ‘old’ URL prefix option you have to add the right URL, so with ‘https’ if you have an https website and with or without ‘www’. For collecting the right data, it’s important to add the right version:
Choose domain if your want to track all your URLs or URL prefix if you want to track specific URLs
When you’ve added a website, you need to verify that you’re the owner. There are several options to verify your ownership. The Domain option only works with DNS verification, while the URL prefix supports different methods. You can find out more about the differences in Google’s documentation: adding a new property and verifying your site ownership.
For WordPress users who use Yoast SEO, get the verification code via the ‘HTML tag’ method:
You can easily copy this code and paste it into the ‘Webmaster tools’ tab within the Yoast SEO plugin:
After saving this, you can return to Google Search Console and click on the ‘Verify’ button to confirm. If everything is ok, you’ll get a success message and GSC will start collecting data for your website.
Features in Google Search Console
Now you’ve set up your account what would be the next step? Well, it’s time to look at some of your data! We’ll explore some of the reports and information available in the rest of this article.
Performance tab
Within the Performance tab, you can see what pages and what keywords your website ranks for in Google. In the old version of GSC you could see the data of a maximum of the last 90 days but in the current version, it’s possible to see the data up to 16 months. Keep in mind that the data is available from the moment you set up your account.
If you check the performance tab regularly, you can quickly see what keywords or what pages need some more attention and optimization. So where to begin? Within the performance tab, you see a list of ‘queries’, ‘pages’, ‘countries’ or ‘devices’. With ‘search appearance’ you can check how your rich results are doing in search. Each of those sections can be sorted by the number of ‘clicks’, ‘impressions’, ‘average CTR’ or ‘average position’. We’ll explain each of them below:
The Perfomance overview harbours a ton of information
1. Clicks
The amount of clicks tells you how often people clicked on your website in the search results of Google. This number can tell something about the performance of your page titles and meta descriptions: if just a few people click on your result, your result might not stand out in the search results. It could be helpful to check what other results are displayed around you to see what can be optimized for your snippet.
The position of the search result also has an impact on the number of clicks of course. If your page is in the top 3 of Google’s first result page it will automatically get more clicks than a page that ranks on the second page of the search results.
2. Impressions
The impressions tell you how often your website in general or how often a specific page is shown in the search results. For example, in the GSC account of our own website, Yoast SEO is one of the keywords our website ranks for. The number of impressions shown after this keyword shows how often our website is shown for that keyword in the search results of Google. You don’t know yet what page ranks for that keyword.
To see what pages might rank for the specific keyword, you can click on the line of the keyword. Doing this for the keyword [Yoast SEO], the keyword is added as a filter:
You can query the data in many ways
After that, you could navigate to the ‘Pages’ tab to see what pages exactly rank for this keyword. Are those pages the ones you’d want to rank for that keyword? If not, you might need to optimize the page you’d like to rank. Think of writing better content containing the keyword on that page, adding internal links from relevant pages or posts to the page, making the page load faster, etc.
3. Average CTR
The CTR – Click-through rate – tells you what percentage of the people that have seen your website in the search results also clicked through to your website. You probably understand that higher rankings mostly also lead to higher click-through rates.
However, there are also things you can do yourself to increase the CTR. For example, you could rewrite your meta description and page title to make it more appealing. When the title and description of your site stands out from the other results, more people will probably click on your result and your CTR will increase. Keep in mind that this will not have a big impact if you’re not ranking on the first page yet. You might need to try other things first to improve your ranking.
4. Average position
The last one in this list is the ‘Average position’. This tells you what the average ranking of a specific keyword or page was in the time period you’ve selected. Of course, this position isn’t always reliable since more and more people seem to get different search results. Google seems to understand better and better which results fit best for which visitor. However, this indicator still gives you an idea if the clicks, impressions and the average CTR are explainable.
Index coverage
A more technical but very valuable tab within Google Search Console is the ‘Index coverage’ tab. This section shows how many pages are in the index of Google since the last update, how many pages aren’t and what errors and warnings caused difficulties for Google indexing your pages properly.
You can see the trends in errors as well
We recommend checking this tab regularly to see what errors and warnings appear on your website. However, you also get notifications when Google has found new errors. When you get such a notification you can check the error in more detail here.
You may find that errors are caused when, e.g., a redirect doesn’t seem to work correctly, or Google is finding broken code or error pages in your theme.
Clicking on the link, you can analyze the error more in depth to see what specific URLs are affected. When you’ve fixed the error you can mark it as fixed to make sure Google will test the URL again:
Fixed the specific error? Validate it so Google can check if it’s gone for real
There are a few things you should always look for when checking out your coverage reports:
If you’re writing new content, your indexed pages should be a steadily increasing number. This tells you two things: Google can index your site and you keep your site ‘alive’ by adding content.
Watch out for sudden drops! This might mean that Google is having trouble accessing (all of) your website. Something may be blocking Google; whether it’s robots.txt changes or a server that’s down: you need to look into it!
We recommend that you monitor these types of situations closely and resolve errors quickly, as too many errors could send a signal of low quality (bad maintenance) to Google.
URL Inspection
The URL Inspection tool helps you analyze specific URLs. You retrieve the page from Google’s index and compare it with the page as it lives now on your site to see if there are differences. On this page, you can also find more technical info, like when and how Google crawled it and how it looked when it was crawled. Sometimes, you’ll also notice a number of errors. This might be in regards to Google not being able to crawl your page properly. It also gives information about the structured data found on this URL.
The URL Inspection tool gives invaluable insights into every URL on your site
Enhancement tabs
Below the ‘Index coverage,’ you can find the Enhancement tab. Here, you’ll find everything you need to improve how your site performs. It has insights in site speed, mobile usability, AMP usage, and structured data enhancements that might lead to rich results in the SERPs.
Speed
The new speed report is still in its experimental stages, but already an invaluable addition. This report gives a good idea of how fast your site loads on mobile and desktop. In addition, it also shows which pages have issues that keep them from loading quickly. The data is based on the Chrome UX report, so real data of real users. Site speed is a difficult topic containing many moving part, so it’s good to learn how you should think about site speed. You can find the answer here: how to check site speed.
Find out which pages load slowly
AMP
One of the tabs is for all things ‘AMP’. AMP stands for Accelerated Mobile Pages: lightning fast mobile pages. If you’ve set up AMP for your website you can check for errors in Google Search Console. Within this section you can see the valid AMP pages, the valid ones with warnings and errors:
Below the chart, the issues are listed. If you click on one of the issues, you can see the affected URLs. Just as in the index section of GSC you can validate your fix if you’ve fixed an issue.
Rich results enhancement tab
If you have structured data on your site, provided by Yoast SEO for instance, it’s a good idea to check out the Enhancements reports in Search Console. The Enhancements tab is the place where all the insights and improvements that could lead to rich results are collected. There’s an ever-expanding list of support rich results. As of writing, that list contains:
breadcrumbs
events
faqs
how-tos
jobs
logos
products
reviews
sitelinks searchboxes
videos
All these tabs show how many valid enhancements you have, or how many have errors or warnings. You get details about the kind of errors and warnings and on which URLs these are found. There’s also a trend line that shows if the number of issues is increasing or decreasing. And that’s just the start of it.
Here’s an example of a how-to enhancement. You can overlay Impressions to get more context for the stats
The Enhancements reports help you find and fix issues that hinder your performance in search. By checking the issues, reading the support documentation and validating fixes, you can increase your chance of getting rich results in search. We have a more expansive guide on the structured data Enhancement reports in Google Search Console.
Sitemaps
An XML sitemap is like a roadmap to all important pages and posts on your website. We think every website would benefit from having one. Is our Yoast SEO plugin running on your website? Then you automatically have an XML sitemap. If not, we recommend creating one to make sure Google can find your most important pages and posts easily.
Within the XML sitemap tab of Google Search Console you can tell Google where your XML sitemap is located on your site:
We recommend everyone entering the URL of their XML sitemap into GSC to make Google find it easily. In addition to that, you can quickly see if your sitemap gives errors or if some pages aren’t indexed, for instance. Checking this regularly, you’re sure Google can find and read your XML sitemap correctly.
We recommend regularly checking the XML sitemap section in our plugin to manage which post types or taxonomies you’re including in your sitemaps!
Links
Within the links to your site section, you can see how many links from other sites are pointing to your website. Besides, you can see what websites link, how many links those websites contain to your site and lastly, what anchor texts are used most linking to your website. This can be valuable information because links still are very important for SEO.
Find out which pages receive lots of links
Within the internal links section, you can check what pages of your website are most linked from other spots on your site. This list can be valuable to analyze regularly because you want your most important pages and posts to get most internal links. Doing this, you make sure Google understands as well what your cornerstones are.
You can even see how many links individual pages get
Mobile usability
The mobile usability tab within this section shows you usability issues with your mobile website or with specific mobile pages. Since mobile traffic is rising all over the world, we recommend checking this regularly. If your mobile site isn’t user-friendly, lots of visitors will leave it quickly.
Handy tips to improve your mobile pages
Manual Actions
The manual actions tab is the one you don’t want to see anything in. If your site is penalized by Google, you’ll get more information in here. If your site is affected by a manual action, you’ll also get messaged via email.
There are a number of scenarios which can lead to these kinds of penalties, including:
You have unnatural/bought links Make sure from and to your site are valuable, not just for SEO. Preferably your links come from and link to related content that is valuable for your readers.
Your site has been hacked A message stating your site’s probably hacked by a third party. Google might label your site as compromised or lower your rankings.
You’re hiding something from Google If you’re ‘cloaking’ (that is, intentionally showing different to content than to users, for the purposes of decieving either of them), or using ‘sneaky’ redirects (e.g., hiding affiliate URLs), then you’re violating of Google’s Webmaster Guidelines.
Plain Spam Automatically generated content, scraped content and aggressive cloaking could cause Google to blacklist your site.
Spammy structured markup If you use rich snippets for too many irrelevant elements on a page, or mark up content that is hidden to the visitor, that might be considered spammy. Mark up what’s necessary, and not everything is necessary.
Security issues
Last but not least: within the security issues tab you’ll get a notification when your website seems to have a security issue.
Search Console: an incredibly helpful tool
Reading this post should give you a good idea of what Search Console is capable of, so I’d like to ask you this: Do you already use Google Search Console for your website? If not, we definitely recommend creating an account so you can start collecting data about your website. Do you think something is missing? Feel free to leave a comment!