Candour

Blog

Episode 5: Google de-indexing, Discover in Search Console and Gallery Ads

Mark Williams-Cook
Mark Williams-Cook , SEO Director
15.04.2019 • 16 min read • Podcast

Play this episode:

Or get it on:

What's in this episode?

Mark Williams-Cook will be discussing:

Google de-indexing and other bugs A dive into the massive bug that caused millions of pages to be dropped from Google's index and what you can do about it, as well as a look at Google's rich data and cache bugs that are also causing headaches for site owners.

Google Search Console updates New reports in the GSC interface, including the Google Discover report and the updating of performance reports to include only canonical URL data.

Gallery Ads A look at Google's new beta 'Gallery Ad' format that is shaking up paid search results.

Podcast transcription:

MC: Welcome to Episode 5 of the Search with Candour podcast recorded on Friday the 12th of April 2019. My name is Mark Williams Cook and I'm going to take around 15 minutes to discuss some of this week's ‘Search’ news that will hopefully make your lives a little bit easier.

Google’s been having several issues in the last week or so, the most notable being the de-indexing bug which I hope you've heard of - it hit millions of sites completely dropping pages from the index. This wasn't their only problem though, they actually had a couple of bugs that have cropped up and we'll talk about them and what you can do to stay on top. There's been a couple of notable updates to Google search console as well, they've added a couple of new reports and sunsetted some old features so we're going to talk about that as well as ‘Google Discover’ which hopefully you've heard of. If not it's something I'll run through briefly with you as well. Lastly there is a new Google Ads format that's been spotted in the wild in beta, I think it's really interesting so we'll have a quick look at that as well!

So Google's de-indexing bug. I was actually thinking about talking about this last Friday on the last episode but the problem seemed to be resolved quite quickly and I don't think I appreciated the scale (anyone really appreciated the scale) at which this bug affected people. So if you haven't heard of it, on Friday the 5th of April - the first time I became aware this was by a chap named Kevin Richard who has the Twitter handle @512banque and he said:

So I retweeted this to see if anyone else had been affected and I've got a few replies. Two days later on the 7th of April the Google SearchLiaison Twitter account that's run by Danny Sullivan gave us some official confirmation. He said:

so this was one of the reasons on Saturday why I decided not to go back and add to our podcast on Friday because it seemed that Google had it in hand and it was quickly resolved.

The following day on the 8th of April this SearchLiaison account said:

so then the following day which was the 9th of April, the SearchLiaison account said:

so this is now three days on. So that was all fine and at this point Google was kind of telling people ‘don't worry, the issue is essentially going to fix itself as pages get recrawled’, and then on the 11th of April we had the update saying:

so the 11th of April being yesterday which is then a week pretty much after this first was reported. They considered it fixed on the 9th, and then two days later they've had to say ‘actually no, we've now fixed it’. So on the 11th of April the Google SearchLiaison on Twitter account said “The indexing issue has now been fully resolved. We apologize for the inconvenience. We appreciate your patience as we restored normal operation.” So the size of this issue I think was quite interesting, I got the sense it was quite big mainly because so many of the SEO’s I follow on Twitter were saying that they had sites that had been affected.

I did a strawpoll in lieu of any bigger announcement or official statement from Google saying how many sites were affected, just to get an idea of how many SEO’s had had sites that they look after affected. That's obviously not representative of individual sites on the web as most SEO’s will manage a portfolio of properties, but from the strawpoll I did - 32% (so that's a third of people that responded) said they had had at least one property affected. So I knew it was it was actually quite a significant issue.

Moz interestingly did some analysis on this to try and estimate the size of the impact; they looked at their ranking data for specific sets of what they knew were stable ranking URLs that had had the same positions consistently for over a month. They broke up the data in the test quite nicely over a distribution of different sites as well as analysing the big three: Wikipedia, Amazon, and Facebook. The headline is the estimate they came up with was that around 4% of the index was dropped which is a staggering amount of pages.

As I said earlier Google's advice was quite casual, I feel their response to this which was ‘we've got it in hand, and it will self-resolve.’ Mark Roth who is an SEO specialist, some people know him for starting a utilitarian men's fashion brand single-handedly encapsulated how a lot of SEO’s felt when he replied saying:

“I think it was pretty poor advice from Google to essentially say do nothing, had to fix a couple of sites myself this morning and know a few others who are having to do the same. If you care about traffic in sales or leads - definitely fix it. Go to GA identify effective pages and prioritize”

I also saw in relation to this, several people respond saying that they had critical pages drop out of the index. One particular case I saw was a site that was ranking very well for some high-volume Easter terms and obviously approaching Easter this is the one time of year that they need to be present and those pages had completely dropped out of the index so they're missing out on tens of thousands of visitors until they got this resolved.

So what can you do if you need to get your URLs indexed and what's the workaround, what is the quick fix well it's quite interesting because we used to have search operators like the ‘info’ where you could type info: and check a single URL if it was indexed but that of course is now dead - those search queries no longer work within Google. You've got this site: query which I've seen quite a few people talk about using. If you do use the site: query and then your domain, this doesn't actually reflect accurately what is in the index so at best that's used to do kind of a rough count of your index coverage if you don't have Google Search Console data, but it's not particularly good advice.

Google's official advice was ‘use the URL inspect tool within Google Search Console’ which a lot of people have been doing but this certainly doesn't scale if you have got a really big site. So if you have got a larger site, I would suggest that you start by comparing date ranges in Google Analytics for organic traffic to see if you've had a big drop-off to specific URLs, and this will at least give you your priority list to start working from. From there you can make sure that your dominant pages and sitemaps had been resubmitted to try and get everything crawled again as quickly as possible.

I did actually find in my research a script that was written by a chap named Steve Walker who is the SEO director at Journey Further, and this script will automate the process of using the URL inspect tool in Google Search Console for you. It is limited to 200 URLs per day, but this looks really helpful because the only other methods you've got to do is either you can write script yourself or go in and mash the button manually yourself and copy and paste all these URLs in. So I've put a link to that script in the show notes at search.withcandour.co.uk. I think that's gonna be really helpful if you've got a larger site and you need to get through a whole bunch of URLs that maybe have been de-indexed.

That wasn't all for Google's issues though in the last week or so, I caught some other people asking about missing metadata issues or missing schema issues. I caught a tweet saying:

and Danny Sullivan who also runs the Google SearchLiaison Twitter account responded to this directly from his personal account saying:

so it's obvious that there's more issues. Google at the moment is saying that they’re unrelated, but this isn't all either! I caught another bug on Twitter that Google's been experiencing and I've confirmed myself and other people have confirmed as well, which is that Google apparently - it looks like they stopped caching pages after the 31st of March 2019. All new domains have the ‘cache error’ when you try and view the cache. This was brought up again on SEO news sites and Google's response this time through John Miller was saying on Twitter:

“The dates on the cache page doesn't really mean much.” He's essentially said to ignore it, much like the advice we've had on the other Google bugs saying it's not indicative of crawl rates. Doesn't really mean whether a page is quality or not, the date is usually just indicative of the last crawl but clearly there is a bug. Now the dates being wrong in my opinion is one thing, the cache coming back with an error is possibly another but it's somewhat frustrating again that essentially webmasters, SEO’s by Google are just told ‘not to worry’ which is obviously going to be the response we're going to get pretty much no matter what happens. They're never going to say to us ‘you should be worried here's what you should be doing’, so I would guess that this is slightly more of a karma face than maybe what's going on behind the scenes, because this is quite a few now - three what I would call front-end that are affecting users, bugs that have surfaced in the last week which in my experience over the last decade or so I haven't really seen that happen before.

Secondly we've got some Google Search Console updates.

The first is that Search Console is now reporting your site's Discover performance data, and a surprising amount of people I've spoken to haven't encountered Discover before - Google Discover. If we call Rich Snippets position zero, then Google Discover would be minus one. It's the content Google surfaces before doing a search. Google describes Discover like this:

I actually know some sites do actually generate more traffic from Google Discover month on month than they do from normal organic search. There are specific ways you can optimize for discover you can find a link in the show notes as well to Google's advice on how to do that. Here's what Google sang on a new report in Google Search Console:

I think surfacing this within Google search console will most likely turn a lot of SEO’s attention and focus into optimising for Discover as well, it's certainly one of those areas of search that's commonly overlooked. I think Google Images is another one, Google images is pretty much the second largest search where it's called going and there's still loads of people that don't bother optimising images.

The second update to Google Search Console is something they actually told us about way back in February and that is that Google have consolidated URL data in the performance reports to the canonical URL only.Sso this means that accelerated mobile pages (AMP), mobile pages, and all non canonical pages will only be counted towards the main URL’s performance in the report. Until this week you could actually still access the non-canonical data in the old reporting interface, but this has now as of the 10th of April been removed so if you're just for the first time switching from the old to the new data and you haven't logged in a while and you're seeing a big shift in your data - that's what you need to know and that's why!

Finally I just wanted to share a really interesting new Google Ads format that's in beta with you. If a Google Ad format or feature is in beta, it typically means that only Premier Google partners will get access. I checked with our team - it's not something that we've been given access to as of yet, so I reached out to a few other people I know and it does seem the new format is incredibly rare. I've included a screenshot in the show notes at search.withcandour.co.uk so you can have a look at it for yourself. I've yet to see this type of ad myself in the wild, but after some digging I found out these new beta ads are called 'Gallery ads’, so I'll describe the ad for you for those who are just listening to the audio. The screenshot shows a search result for the term ‘holidays to Croatia’, and at the top ad space you have a single headline - so that's the blue headline at the top which says ‘Croatia holidays grab a great deal today’ and directly below that you've got the green URL which in this case is for jet2holidays.com. So there's no second line or description here, but what's different and what's below it is actually a really quite large picture that's taking up on this screenshot a quarter of the screen. So we can only see the top of the second ad here. There's a big photo of a beautiful coastline, I assume in Croatia, and in the bottom right of the image it says one of six and then there's one more line below the image that says ‘it's not too late to bag your ideal jet2holiday.’ So what we're seeing here is obviously large images within Google Ads that are taking up a massive amount of real estate which is very very significant.

I struggled to find someone that knew much more about these ads but I did manage to speak to Dave Hatton who is the Head of Paid Media at Reddico, who is able to confirm a few more interesting details for me.

Apart from the holiday and travel industry he's also seen these ads live in the automotive industry and he points out it's actually a really cool format for car ads. Secondly it's very likely you'll need to be a really top end Google Ads spender to qualify for these beta ads, and thirdly he helpfully confirmed that Suzuki and Jaguar Land Rover are currently using them out in the wild too so it'll be interesting if any of you listening can find examples of these as well. He was kind enough as well to give me some extra thoughts which were these: Dave said “it sounds to me like a similar process and Industry availability as the old image extensions Google released on beta before, the gallery/carousel approach is an upgrade on them though and they seem bigger in general too - I have a feeling this time they'll be around to stay. Could be a game-changer really, even more fighting for that number one position. Who would have thought Google would want to do such a thing and make advertisers fight even more for position one.

That's everything for this week's episode of Search With Candour, you can get all of this supporting show notes at search.withcandour.co.uk. I really hope you’ve taken away something useful and you'll tune in to our next episode on Monday the 22nd of April. My name's Mark Williams Cook, take care!

More from the blog