Episode 49: The SEO gap in web development

Play this episode:

Or get it on:

• Apple iTunes
• Anchor
• Breaker
• Google Podcasts
• Overcast
• Pocket Casts
• RadioPublic
• Spotify
• Sticher

What's in this episode?

In this episode, you will hear a SearchNorwich recording from Mark Williams-Cook giving his talk "Won't somebody think of the robots?". Mark shares his experiences of vital SEO tasks that are often missed during web builds, as they don't directly impact the end user.

Full video:

Slides:

Won't somebody think of the robots? - Mark Williams-Cook - SearchNorwich 4 from SearchNorwich

Transcription

MC: Welcome to episode 49 of the Search with Candour podcast - The SEO gap in web development. My name is Mark Williams-Cook and this episode was recorded on Sunday the 23rd of February 2020.

Which of the aspects of SEO are commonly missed by web developers is what we'll be talking about in this episode. before we kick off I want to do an announcement for our listeners who are based in and around Norwich and that's that our next free SEO and PPC meetup, Search Norwich is happening in about three weeks time, on Thursday the 12th of March. As part of this event, one of our sponsors SISTRIX is offering a whole day of training on their platform for free.

So SISTRIX is this whole SEO toolbox for those that aren't aware of it. There's only one space available per company, we've been given eight spaces for the day, five of those went pretty much immediately. So we've got three spaces left. The training is 10:00 a.m. till 5 p.m. foods provided and you'll learn how to discover growth potential with new keywords, find broken links, see web pages, on-site errors, things like this. This is suitable for beginners or advanced really, anyone who's in SEO and wants to learn a little bit more about that SISTRIX toolbox. You can find a link to sign up at our show notes which are out search.withcandour.co.uk or just google SISTRIX toolbox training seminar; you will be able to find their page that lists a few locations they're running at and you can just click down to Norwich and sign on there.

Search Norwich itself is on the same day 12th of March, it kicks off at 6:15 in the evening and we're joined by Natalie Mott, who'll be talking about SEO myths and misconceptions that still exist in 2020, so I'm sure she's gonna have a lot to talk about there. We've also got Ross Tavendale, the MD of Type A Media; he's coming down to give a talk about SEO strategies versus tactics and the most common problems businesses face understanding and implementing SEO. As usual, the event will be supporting a charity, we're going to have the Benjamin Foundation come again. The Benjamin Foundation actually was the first charity we supported when we started the events over a year ago, they're coming back to talk to us about the work they're doing in and around the city with homelessness and on that same thread, we support The Feed as well who are a charity about getting people off the streets; they provide our food for the event. So it's 12th of March, you can find out more about the speaker's, the event, what food is available if you have a look at searchnorwich.org.

The talk I want to show you today is from a Search Norwich at the beginning of last year, so beginning I think it's January 2019 it, funnily enough, was when I had to deliver because we have a speaker drop out about three or four hours before the event, so I ended up doing a talk I hadn't done before, on a subject that's always at the front of my mind. You know, because I manage a business that does both web builds and SEO, it's this natural blind spot that occurs during the development process when it comes to SEO and I want to make clear here, that I'm not saying this is developers or that developers are at fault, these oversights are the ones that only affect search engine robots.

So it's totally understandable when you think that a web developer will likely approach a build project with the user at the centre of their mind, hopefully. So they'll be thinking about what makes things better and easier for the user and the things I'm talking about these gaps are things like server-side rendering, canonical tags, hreflang, schema, they all have zero direct impact on the user when they're on the site, all of these things are there just to help search engines understand what's going on. So unless you're aware of them or unless the CMS, I guess takes care of them without you knowing, you're going to miss them. I wanted to show his talk with you today just because over the last couple of weeks we've had some quite public cases of websites basically tanking themselves in search because these gaps haven't been closed. Enjoy.

I've jumped in with a two hour, from beginning to end prepared presentation for you about SEO, so you're getting a double dose of SEO. I've called it “Won't somebody think of the robots?” and what I want to talk to you about is the parts of SEO that happen, that are really really important, that you can't see as a user, so they commonly get overlooked. So this talk is for one, people who are managing website projects, so there's gonna be five things I talk about and you'll have a list so if you're in that situation, have we done this? Okay, it's for you if you're working in SEO to improve your relationship with developers and if you are a developer, I've got some technical implementation bits in here as well for you.

So for those that you don't know me, my name's Mark Williams-Cook, I’m a Director at Candour, I’ve been working in SEO for about 15 years. I spent a few of those years as an affiliate marketer, sitting at home getting up at the crack of midday, working on my own websites, thinking it was amazing and for anyone that has worked at home for a long period you might be able to sympathise or empathise that you do after a while, you hope that pigeon comes back, it's kind of lonely. So I decided it'd be a good idea to go agency side, so I now look more like this and I've been really, really, really lucky. I have got to work with people like Expedia, Hitachi, Fat Face and really get some big SEO challenges, which is something I never would have done if I stayed like that. And one thing I have noticed and this is as coming into agencies, with development teams and especially when we go and talk to clients who have in-house development teams is you get introduced as, hey here's mark he's the SEO, and developers look at you like this, they're like alright there's the guy, he's gonna tell me I'm doing my job wrong and he's gonna get it all wrong and it’s kind of fair enough because you'll find, maybe in this group, some of the loudest people that talk about SEO, sometimes not always, give some of the worst advice. so that does manage to trickle down and you get problems where developers are told to do things and it turns out it was wrong, it's a waste of time and it's kind of frustrating because the people who do really get into it and understand it, give very frustrating answers. So they realise the complexity and you're asking the question and they'll be like “well it depends” or “maybe” and that's the truth because there isn't a one-size-fits-all solution. so I've got five things I want to whiz through that you might come across and why they are really important and you might miss them in your build.

So the first one, it's got a really kind of long scary name; faceted navigation and canonical's but I guarantee you all know what I'm talking about. It's like if we're going to buy some running shoes and I do a search for men's running shoes and I come here, on the Nike website, you've probably all done online shopping, and we're all aware we can do things like we can change the sorting, so I want to sort by price or I want to sort by size. Now as humans, this is super easy to understand, that these two things are basically the same page and I can share one or I can share the other and someone can say here’s what I'm looking at. The problem lies with search engines that to do this, to change this order, we normally change something in the URL and why this is a problem is that search engines think a new page is just something with a new URL, so the situation you end up in is, you might have 20 different versions of this page and Google's like, hang on there's 20 pages that say men's running shoes, I don't know which one to rank and the end scenario you have is, rather than having one page that ranks really well, you get one pop up and it's like 10th and then another 20th and you just don't rank at all.

Fortunately, we do have something in our toolbox to fix this, which Nike has done, it's called a canonical tag - for developers, this is now really coming a lot in dev cycles but it's been around for years and it was ignored for years and it's essentially a way of saying to Google, hey if you've got lots of these pages that look similar, this is the one, this is the original one, so any links or any time you rank this page, this is the one we want to show. So you can forgive a developer for maybe missing this because they're focused, rightly so, on the user and this is no user problem, you know we don't care about this we understand it, search engines don't. It's one example where, as an SEO, you add value because there's something you're doing for robots, that might otherwise get missed in the development cycle. As you can tell I've got a couple of examples of these.

So this is actually, we did a technical audit on one of the UK's largest cashback sites, this was back in 2014 - it's a good example of how long these canonicals have been a thing. And just solving this issue, across a site with a few million pages took them from about 600,000 visitors a month, to around about 1.3, 1.4 million visitors a month, just solving this one issue. So the number of pages Google thought they had, went right down, but it started to realise which ones should rank. So this problem really scales with big sites and there's a 1B here as well which is, as I said earlier, things are quite complicated. So it's not always enough to just read the best practice and say, right that's how I'm going to do it, which is again really tempting to do as a developer because you follow the documentation. So to give a techie example, these canonical tags we're talking about, there's a way to put them on your website, using Google tag manager, which is a tool Google's got, you can put them on with JavaScript and then what happened was, we were doing this for some clients with some good success and we do it this way because their websites old and it's hard to update and we're like, brilliant this works really well.

Then Google last year said, oh if you do this through tag manager it won't work, and I was like huh that's weird because we're doing it and it's working. So I said to Google, hey we're doing this, so is there an exception if we use your tools that are allowing it to work and Google said no it's never worked, it doesn't work and I was like, huh it really seems to be working and then someone who was a lot more switched on and motivated than me did a full-blown study and found out that actually Google was wrong about their own technology and it absolutely definitely worked and they weren't trying to intentionally mislead us as a community, it just so happened that these things are genuinely really complicated. so it can be difficult to know where do you get your SEO advice from if sometimes you can't even trust Google. So sometimes experience does win out and so the summary here is basically, even with simple things like telling two things apart which look very different, Google's sometimes really bad at it and can get it wrong.

Another great example is site migrations, so site migrations is different types migrations we do it might be, you migrate your design, it might be you migrate your platform, it might to redo your whole website and one thing that scares me is we speak to clients and they say something like, oh don't worry our developer said they're redirecting everything - it's handled, and my heart rate starts to rise because again, it's not that simple. There's a level of correctness you need to get to and technical correct is the best kind of correct in these situations and I've got a great example for you.

So House of Fraser, rest in peace, had a very simple job to do which was they were going to migrate from their non-secure to new secure URLs, dead easy, all you have to do is redirect one to the other, no problem. Now when they went through the development bureaucracy, what ended up happening was actually three or four different hops before we got there and again to a user if you're testing that, hey it works - yeah we tested it, we went to a page and you know it took less than a second and I got there in my browser, so got through, got published, got live - this did not go down well with Google. House of Fraser halved their search visibility after this migration went live, that's around three, three and a half million organic visitors per month that represents, so I'll let you work out their conversion rate average basket and how much money that cost them. Again this came from things that are invisible - you can test it for the user and it works fine - but someone who isn't doing SEO or someone's doing SEO hasn't got involved and had a look at that.

We had a really similar situation where this is from a proposal I did about four, five months ago. We had a client come to us and say, “hey we moved site about seven or eight months ago with someone else we did a full migration” - you did for migration? “Yeah, developers said they redirected everything fine and we've lost about thirty percent of our traffic year-on-year.” and this is a company that their leads were high worth leads and they pretty much relied on traffic from Google, so this meant bottom line, they've probably lost a quarter of their revenue, which is significant, you know, its massive for a company like this. Luckily they had the analytics to the old site, we looked through it and actually we found that over 50 percent of their old URLs were not redirected and it was all things like, where there were URLs like we saw earlier with different options in and tracking URLs, marketing URLs, that had links, those links were getting the website to rank and there and forgotten about. So we audited it and found that 900 of them went to dead pages, another 400 caused server errors and this is what's caused this drop for them. And again if you just visited an old page, as a user testing it, it went to the new site, everything's fine, put it live.

JavaScript. So Javascript is super exciting; I kind of feel like this when JavaScript comes up because it is the future of the web in many ways. There are lots of exciting things you can do about it, but from an SEO point of view is kind of terrifying because there's this misunderstanding that Google can understand JavaScript, so it doesn't matter what JavaScript framework we use, Google can sort that out. That's maybe like a quarter truth, so what's actually happening is, we're all aware of our friendly Googlebot, which will come and look at our websites and all Googlebot's doing is checking for links on a page, okay, it's not executing any JavaScript. so this means, if for argument's sake you had a website and it needed JavaScript to load up the links Googlebot would go there and be like, huh it's just the one-page nevermind send that page back, which goes to caffeine which is Google's indexer, which will then execute the JavaScript and it will find, oh it had 10 links on it brilliant and it will send Google but back to look at those 10 links. The issue is that executing JavaScript is really resource heavy and Google told us at the chrome dev summit, it can take up to a week from when Googlebot finds that page to when they render it. So if you've got parts of your site that are reliant on JavaScript if you've got a 10,000-page site, it's going to take weeks or months for Google to get around your site and you can imagine, this as huge impact on the types of results that come up in search and how you rank.

There's actually a technical solution, so for devs in the room Google recommends this thing called ‘dynamic rendering’ which is what they call cloaking nowadays, which is that you deliver your website as usual to users and they can understand, they can deal with it and you have what's called a ‘server-side rendered version’ for Googlebot which basically means you run all the JavaScript, how the page ends up, that's what you give Google. So you're literally giving two different copies of your site out, depending who's asking for it. and this is one of the ways around these technical hurdles. Again, we've seen sites go live with these frameworks and it's absolutely hamstrung rankings and everyone’s scratching their heads because it's like, I don't understand we did a test crawl and it all worked and the user testing was fine and it's another example of this invisible SEO.

This is kind of my last one. So internationalisation; so that's when you have a website and it's available in multiple countries, multiple languages or both and again, the common mistake is we'll see someone say, hey we'll just redirect users automatically to their country, that's the easiest thing to do right, that's helpful. Interestingly, as of a few months ago, if you're doing that quite aggressively it's actually illegal now inside the EU to do that, which is one reason not to do it but for many years it's Google Webmaster guidelines and they have been really clear on this so this says do not use IP analysis to adapt your content, IP location analysis is difficult and generally not reliable furthermore Google may not be able to crawl variations of your site properly. That's because our friendly Googlebot normally comes from the US of A, so when it's landing on your site, if you're redirecting people based on where you think they're from, it's pretty much always going to land on the US version. So all the other versions of your site basically pretty much don’t get crawled, don't rank properly. A solution is something like Apple, where you just offer the user which version they want and you can save that and they can change it, but the SEO behind the scenes is again, there's a tool in our box to deal with this called hreflang annotations, where you can specify to Google, this is the US version, this is the Spanish version, and you can specify different languages for one place as well. as so for instance in Switzerland, you can say here is the French version for Switzerland, here is the Swiss-German version for Switzerland, here's the Italian version etc and what this will do is, it will give you a much better chance of the correct page just ranking in Google automatically which is what you want. All of these, when we publish the slides online, I've linked you to the extra information you as well.

Lastly, something I'm not going to talk about, I just want to mention it's schema. Schema is super important for SEO. the reason I'm not going to talk about it is we had Andrew Martin standing here, eight weeks ago who gave a brilliant twenty-minute presentation on schema. It's on our YouTube channel, watch it! It's very important, it's one of the two things Google said to focus on for 2018, so if you haven't focused on it yet, we’re in 2019 so you can play catch-up and I guess the thing I want to end with is, if you are a developer or if you are an SEO just focus on the fact that developers, do need good SEOs to help them out and vice-versa. so you know while we'll do our best to keep up with these technologies if you're a developer, you're the person on the front line doing the job and we're relying on you to help us do that as good as we can. that's everything I have to tell you about SEO, thank you.

I hope you enjoyed that talk and got something from it, you can get hold of the slides, see the video, the rest of the show notes and transcription at search.withcandour.co.uk. As usual, if you have enjoyed this episode, if you're enjoying the podcast and you're listening online, don't forget you can subscribe on your favourite podcasting platform and that's everything I've got for you in this episode, I hope you have a great week.