Exploring the new Googlebot user-agent strings, best practise on internal...
Or get it on:
In this episode you will hear a SearchNorwich recording of Chris Green giving his talk "Essential technical SEO learnings from launching and migrating over 120 websites". Chris goes into great detail about the importance and challenges of site migrations from both a technical and communication point of view.
MC: Welcome to Episode 31 of the Search with Candour podcast! Recorded on Friday the 11th of October 2019. My name is Mark Williams-Cook and this week we are going to be talking about site migrations and more specifically, we'll be delivering you a talk done by Chris Green, who is now the Head of Marketing innovation at Footprint Digital.
Chris is an incredibly experienced SEO, who I followed over many years. He was kind enough a few weeks ago, to come and make a trip to Norwich and give a talk about site migrations. It's a fantastic talk and Chris draws on his experience of migrating over a 120 websites, over the last many years, and he talks about, not just the technical aspect but the often overlooked, internal communication expectation setting aspects of a site migration, that are really important. I hope you enjoy it!
CG: Cool, so a fairly snappy title talk, kind of, wanting to try and sell the gravitas of some of this maybe almost too much before we start. but I'm head of search but I'm an SEO apologist of too long, blagged a job as an SEO after content writing for a little while and things kind of got out of hand. I think of migrated a lot of websites, I've put a 120 on there, but I think the actual number is probably higher, but I was told that seemed/ sounded unrealistic, so I've gone for a conservative number it's quite a few.
So as I said, I've migrated a few of these, to say the least and I've seen the good, the bad, the ups, the downs - 120, probably more. To give you an idea of the makeup of it because some of you are probably data people and I got to make a new graph, the kind of spread or calibration of these, most of them are WordPress some of the magento and then some are on whatever other CMS that's being built or used for the job.
One of the key points of this is, 70% of the websites that I've worked on, I worked with in the company that built and launched them, which is kind of important so bear with me. I get a chance to refine this process, I get to refine what works, what doesn't work, I get to see it happen so repetitively and I get to influence that directly, but it does kind of give me a unique sort of position on it all. Without wishing to brag, I believe that I'm in an envious position on this, if you kind of subtract the damage that my growing websites is a living kind of does to a person and still got hair which I'm grateful of. SEO’ing, which I should’ve put in inverted commas, these CMS's is not hard fundamentally, if you know what you're doing and actually working in the same room as the development team has its advantages, some minor ones but we forgive them those because they're cool people.
Who is a developer in this room? Sorry, I will make the casual you know, will kind of draw attention to you all unfairly and all sorts, but what I want you to do as part of this though is how I learn from me and learn from the others you learn from some of the successful examples. Actually I think I've only shared with you the not so successful ones, as kind of, an active, maybe hubris and not enough people do that. I haven’t named and shamed everyone because I'm also not daft, but come on, let's learn from this.
So this, this is stressful stuff and that kind of polar bear has represented me on a number of occasions, mainly with migrating websites but also other things. But when you're standing at the top of a migration like that, for those who don't know this is visibility data, this is how
visible you are within Google, if you're standing at the top of that, you don't know what's about to happen after the migration takes place, that's a pretty scary place right?
Now this projector is not as good as I hoped, that is s*** and but that emoji kind of does the rest of the story. Right so you don't ever want to be in there, but you know me saying this you're stressful stuff, this is Lush - I've named and shamed them because it's a public data and it's kind of a good story. It happened a long time ago, you never want to be the person that's worked on that project ever and if you don't like Lush. I think it's already been mentioned at least once tonight, but let’s do it again - migrating a website is the most risky thing you can do for a website from an SEO perspective, especially if you put any hard work or time into building it. Some people in projects get that, others really, really don't, so if you're really not sure of that before you start, hammer this one home, you know you'll be called the naysayer in the room, but that's more satisfying than I told you so, especially with that.
So in this time, I’m going to give you some of my essential learnings from my experiences. Now, these aren't sort of technical, technical minutiae, so if anyone's nodding off already because they've heard an SEO go off on one, I will go off on one but not about the small stuff so hopefully it will be slightly more interesting - Are we ready?
Good, right reaction! Can someone time check me if I go on too long. Lovely, okay that might happen.
Number one; key essential point, migrations do not have to equal a drop and drop in traffic and rankings. Quite a simple one right? But I've heard this quite a lot. I've heard other account managers, ones that I've worked with, not that many, others that I've had the opportunity to work with as a third party expect at least 15% drop. Simple question, why do we expect that? Well this is expectation management, which is important, but it's from someone who gets it wrong often or doesn't have confidence in their own ability to maintain this - could be self deprecating, it could be you know set the expectation low and come out as the hero but that thought process that you can start a project with can damage the entire thing because it leads to bad decisions.
When ranking drops are expected however, is when you start cutting out content unnecessarily. This is a real example, it was from a project I did a long time ago, Rentokil wanted to cut - they had a three thousand page website, they wanted to cut it to a 30 page - that would have been a bad idea! Someone just shook their head - good person. If the content focus is change that can also hurt rankings understandably, you change the focus, what the website is doing, what you want people to do, Google is going to treat it differently.
When you lose an argument in design, when your sitemap goes up in flames, there's a good there's a big point here about actually compromising, but that can ultimately lead to a drop in the rankings and this one brings me on to an example, one of the ones that I've worked on. So again visibility data red represents old website, blue represents a new one. now the decision that was made as part of the process was that we don't want the site structure to be as SEO e but we will pay more to market it so we can maybe offset these changes with new content and marketing efforts later on. hmm wasn't hugely happy with that but they sai they were going to up their retainer to offset it and they were very strong on that conviction. we got to about here and they say that's thing we don't want to spend anymore at the moment and sadly that didn't get an awful lot better. that's now though when we're still with them.
When it is okay not to rank, first thing when people are aware, so SEO and actually you know not just SEO is but non SEO, they see keyword search volume it's available and they like I've got to have that because the bigger number is better than a smaller number but you don't have to rank for everything and if you is the SEO or the person that's holding this project makes people aware that's fine. It's okay not to rank when those keywords just aren't important, the commercial intent is not there or what's this traffic going to do it's not going to help you so that's fine. But also when it's a phase 2 or Free Edition, so when someone decides midway through the project that they want to reach for something further or they want to change the spec, that's not just an SEO thing that's just a web build kind of issue but when it's phase two or three set the expectations.
Now to control this, something I'll call C, but that is setting expectations early and that's probably one of the most important things I've learned in SEO is that my biggest challenge isn't actually Google as much as I kind of shake my fist at it, it's actually the client expectations I'm working with and setting that accurately and controlling this process early on is probably one of the biggest keys to success or certainly perception of such. If the objective is cutting site content by 90 percent, objective to can't be retaining all existing traffic - there is a small Asterix if anyone's going to be slightly pedantic and say cutting you know excess content, cutting zero view content, there's there are there's content that can damage you but this isn't the norm, please don't tell people it is. You need to get organized first, ducks in the line, concept got me and I spent too long making the slide. I'll come back some more on this later.
At the end of each section I'm trying to give this is like a little bit of a condensed version of my ramblings, the slides will go online later so don't worry about taking one down. But be organized, benchmark - so tell people where you are currently, it's how you get set expectations - if they don't know what they're currently ranking for, they might discover they're not doing as well as they thought they were after you've launched the new site, trying to convince them that wasn't you, it's hard. Any decision which will detriment post-launch performance needs to be raised at that point in time, it's okay to make that decision but a bad thing will happen - if they then don't like that, then you can have that there and then just set those expectations early.
Cool, point two! The most meticulously planned migrations are very very easy to sabotage and I've seen this too often. A migration can be condensed into a list of really simple tasks; it's actually not that difficult really, it's what I'm not talking to you about the point-by-point process. You know, there's all this stuff putting all these ticks in separately sounded great when I made it, all of these things you can kind of do you can make sure that they work I built a list of more stuff if you actually want to go and see what something looks like. It's not an exhaustive list so don't quote me but there’s more there. The important point is, if you meet any one of these your site could get D indexed, you can create some spider traps so you could keep google caught up in in content that you don't want it to, it could kind of freak it out. You can effectively point Google back to your development website, canonicalize back which is ultimately a bad thing or maybe your 301 redirects don't work and all of the other disastrous stuff potentially worse, who knows. But these checkpoints part the reason why these checklists don't fail is that your checkpoints on these lists is not frequent enough, you kind of you do it once you assume that it's done and you walk away.
Who's read a book Called turn the ship around? Not entirely sure, it's a bit random but it's a nuclear submarine captain which is entirely relevant for it, I do it on a day to day basis! Basically, changing the behaviour rof it of a team, fascinating especially if you like large submarines, but there's a concept in there called a little rudder far from the rocks which actually I think has a real purtance to this and to illustrate that Cromer Pier is in this screenshot because I've localized my content, and an impossibly large cruise liner that probably won't fit there. So concept a little rudder far from the rocks you see Cromer Pier fast approaching, you turn hard and you miss it you, maybe wave on the way past but that's all that’s done, you make that discovery early, or sorry, too late, you still try to turn, you try to avoid it. This in a site launch context, is a team somewhere, working overtime - it might not be an SEO, it's probably a developer - sorry guys you get shat on in this process, I don't encourage that behavior but that happens.
What other stuff? well okay you try and reverse course, you and pull hard back, you know let’s avoid this! This is realigning expectations, actually you know what we've gone back to our spec or we've gone back to what you've told us to do, it's not gonna be quite how you expected it but we're still cool right? Then you evacuate the pier because this isn't going to be fun, so actually this is a last-minute scramble to minimise losses. What can we affect that will cause the least amount, what can we do after the launch, what planning, what preparations, but that still happens. No one's getting candy floss. So I enjoyed making those slides too much to not put them in there, but what well where, what you know, the later you leave these kind of exploration is the more trouble it does cause you, trust me - I've left it too late on some occasions; I haven't torched Cromer Pier.
So, your migrations list, pre and post launch is really critical but what is more critical is the person that runs that you know, builds the project or plans it shouldn't be the one that signs that list off. You know you don't proof your own work when you send it, you don't, all this is too important to kind of, leave it up to human error. Little and often meetings are great to ensure good comms, particularly if you have an SEO who's very excitable that loves to talk about it, if I have a project members don't they nod off things get missed; little and often helps much better and ensure your launch timeline includes ample time for these checks and you cannot sacrifice that. it is not worth it, believe me, because it's hard to undo.
Cool, number three! This is a list of ten by the way so you can kind of workout how close we're getting! A new platform and high technical standards won't guarantee rankings, which is a shame, but I'm a tech SEO man and I hate to admit this but you've launched a new website you've nailed everything, you've got it, you’ve got a clean crawl - all of these things that you can't have audit websites for that you constantly see after you've done all of this, it's perfect as far as you're concerned. But did you actually fix the original problem? You know, sometimes you may have 404 errors, that is not good, but that is not stopping you ranking because if anyone crawls or looks at a big website, a big brand they're covered with problems, they're covered with blemishes and issues, it doesn't actually stop them - I know it's not a fair comparison but the original problem is often left. You know I love this one, all images now have alt text and optimized title - woohoo, well done - is that a significant value add to this? There will be some websites where it will be, but if it's a wordpress blog with 80 pages, yeah it's not pushing the needle is it. Your meta graph of meta descriptions are written perfectly yeah, but we've done a few studies on Google rewrites, these an awful lot which isn't great and not direct ranking factor discussed later. You've got correct schema on every page but schema doesn't really mean extra rankings does it? - again discuss later. I'm making some sweeping statements, you can more than happy challenge me on these! You've got no crawl errors, amazing but that's a standard unless that was a significant problem before it's just good hygiene, again sorry, and you got no excess blow, all of the content that is indexable is exactly what needs to be seen, but again it's just good housekeeping - it's not what's gonna make you better than everyone else, unless everyone else you sh#t.
Bonus point, the good website does not equal ranking better. no matter how slow and loudly I repeat that sometimes it doesn't always seem to land. Google doesn't care what it looks like and actually in some markets, users don't care what it looks like which can be hard to hear especially when all the colors are contrasting and all the calls to action are loud and ugly but sorry. don't assume tech SEO hygiene equals ranking improvements, more importantly, understand what tech SEO can't fix as part of this because you probably need to talk about that before you get to site standards and a lot of what you read about is the best practice isn't really applicable broadly across the spectrum or the impact of best practice won't always be that significant.
This feeds back into setting expectations because again, we'll sort all this out, your site will rank better, sitemaps not changed, it's still a problem. None of the major SEO or major CMS's or platforms are so bad you can't SEO them and this is my bold statement alert. I don't actually think I think that a successful website can be built on any modern functional CMS, which by implication mean I'm putting like Wix and a few others into that lot - no boo’s or hisses which is good - I actually did a bit of a search around search intent around Wix just to prove my point. Wix is bad, so slow, garbage, bad for SEO etc etc - it's very easy to point fingers when things go bad. I appreciate that that screenshot may be a little too soon. But assuming that your CMS can complete the basic tasks that it needs to, it can potentially rank or SEO’d. Shopify is not Magento or WooCommerce, but Shopify can still work; I don't like Wix but if you're making a blog, it can still work. I'm not advocating these I'm still and still an open-source guy but please don't be defeatist and walk away at that point and don't confuse disliking a CMS with being unable to work with it, time and a good developer can fix most problems if you don't have that, you've got bigger fish to fry than just your tech SEO because you probably can overshoot your website.
Ah yes, but this one in the middle. Sometimes weird sh#t happens, I put this in the middle because I don't want us to finish on this point because that might undermine what I'm trying to say, so I put it in the middle. Even if you think you have nailed it, sometimes that isn't enough. Is it the lack of skill? Are we unlucky? Is it Google trying to trip us up? Are we bad people? Maybe it's because the system is complex and opaque because it is or it's all of them - yeah it could be all of them, sometimes it's just because you've done a bad job.
This is my migration from hell this is one of my personal examples that I've worked on and that red line is when we launch the new website and that red line doing that implies that I may have lost a few nights of sleep. had a fresh look had 75 plus pages of new unique content SEO standards 301 redirects most comprehensive that ended up telling a bit Trumpy and log files pre post launch were okay. things were getting crawled that nearly - technically I felt that this was tight. But why would I do that? Now actually I took a bit of comfort that it continued to fall, long after that sort of, post launch settling down window, because actually that - we've done some digging since and there's something far bigger at play here, kind of some suspect linking and some negative SEO and a few other elements that can suggest a multiple factors yeah so it's not just the migration, I don't actually think on this project I'd totally screwed up! The site still went in the wrong direction.
The vast majority of this migrations this doesn't happen you know, I've migrated a lot of websites and launched a lot of platforms, this is rare, this is really rare for me - not bragging - but it should be rare because it's relatively simple.
I missed that point though didn't I? No I didn't, that's fine. So if you, if you can't find the answers related to on-site factors they could correlate elsewhere. Who’s tried launching a new website to find out Google's launching a major algorithm update at the same time? At least one of us, two of us, thank you. Um, that muddies the water because finding out what the problem is can be tough. Find someone else to bounce ideas off, who you trust and who’s not in the project because sometimes they'll help you see stuff you might’ve missed. Don't get complacent, I'm good at my job I haven't got anything wrong, create fallback plan - be careful which stakeholders you tell it to because sometimes they can see that as admitting defeat early but that can help.
Number six! Most launches migrations fail because they're complex. I'm conscience, am I kinda running out of time? Good, love it and I won't rush this one. so more variables equals more chances to screw things up, so a bad idea changing the CMS, changing the protocol so non secure to secure and the domain in one go - don't do it, please; even if you're in a real hurry! An okay idea, maybe two out of the three, but it's a third better and a better idea. Change the CMS first because that's a big deal and then wait, which is very bad if you're chronically impatient but how you pitch this, this is a choice between, well struggling to identify the cause of the drops which can cost weeks months, depending on the size of the project, thousands or millions of pounds
dollars etc, or being waiting weeks between these events and actually being more you know, you can set expectations far better. I've seen too many fail by this, this is kind of hubris is ambition we've got this nailed, I've done this tens of times, it's not always great. So my recommendation here is make one large change at once and clearly spell out the risks of adding each additional variable. But also if people are insisting on increasing the variables well you just gotta increase the expectations of how long it's going to take you to check this stuff because they're making your job harder; don't say ‘yeah we can launch it at the same time’ because you've effectively got more work to do, so give yourself the breathing space.
Yes, I love this one. So number seven - invest buying upfront and it saves massively on fixing screw-ups later on. and being in digital marketing I'd turn to the Dalai Lama from the inspiration. in order to carry a positive action we must develop here a positive vision; get everyone bought in get them to see what you're trying to do before you start. I don't often go back to Dalai Lama but it seems very fitting here, otherwise make friends with developers. I mean, thanks guys. I love how like he just does not want to be there - I've used that slide about eight times. and the other thing is dogmatism is the biggest enemy of SEO and I don't know why Donald Trump's appeared on that slide, it's completely coincidental, but actually being that one that sits in the corner. my implementations are not being made I'm now not bought into this process I'm going to help, doesn't actually help the processing in fact, it makes things entirely worse because SEO are making, they’re great at making themselves really unpopular on projects. some of this is really hard to read but, so I don't know these social media icons don't have an alt text,someone fix otherwise we can't launch the website, doesn’t actually matter… your heading structure is broken, your sites will fail, which seems a little bit extreme. all I've seen and heard variations of all of these, I may or may not have given one or two of these recommendations in the past to. I don't really matter, I'm not advocating bad standards and relaxing it, but you're just gonna annoy people. Let go of the stuff you know is BS, you know, if you're holding up an entire project and making someone's life trickier and it’s not actually gonna make a difference, it isn't helping anybody.
Now you know equally, you have to compromise, that's fine. But you have to stick with the stuff that matters you know, again some of these points, where's the breadcrumb? You're not going to convince me that I don't get to have a breadcrumb on desktop and mobile, sorry people. My designers don't like me for this but it always makes its way on there at the end, I think it's pretty important - prove me wrong, I'd love to it. Um, these filters are going to waste a lot of crawl budget, these pages are more than three hops from home; this is the stuff you need to be making people aware of and getting them to change. so people do perform better when they understand why they're doing what they're doing, I mean that's that's not an SEO thing, let's just get them bought into it right. but confront any objections early on if someone is if someone is setting their cross you know like this at the other end of the meeting table get that out sooner than later and don't assume that everyone's goals are the same as your own; find that shared goal, work towards it, don't be dogmatic, don't be that one.
If you launch in website with someone, surely everyone wants it to succeed. There's some questions about large websites and corporate politics but we'll leave that. Assumptions lead to the biggest mistakes, I could have said put a picture of an ass in there and gone without, but I left it. I thought such-and-such would have checked the redirects before launch - it's good I'm glad I'm not the only one that's heard that. It was built on WordPress, it shouldn't have been a problem. Lovely, why would they have added a second canonical tag? I don’t know. Managing this can be tough, but let's be honest, we have to assume things because websites are vastly complicated and you cannot explain everything, you know is href, that's a link. I've seen people build these on, in weird ways, you know but assume, you want to assume that people do these rights - it's a web title, it's a canonical tag but the point is that everything Googlebot sees, as an SEO you have to be pretty obsessed with and you can't leave things up to chance. there's your sheriff's badge, you've got to police it you know. One of my team are basically called Batman because every time she sees an issue, she swoops in and make someone's life difficult in order to fix it. I'm just glad when it's not me.
Trust that everyone has the best intentions, but monitor it like you expecting the greatest screw-ups. I'm sorry if I keep moving out the way cameraman. Yes there's another migration that I worked on - blue is old, red is new - these random spikes are actually a measurement issue, I can explain it later but it’s actually boring. We launched this website, well it's, we launched this website it's very light this green isn't it, each of these blue spikes i've denoted with red arrows is a problem because that is old redirects not being listened to anymore. the first one cost about two weeks disruption in terms of the original migration and it was caused by somebody - the host of the website deleting the htaccess file and replacing with a blank one and not telling us which i got slightly miffed about and then it kept happening. So you've gotta monitor this, I don't think they're bad people and trying to sabotage the website, but you know back to number five; sh#t happens. So check everything you can, everything that Googlebot sees you have to be obsessed with, don't shortcut the pre and post launch checks, we said that at least once and don't assume that every website will be built the same way or correctly. You know a lot of tech SEO is just good websites, which is kind of patronizing when you say that to a web developer but it is but not everyone does it in the same fashion.
Cool. SEO builds where bad websites but so do designers and developers, and what I mean by that is - in isolation. The vast majority of cases, one personal skill set will make a bad website and if it works you're either lucky or you're paying someone a shedload of money. But those who have suffered poor migration experiences in the past usually have an over reliance on an SEO in terms of the new build, they've learned their lesson, they're not going to let that happen again. Great an SEO friendly template, plan out the information architecture and tell the content writers what to write. I'm a tech SEO, that's a bad idea. As long as content that ranks well, is not exactly the same as content that sells well, you need more people around the table, again some people may want to debate this later on but fundamentally content that converts is not exactly the same as content ranks. It will be/ could be soon but collaboration is what's needed here. So, iterate the site map based on the keyword research and findings, as the SEO I want to be working on the sitemap of someone who spent time getting to the client, the brief, the uses, the personas - I want to be iterating there. I want to be recommending information architecture change based on the keyword research and then I want to outline the content, the quantities, what content needs to be there, who ranks, based on the competitive landscape you know, this information that the content team might not have but I do - this is a collaborative process. So develop a healthy respect for the key project team; agree involvement in milestones, when SEO is needed but know when something isn't yours to own, you know round pegs fit better in round holes.
Nearly there! On time as well, lovely.
The biggest variable in this process is often organisational size, not the CMS. That one always fails to disappoint; I suppose, you know, you've been given the CMS platform that you've never heard of, that's not your biggest problem; it's why you've got to that point that could be. More stakeholders equals more opinions, we all hear about hippos but there's only so many that you can encounter before things get boring. Recommendations inevitably get watered down, which usually leads to things going badly, dolls get compromised. I can't claim to owning this statement but this is a bit of a truism, that not all tech SEO problems are technical. In fact actually, startlingly few are, which if you are a technical person this one is hard to really sit with because you have to deal with humans and weird stuff like that rather than Googlebot.
Sometimes a battle is better fought with the CMO rather than the CMS, which often if you're brought in too late in the process you might not be the one to make that, but if you are working on a project and there is a project lead or someone right at the top of table, you need to make sure this one lands, if you can. It’s not always easy, but take as much time framing the recommendations as you do formulating them, very often the solution is simple, getting yourself there is the problem.
Cool, so let's wrap it up. If you really struggle to know where to start in this because every migration is different, they're all vastly complicated, they all different challenges but invest in setting expectations - early being the 11th hour SEO, it's a recipe for disaster more often than not, unless you're damn good or the project's easy. Benchmarking because I've saved thousands, if not tens of thousands, of visits and traffic by just going, actually guys we're about to cut this page, but it gets a lot of traffic where is it going? It's not a difficult process, you know, benchmark it. I've been lazy and just link to a moz guide but there's some pretty good stuff in here that can help you get on that process. And actually the biggest issue why I see migrations fail is your redirect mapping, you know old pages to new isn't comprehensive enough, you've taken liberties and shortcuts.
Now actually this is Simon, I work with Simon, he’s developer - he's probably one of the cleverest people I know and he inadvertently makes me look good. We did a study because developers will often push back on too many redirects because it can slow a site down, did a bit of research that suggests that might be true, that might not be, but comprehensive 301 mapping is what you need and then there's the technical messy bit later on. But as the SEO stakeholder, redirect everything that you can that makes sense, unless your site is millions of pages and again, then you can talk to me, a slightly different problem.
And finally, oh yeah be excellent to each other! Lovely, thank you.
MC: That was Chris green from Search Norwich 8 and our next search Norwich is in November, if you'd like to come and we've got Andrew Smith the ex head of SEO for companies like Expedia, Cheap Flights and eDreams, who is coming to talk about big site SEO and how to manage that. We've also got Steven Van Vestan who is the co-founder of ContentKing, who will hopefully, very entertainingly, be talking to us about SEO disasters and cracking that open and how talking about how we don't actually like to talk about them but he's going to bring some great examples and do some autopsies on what went wrong with this. and you can get all of the show notes as usual at search.withcandour.co.uk. Okay and transcriptions will be coming soon, I promise. Next week, which will be Monday the 21st of October I'll be back with Rob Lewis and we will have an episode for you about blackhat pay-per-click.
Exploring the new Googlebot user-agent strings, best practise on internal...
In this episode, we'll discuss the first data on the September core update,...
Get in touch