Skip to main content

Setting up Crawler Tools

Lesson 11 from: SEO Fundamentals: The Ultimate Framework

Cleo Kirkland

Setting up Crawler Tools

Lesson 11 from: SEO Fundamentals: The Ultimate Framework

Cleo Kirkland

buy this class

$00

$00
Sale Ends Soon!

starting under

$13/month*

Unlock this classplus 2000+ more >

Lesson Info

11. Setting up Crawler Tools

Lessons

Class Trailer

Chapter 1: Introduction

1

Intro- Course Overview

01:28

Chapter 2: SEO 101

2

What is a Search Engine, and how do they work

05:40
3

What Kind of Work do SEO Practitioners Do

02:01
4

Popular Tools for SEO

05:55
5

The Evolution of Google's Algo

03:24
6

Quiz: Chapter 2: SEO 101

Chapter 3: Technical Analysis

7

Analysis Introduction

03:09
8

Reporting your Findings

02:51
9

Checking Google's Index

04:12
10

Checking Google Search Console

04:46
11

Setting up Crawler Tools

04:47
12

Interpreting Crawler Tool Reports

08:14
13

Interpreting Page Speed Tool Reports

03:35
14

Checking Schema Markup

02:57
15

Quiz: Chapter 3: Technical Analysis

Chapter 4: Content Audit

16

Content Audit - Introduction

01:12
17

Basic Keyword Research

03:34
18

How to Determine What Type of Content You Should Create

03:31
19

Qualitative Content Audit

03:25
20

How to Take Your Content Inventory

07:55
21

Reviewing site Architecture

03:54
22

Keyword Optimization

06:29
23

Keyword Gap Analysis

06:18
24

Keyword Opportunity Sizing

06:24
25

Quiz: Chapter 4: Content Audit

Chapter 5: Link Audit

26

Competitive Link Audit

05:24
27

How to Find Short Term Opportunities

07:15
28

How to Check if a Domain Link is High Quality

02:06
29

The -Oops Something Happened- Audit

08:14
30

Quiz: Chapter 5: Link Audit

Chapter 6: Outreach

31

Finding Great Contributors on Fiverr

03:15
32

Finding Great Contributors with Ahrefs

00:48
33

Finding Contact Info using NinjaOutreach

03:42
34

Sending Emails Through NinjaOutreach

05:24
35

Quiz: Chapter 6: Outreach

Chapter 7: Project Management

36

SEO Project Management - Introduction

00:52
37

Wireframing and Product Specs

03:27
38

Reporting on the progress of your work using Asana

05:38
39

A word on Weekly-Monthly Reporting

08:21
40

Reporting Your SEO Impact

04:01
41

Top Tips for Project Managers

07:18
42

Quiz: Chapter 7: Project Management

Final Quiz

43

Final Quiz

Lesson Info

Setting up Crawler Tools

google search council and google's own index give you a lot of information but they don't tell you everything. They don't for instance tell you if all of your title tags are unique or if you're using the appropriate rel canonical tag for that U. R. L. For this. We prefer using tools like screaming frog, deep crawl and site bulb. Now each of these tools attempts to imitate what google bought or google sees but they aren't perfect and in fact they tend to discover different problems. So if you have the luxury of buying all the tools I'd recommend it. Or if you can find a freelancer on fiber who has all these tools out of recommend them having them run all the audits with each tool with that said I'm gonna walk you through my personal favorite tool which is sight bulb and I'll show you how to set up a crawl, read the report and walking you through this, you'll be able to get the gist of how to use the other tools. The first thing I should mention right off the bat is that there's three di...

fferent types of crawls, Discovery crawls where you're searching for unknown unknowable problems. You didn't know what was wrong site map crawls where you check the same you or else to see if something happened each week each month and then there are Q. And a crawls where you cross subset of U. R. L. S. After the team has updated something. Or maybe you were curious about a folder. So you'll crawl just that folder for the first time you audit a site, I highly recommend starting with the discovery crawl rather than just crawling the site map as our goal is to discover the problems that we didn't know about finding those weird you RLS that our site accidentally creates and all that kind of stuff. So next we want to set up our crawl site bulb has to resources that walk you in detail for the whole set up of the crawl. I'll link to them in the comments but let me walk you through my quick setup, punching the domain in this case it's kids health dot org. Note that some sites may prevent you from crawling easily try health tap for instance and you'll see all these errors pop up and it might be hard for you to crawl it, but you can work with a specialist and they'll likely find a way around it. So check out some of the technical SEO specialists on fiber and they'll help you crawl different websites, change little euros here and there. Next we want to tick off the page speed even though we will run a separate tool for page speed. Finding page speed. Like this is very helpful because the other tool will use, although it is more robust in its analysis, it doesn't crawl every euro like site bulb will we also want to check off the page resources which will instruct the crawler to touch every asset on the site. It will get every image, every plug in, every CSS script, every everything on the site. And it's important because if we have a bunch of broken images on the site, this will catch that. If you verify the site, all writing google search council, then tick off google analytics and google search council. Then finally check crawl website and xml site maps to give yourself the broadest possible crawl. If you don't click xml site map, then the crawler will only start from the homepage and spider and touch every link that it discovers from the homepage. But if you have a whole bunch of links that aren't on the homepage, it may not reach them. So when you check xml site map and crawl website, it will start its crawl on every you are L and go from there. So you're likely to cover the entire site. And if you're crawling really, really large sites, you should check out these advanced options below. Um I would also mention that you should save html so that you can track the changes on a week to week basis. I use another tool for this that has a really great ui that I mentioned earlier called S. C. O radar. But if you could only have one tool for technical seO I'd recommend site bulb and just saving the html for every crawl that you do. Finally to mention before you start your audit, you may want to let your development team know about the crawl before you run it, it's possible your crawler could tie up important resources for their servers, which would cause the site to crash, especially if you're running during peak times. Um, It's happened a few times for me, um, and I owe it to my friends and engineering to mention this. So without further ado, let's press go.

Class Materials

Bonus Materials with Purchase

SEO Research Spreadsheet