In going through Search Console’s crawl stats, we noticed that generally speaking, Google crawls a very small amount of data per page. It makes a lot more sense that Google is crawling just the html. Our next thought went to, how much html will Google crawl? It would be important to know as there might be a limit and anything above that limit might never be seen by Google.
Test Downloads
Transcription Below
How Much Will Google Crawl?
I talked about this at SEORockstars where I noticed when I was looking in Search Console, Search Console gives you these crawl stats and what they show is basically how much Google is crawling your site in a single day, it will show you the number of pages that were crawled and it tells you the amount of kilobytes that were downloaded. What I realized, when you do a simple math, that is only 15.8 kilobytes per page. That is not a lot of information. In fact, that is a very small amount of information. The average page size is 2.5 megabytes. So I was thinking about that for a second, they are either looking at the first two words of your metatitle and moving on or they are not looking at your page exactly and I think what they are looking at is the html. The average html is about 80 kilobytes so that would make a little more sense on averages especially on this situation that you could get a decent chunk of a page if you crawled in 15 kilobytes.
What I wanted to find though is how much will Google crawl? What is that actual number and with this being so small, maybe a 100 kilobytes might be it and that is the scary thought because there are pages larger than that in their html. So, for people that recommend really long form pages, that could be a real disservice because perhaps Google is not getting through half of that. That was my thought process in doing this test.
This test took me about 3 months to run because what I did was, I first started off with building out pages up to a 100 kilobytes by 5 kilobytes so I started at 5, 10, 15, 20, etc. and I got to 100 and actually, Google crawled them all so I had to keep going.
This is what it looked like. At the very end was a target keyword. The concept was, if Google crawls all of these content, then we know Google is crawling 5 kilobytes. This is from Pingdom, I pinged the page and you can see the whole page is just all html so nothing would get in the way. I just built the page entirely in html. As I built out the pages, the 10 kilobytes, a new fake keyword would go to the end of that. We could see that Google has cached the page and indexed it – you can search by the keyword and here it is, it pops out. That is how we could confirm that Google is crawling the whole thing. So then, the idea is that we just need to keep building the pages out until we find the point that Google stops and then that is the end of the road.
I built up to 1000 and we were still fine. What I did was I built out until 3000 cause I got tired of building out these little chunks and waiting for them all to index. Hilariously, all I needed to go was one more hundred because 1000 is where it stopped.
Here is the 1000 kilobytes, you could see that it was cached and indexed and then here is the 1100, the page is cached and it is not indexed. The road stops at about 1000 kilobytes. The good news is that, that is a huge page. A 1000 kilobyte of html, more likely than not is a massive page. So there probably is no size restrictions there. The thing that you might want to worry about a little bit is that, it varies site by site. Some sites in my Search Console gets 8-9 kilobytes per day, some are in that 15 range, some are over 20. So you need to go in and look at what your site is getting but let’s say that you build a page that is 60 kilobytes of html, it might take three crawls, presumably that Google goes in and remembers where it left off and then continues from that point through the page and if that page is getting crawled once every 5 days, that could be 15 days before Google gets through the whole thing. The other problem that you could run into is that maybe if you do a change at the bottom of the page and Google is in the process where it is at the top of the page, it might not get there for quite some time, depending on how much it has crawled, is crawling, how big your page is, and how often it is crawling.
So, if you make any changes at all, cause what is interesting is that this test site was getting one kilobyte crawled a day. But, I could do fetch and render on all of these pages and Google will crawl the whole page. What this tells me is if you make a change on your page or you build a brand new page, you should go into Search Console and fetch the page, Google will crawl the whole page then. If you don’t do that, you really run the risk of Google not seeing the changes for quite some time. So you could make some on page changes that you expect would do something in just a couple of days because it is a significant change or you are doing something that is very strong like with your h1s and metas, there is a chance that Google will not get there for quite some time. So go into Search Console, fetch your page, and I think you will get your results a lot faster.
A lot of people use fetch as a best practice because it is just a good idea, but this is why it is a good practice. The other interesting thing is that pages that were over 30 kilobytes, I would hit fetch, they would cache quite quickly but it would actually take 2 or 3 days for them to index which was interesting as well. So, the caching and the indexing, appearing in search results are not one to one. You will cache quickly, Google will crawl it, cache it, it sees it, but it will actually take a couple of days before things filter through into the index so keep that in mind as well. So you’ve got a full site, a big page, in a live environment, it could take a few days for things to actually crunch and for Google to understand what is going on in there even if it has crawled your page quickly.
Is there a penalty if you manually fetch your pages every time that you make a change?
I do not see why there would. It is a feature within Search Console for you to use and it doesn’t say not to do it regularly.