How To Perform A Niche Analysis Using Our SERP Crawling Tool

SERP Crawling is a service that allows you to:

• scan Google's top search results;

• choose any region (or several);

• select a convenient scanning schedule;

• obtain data via the API.
  
  

Elements in search results that can be received via the SERP Position tool:

• website position (according to the region and schedule you've set);

• domain;

• target URL;

• title;

• description;

• the number of results;

• paid search results on the phrase.
  

A report looks like this:

The process is carried out in the following way:

You can store data obtained by the parsing in a database (My SQL or Big Query, etc.) and make reports, graphs via Data Studio (optionally).

Let's say that you want to assess the site's niche in terms of SEO regularly. You'll want to keep an eye on what's happening at the top and adjust your promotion plan accordingly. Let's explore how on-page analysis tools, crawling, and Google Niche tools can assist you.

Serpstat API returns JSON objects during crawling, which can be simply transformed to a table display. The parsing results, for example, can be uploaded to BigQuery and subsequently evaluated there:

We lack CTR and keyword volume for a complete picture. You can calculate the traffic for each website from the top for each day if you combine this information. If you don't require such specifics, you can just get the total traffic.

Let's have a look at the example. We'd want to research the SEO niche in Google USA, and we've already received 64 requests in this regard. Every day, we analyzed tops, got volume from the Serpstat API, and, based on the position, found CTR statistics in Google search results. We won't go into the technical implementation of this solution because you can use whichever way you choose to process and store JSON objects. I used the R + Google Sheets + Google BigQuery combination, although Apps Script can be used instead of R.

We'll add three columns to the final table with data — CTR, Queries (frequency) and Traffic. Then it's time to go to Data Studio and build interactive reports based on this table.

In Data Studio, you connect a data source, and in 15 minutes you collect a report on traffic and positions:

The average position by days is shown on the left graph, the average position by domain is shown on the right graph, and numerous useful numbers (average, lowest, maximum position) are shown separately in the upper section of the report. On the left graph, the sum of traffic each day, and on the right graph, the total amount of traffic on the domain are shown at the bottom.

We'll not pay attention to some obvious or non-obvious numbers, but let's see how we can work with such a report. Also, we will not distribute positions and traffic to different sheets of the report — this is how all of our data will be placed in one screen. I created the CTR for this report myself, any reference is coincidental :)

We select the domain we require (let's say udemy.com — we'll deal with the SEO courses section) and examine how the average position and traffic vary over time. We may see variations in traffic in the example below with a relatively little change in the average position. The reason for the drop in traffic may be a traffic drawdown.

Let's limit our report to data to top 10. Here we see the same — the average position has risen, and the traffic has fallen. This can be explained by the fact that the top 10 dropped out keywords that brought a fairly large amount of traffic.

We also see that most of the traffic for udemy.com is brought by keywords from the top 10 and any changes in this range in the positions will greatly influence the traffic.

In the phrase selection drop-down list, we may see all the phrases in a specific top. You can achieve this by creating a second table called Phrase — Date — Position. You may also start saving changes in positions across the domain in the data source for the report by creating tables with lists of phrases by domain for different dates to compare changes in the top.

Let's take the keyword SEO trainings that brings traffic for Udemy and see what happens there:

The Positions report was not informative, but looking at the Traffic we can see that Udemy is the winner of all and the main competitor is Search Engine Watch. Let's compare these two domains.

Here, the situation is strange: with a decrease in the average position, the traffic of Udemy has risen, while in Search Engine Watch these indicators behave the same way. This can be explained by the principle of report building. Udemy began to appear in other positions as well, and the average position fell. In this case, I count traffic simply as the sum of traffic for all positions, that's why it has grown.

Let's compare Udemy and Search Engine Watch for all requests. We see that with a close average position, the traffic of the sites is very different.

For example, you can look at the traffic for each day right on the graph:

For each of the sites, you can see the proportion of traffic in the niche. To do this, you can create a separate report if you need to check this index regularly. We'll do a simple calculation.

We take the data for November 23. The total traffic is 237.3 thousand. The traffic on Udemy is 2.4 thousand, this is about 1%. At Search Engine Watch traffic is 1.7 thousand, this is 0.7%

In our case, we only received data for one region. You can compare the domain in different regions or track changes in request results based on the region if you parse the tops in multiple locations.

We also left out special element analysis and contextual advertising. With this information, you may make CTR corrections based on their availability, allowing you to generate more accurate traffic estimations.

Another possibility is parsing snippets. You can analyze snippets of sites in the top 10 and take additional information from there for optimizing pages. For example, all your competitors offer free shipping directly in the snippet, but you don't, and because of that you lose competitive advantage.

You can add all this information to a report in Data Studio, and it'll turn into a comprehensive top analytics system. Such a thing is always convenient to keep on hand. Start by collecting data and further improve it as needed.

Here we collected the key differences between our popular tools: Сrawling, Rank Tracker, and Top by a keyword API: