CTR study
Last updated on 2021-07-07
1 Introduction
Does organic CTR impact SEO rankings? In this study, we want to answer this question and test for the effect of increasing Click-Through Rates (CTRs) on Google Search rankings.
We focus on a particular niche: car accident lawyers (personal injury lawyers). We “artificially” increased the number of visits to randomly selected car accident lawyers’ websites and measured the impact on Google rankings.
By “artificially” increasing the number of searches and visits per day, we improve the CTR, which is considered a ranking factor. As background, see Larry Kim’s post for a captivating discussion on the issue.
2 Methodology
We launched campaigns for 60 selected personal injury lawyers’ websites with an organic traffic provider. Each campaign is “artificially” boosting the number of searches and clicks for the keyword “[city] car accident lawyer”. By doing so, we increase their CTR for 30 days.
We collect the position (Google Search ranking) of the website daily from the traffic provider’s interface. We then analyzed the position of each website throughout the study to assess the strength of the effect and effectiveness of using a traffic provider.
One of the major caveats of this study is that we used the traffic provider both to boost the number of visits and to measure the effect of this boost. This means that we were not able to have a control group.
Finally, it is worth noting that while the ranking of any website is expected to vary over time, we anticipate to see a possible impact when looking at many websites. This is the reason we collected 60 samples from a variety of cities and initial rankings.
See also the Detailed Methodology in the Annex.
3 Research Findings
3.1 Ranking Over Time
We started by visualizing the rank evolution of the 60 websites. Each line represents the ranking of a personal injury lawyer’s website for the keyword “car accident lawyer” over time. The pink line is a LOESS regression (locally estimated scatterplot smoothing) and represents a smoothed average, similar to a moving average but better.
We observed the following:
- There is variability in the ranking of the websites, which will blur the picture.
- Some websites did see an increase in their ranking over the month. Others saw a decrease.
- The smoothed average is increasing slightly, but the effect of boosting the visits for one month for one keyword is not extreme. It seems to be around +2 positions.
- Something seems to have happened at around week 2 in the study (approximately March 19th). There is a small jump in the ranking of many websites, forming a visible pattern. Also, it seems that the variance of the ranking decreases. The picture seems to stabilize. One potential explanation could be an update in Google’s algorithm. However, the Rank Risk Index of RankRanger did not detect an update on March 18th or 19th (but an unconfirmed update on March 10th). It did, on the other hand, observe increased Google SERP fluctuations on March 13th and 14th. Another alternative explanation could be that two weeks is the time needed for the Google algorithm to capture and adjust to the boost in visits.
3.2 Day-to-day Fluctuations
As we previously showed, there is a high variability in the rankings, even at the daily level. We can use an indexed line plot withall law firms finishing at 0 on April 5th to show the fluctuations over time. On this chart, a sample starting on March 5th at an indexed rank of, let’s say 5, gained five ranks during the study.
We observed again that:
- There are large fluctuations. Some websites saw a change of 10 ranks or more over the period. Several samples saw a difference in the ranking larger than ±5 in a single day.
- The rankings seem to fluctuate less after March 19th. The variance of the ranking for March 19th is around ±5. After that it seems to be ±3.
3.3 Comparing First Week to Last Week
We saw previously that there is variability in the ranking. Thus, we will compute the rank improvement by comparing the average rank in the first week of the study to the average rank in the last week of the study in order to obtain more stable estimates.
This slopegraph is very useful to highlight the difference in ranking between the first week and the last week for all the sample websites. Again, we observe that the law firms’ websites have different fates. But globally, there is nevertheless an increase of 2 ranks in the Google Search ranking.
Because we collected several samples (60), we can test if this difference is likely due to chance (explained by intrinsic variability in the ranking of websites) or is likely a “true” effect that would show again if we were to repeat the experiment. A paired t-test comparing the average ranking of each law firm during the first week and last week shows that the difference is statistically significant. Thus, we reject the null hypothesis that the difference in mean between the first week and last week is zero in favor of the alternative hypothesis: the true difference is not equal to 0. In other words, we indeed observed a non-zero, positive increase during the study.
| mean of the differences | 2.004 |
| degrees of freedom | 1887 |
| t statistic | 27.3181 |
| p value | 9.755209e-139 |
| confidence interval | [1.86, 2.15] |
Is this effect stronger for websites that started on the last page?
The positive effect is, maybe without surprise, more substantial for websites that started on the third page (+3 rank on average, only +1 for those that started already on the first page).
4 Conclusion
The Google Search ranking of the sample personal injury lawyers’ websites can vary a few ranks daytoday. Not all the websites experienced a rank increase after a month of boosting their daily number of visits after a Google Search for the keyword “car accident lawyer”.
Nevertheless, we observe a significant positive effect of +2 ranks, on average. Moreover, this effect is larger for websites ranking initially lower (e.g., those that appear on the third page of the search results).
There is an interesting pattern at week 2 after the start of the study. This is when the ranking increases and stabilizes for most of the websites. Two weeks could be the time needed for the Google algorithm to capture and adjust to the boost in visits. Alternatively, it could be the date where Google changed its algorithm. However, the Rank Risk Index of RankRanger did not detect an update on March 18th or 19th.
4.1 Discussion of the Limitation of the Study
This study has a few limitations.
First, we increased the number of visits for only one keyword of interest (“car accident lawyer”). However, it is arguably the most important keyword, and we multiply the estimated number of daily visits by 30 (minus a bounce rate of 50%).
Second, we could not build a control group where we would not boost the number of visits, as the traffic provider does not allow tracking the ranking of websites without visiting them. However, ranking is a zero-sum game. So if our observed samples ranked +2 positions higher (on average), it means that other websites must have ranked lower.
Third, we use a traffic provider to both boost and measure the impact. We thus rely on its own Google Search ranking to assess the tool. We see, however, no reason to not trust the traffic provider Google Search ranking measurements.
Finally, we acknowledge that the results of this study are specific to the samples and parameters chosen: personal injury lawyers’ websites, “car accident lawyer” as the keyword, a calibrated boost of 30 times the estimated number of daily visits, and the traffic provider parameters and bots, etc., are thus not easily generalizable.
5 Annex
5.1 Detailed Methodology
5.1.1 Website Selection
We selected 3 personal injury lawyers in 20 U.S. cities. Then, we randomly picked one website appearing on the first page, second page and third page of the results of a Google search with the keywords “[city] car accident lawyer”. We use Ahrefs Keyword Explorer for this selection. We only select websites that appeared on the first to the third page of the Google Search results (according to Ahrefs), because SERP Empire, our traffic provider, only collects the results of the first 3 pages. In the end, we obtained a sample of 60 personal injury lawyer websites in various cities and with different Google Search positions.
5.1.2 Boosting Calibration
We wanted to give each website a comparable boost in visits that looked organic. If we delivered the same boost to the personal injury website ranking 2nd in a large city where there is a lot of competition, let’s say LA, to the website ranking 28th in Fresno, we would run into two problems. First, the comparative size of the boost would be extremely different: insignificant for the top personal injury lawyer’s website in LA, but oversized for the lower ranking website in Fresno. Second, the boost in Fresno would not look “organic” and would likely be detected by Google as not being real organic traffic on the website and be discarded.
Therefore, we calibrate the boost in visits to an estimated number of daily visits for each website. Here is how we do it:
We multiply the average number of daily searches in a city for the keywords “[city] car accident lawyer”, computed from the monthly volume on Google Ads, to the expected CTR of the website, computed from is Google search ranking (provided by Ahref) to a ranking CTR factor (provided by the Google Organic CTR History of Advanced Web Ranking). This ranking CTR factor estimates the CTR in the relation to the position in the Google Search results. Indeed, a website ranking first is much more likely to be visited than a website ranking lower. By doing so, we get the number of estimated daily visits.
Then, we multiply the estimated daily visits to a common multiplier, the “boost”, to obtain the calibrated number of daily keyword searches that we will add to each website with SERP Empire.
In short, for each website:
- \(\text{Ranking CTR factor} * \text{Daily Searches for keyword in city} = \text{Estimated Daily Visits}\)
- \(\text{Estimated Daily Visits} * Multiplier = \text{Daily Keyword Searches}\)
Where:
- \(\text{Ranking CTR factor} = \text{current ranking} * \text{CTR estimate from Google Organic CTR History study}\)
- \({Daily Searches for keyword in city}\) is computed from Google Ads monthly volume for the keyword “[city] car accident lawyer”
- \(\text{CTR estimate from Google Organic CTR History study}\) is obtained from the Organic + Local Pack (source) and is:
| Google Search position | CTR |
|---|---|
| 1 | 13.13% |
| 2 | 8.88% |
| 3 | 5.21% |
| 4 | 4.12% |
| 5 | 1.83% |
| 6 | 1.53% |
| 7 | 1.29% |
| 8 | 1.05% |
| 9 | 1.01% |
| 10 | 1.08% |
| 11 | 1.10% |
| 12 | 1.02% |
| 13 | 0.87% |
| 14 | 0.76% |
| 15 | 0.62% |
| 16 | 0.56% |
| 17 | 0.60% |
| 18 | 0.48% |
| 19 | 0.44% |
| 20 - 30 | 0.25% |
- The \(Multiplier\) is how much we multiply the \(\text{Estimated Daily Visits}\): 30
This is the study plan:
| Sample | City | Keyword | Baseline City Daily Searches for the Keyword | Google Search ranking (from Ahref) | Ranking CTR Factor | Estimated Daily Visits | Multiplier | Number of Daily Searches on SERP Empire |
|---|---|---|---|---|---|---|---|---|
| 1 | San Antonio | San Antonio car accident lawyer | 24 | 8 | 0.0105 | 0.25 | 30 | 8 |
| 2 | San Antonio | San Antonio car accident lawyer | 24 | 13 | 0.0087 | 0.21 | 30 | 6 |
| 3 | San Antonio | San Antonio car accident lawyer | 24 | 27 | 0.0025 | 0.06 | 30 | 2 |
| 4 | Los Angeles | Los Angeles car accident lawyer | 147 | 4 | 0.0412 | 6.04 | 30 | 181 |
| 5 | Los Angeles | Los Angeles car accident lawyer | 147 | 13 | 0.0087 | 1.28 | 30 | 38 |
| 6 | Los Angeles | Los Angeles car accident lawyer | 147 | 23 | 0.0025 | 0.37 | 30 | 11 |
| 7 | Brooklyn | Brooklyn car accident lawyer | 33 | 5 | 0.0183 | 0.61 | 30 | 18 |
| 8 | Brooklyn | Brooklyn car accident lawyer | 33 | 13 | 0.0087 | 0.29 | 30 | 9 |
| 9 | Brooklyn | Brooklyn car accident lawyer | 33 | 26 | 0.0025 | 0.08 | 30 | 3 |
| 10 | Miami | Miami car accident lawyer | 63 | 9 | 0.0101 | 0.64 | 30 | 19 |
| 11 | Miami | Miami car accident lawyer | 63 | 12 | 0.0102 | 0.65 | 30 | 19 |
| 12 | Miami | Miami car accident lawyer | 63 | 29 | 0.0025 | 0.16 | 30 | 5 |
| 13 | Houston | Houston car accident lawyer | 180 | 9 | 0.0101 | 1.82 | 30 | 55 |
| 14 | Houston | Houston car accident lawyer | 180 | 11 | 0.0110 | 1.98 | 30 | 59 |
| 15 | Houston | Houston car accident lawyer | 180 | 23 | 0.0025 | 0.45 | 30 | 14 |
| 16 | Chicago | Chicago car accident lawyer | 43 | 3 | 0.0521 | 2.26 | 30 | 68 |
| 17 | Chicago | Chicago car accident lawyer | 43 | 15 | 0.0062 | 0.27 | 30 | 8 |
| 18 | Chicago | Chicago car accident lawyer | 43 | 26 | 0.0025 | 0.11 | 30 | 3 |
| 19 | Dallas | Dallas car accident lawyer | 43 | 8 | 0.0105 | 0.46 | 30 | 14 |
| 20 | Dallas | Dallas car accident lawyer | 43 | 20 | 0.0025 | 0.11 | 30 | 3 |
| 21 | Dallas | Dallas car accident lawyer | 43 | 28 | 0.0025 | 0.11 | 30 | 3 |
| 22 | Minneapolis | Minneapolis car accident lawyer | 120 | 5 | 0.0183 | 2.20 | 30 | 66 |
| 23 | Minneapolis | Minneapolis car accident lawyer | 120 | 15 | 0.0062 | 0.74 | 30 | 22 |
| 24 | Minneapolis | Minneapolis car accident lawyer | 120 | 25 | 0.0025 | 0.30 | 30 | 9 |
| 25 | Las Vegas | Las Vegas car accident lawyer | 63 | 4 | 0.0412 | 2.61 | 30 | 78 |
| 26 | Las Vegas | Las Vegas car accident lawyer | 63 | 16 | 0.0056 | 0.35 | 30 | 11 |
| 27 | Las Vegas | Las Vegas car accident lawyer | 63 | 29 | 0.0025 | 0.16 | 30 | 5 |
| 28 | Orlando | Orlando car accident lawyer | 20 | 2 | 0.0888 | 1.75 | 30 | 52 |
| 29 | Orlando | Orlando car accident lawyer | 20 | 20 | 0.0025 | 0.05 | 30 | 1 |
| 30 | Orlando | Orlando car accident lawyer | 20 | 30 | 0.0025 | 0.05 | 30 | 1 |
| 31 | Tampa | Tampa car accident lawyer | 33 | 9 | 0.0101 | 0.34 | 30 | 10 |
| 32 | Tampa | Tampa car accident lawyer | 33 | 15 | 0.0062 | 0.21 | 30 | 6 |
| 33 | Tampa | Tampa car accident lawyer | 33 | 27 | 0.0025 | 0.08 | 30 | 3 |
| 34 | Kansas City | Kansas City car accident lawyer | 11 | 8 | 0.0105 | 0.11 | 30 | 3 |
| 35 | Kansas City | Kansas City car accident lawyer | 11 | 18 | 0.0048 | 0.05 | 30 | 2 |
| 36 | Kansas City | Kansas City car accident lawyer | 11 | 25 | 0.0025 | 0.03 | 30 | 1 |
| 37 | Austin | Austin car accident lawyer | 33 | 3 | 0.0521 | 1.74 | 30 | 52 |
| 38 | Austin | Austin car accident lawyer | 33 | 16 | 0.0056 | 0.19 | 30 | 6 |
| 39 | Austin | Austin car accident lawyer | 33 | 25 | 0.0025 | 0.08 | 30 | 3 |
| 40 | Nashville | Nashville car accident lawyer | 11 | 7 | 0.0129 | 0.14 | 30 | 4 |
| 41 | Nashville | Nashville car accident lawyer | 11 | 15 | 0.0062 | 0.07 | 30 | 2 |
| 42 | Nashville | Nashville car accident lawyer | 11 | 24 | 0.0025 | 0.03 | 30 | 1 |
| 43 | Baltimore | Baltimore car accident lawyer | 24 | 7 | 0.0129 | 0.31 | 30 | 9 |
| 44 | Baltimore | Baltimore car accident lawyer | 24 | 18 | 0.0048 | 0.12 | 30 | 3 |
| 45 | Baltimore | Baltimore car accident lawyer | 24 | 29 | 0.0025 | 0.06 | 30 | 2 |
| 46 | Queens | Queens car accident lawyer | 7 | 3 | 0.0521 | 0.36 | 30 | 11 |
| 47 | Queens | Queens car accident lawyer | 7 | 15 | 0.0062 | 0.04 | 30 | 1 |
| 48 | Queens | Queens car accident lawyer | 7 | 24 | 0.0025 | 0.02 | 30 | 1 |
| 49 | Fresno | Fresno car accident lawyer | 16 | 6 | 0.0153 | 0.24 | 30 | 7 |
| 50 | Fresno | Fresno car accident lawyer | 16 | 14 | 0.0076 | 0.12 | 30 | 4 |
| 51 | Fresno | Fresno car accident lawyer | 16 | 25 | 0.0025 | 0.04 | 30 | 1 |
| 52 | Charleston | Charleston car accident lawyer | 5 | 4 | 0.0412 | 0.19 | 30 | 6 |
| 53 | Charleston | Charleston car accident lawyer | 5 | 13 | 0.0087 | 0.04 | 30 | 1 |
| 54 | Charleston | Charleston car accident lawyer | 5 | 22 | 0.0025 | 0.01 | 30 | 1 |
| 55 | Denver | Denver car accident lawyer | 33 | 9 | 0.0101 | 0.34 | 30 | 10 |
| 56 | Denver | Denver car accident lawyer | 33 | 12 | 0.0102 | 0.34 | 30 | 10 |
| 57 | Denver | Denver car accident lawyer | 33 | 23 | 0.0025 | 0.08 | 30 | 3 |
| 58 | Boston | Boston car accident lawyer | 16 | 10 | 0.0108 | 0.17 | 30 | 5 |
| 59 | Boston | Boston car accident lawyer | 16 | 15 | 0.0062 | 0.10 | 30 | 3 |
| 60 | Boston | Boston car accident lawyer | 16 | 29 | 0.0025 | 0.04 | 30 | 1 |
How to read the study plan, for instance, for the first personal injury lawyer in San Antonio: We multiply the baseline number of daily searches in San Antonio for the keyword to the CTR, which depend on the ranking, to obtain the estimated number of daily clicks for the sample 1. Then, we multiply this number by our multiplier to obtain the number of visits that we will pay SERP Empire to do on this website daily.
5.1.3 Traffic Provider Parameters
We use the traffic provider SERP Empire to visit our 60 sampled websites the number of days indicated in the column Number of Daily Searches on SERP Empire on the study plan above. SERP Empire bots will vary their exact number of visits around this average to make the traffic look more organic.
In addition, we used the following SERP Empire parameters:
- The country and region are specified
- 50% of mobile searches
- Two pages per session
- Average session duration: 60 seconds
- Bounce Rate: 50%
We boost the daily visits of the websites for one month.
5.1.4 Data Collection and Quality
From the SERP Empire dashboard, we manually collect the rank of each of the 60 websites every day.
A few comments on the data quality:
- Sample number 6 is removed from the analysis as the website could no longer be found by SERP Empire for the majority of the study (from 2021-03-14).
- There are a few websites that could not easily be found by SERP Empire and needed to be modified. Thus, there are a few missing days at the beginning of the month for some samples.
- The initial ranking obtained from Ahrefs determined the CTR ranking factor is sometimes a bit different from the ranking on SERP Empire, which is to be expected. Nevertheless, it is quite similar (it explains, for instance, why some websites ranked above position 30).
- We boost the visits for one keyword only: “[city] car accident lawyer”.