Google is the most used search engine (92% market share). We wanted to know how Google search results compared to other search engines, namely Bing, DuckDuckGo, and Yahoo).

We analysed about 56k random keyword searches for 4 search engines including Google, and came up with some insights based on the 30 top ranked results.

1 Summary of insights

  1. The results of Google’s competitors are more similar between themselves than similar to Google’s. DuckDuckGo is the most similar to Google (31%) and Yahoo is the least similar (29%).

  2. For all search engines, the similarity doesn’t change much with search volume, though search engine results are consistently more similar to Google’s for medium volume (between 10k and 100k) volume, with gains between 1.5% and 2% when comparing volume of more and less than 10k.

  3. Similarity differs by category. Real Estate and Travel & Tourism are more similar across all search engines, while Apparel related results are less similar.

  4. Similarity is higher for long searches.

  5. The different search engines feature in different amounts big domains such as amazon in their top result.

  6. Searches that tend to yield results predominantly linking to a specific domain (domain specific), OR non domain specific at all, show lower similarity to Google on average.

  7. In more than half of the cases in any search engine, Google’s top result is found in position 1 to 3. For 25% to 35% of keywords for a given search engine, Google’s top result won’t be found in the top 30.

library(tidyverse) # package used for data wrangling
library(ggtext) # for text formatting
library(ggforce) # for donut charts
library(patchwork) # for combining plots together
# options(dplyr.summarise.inform = F)
# "datasets_for_report.Rdata" is built by the following commented calls
# source(here::here("scripts/01_build_light_dataset.R"))
# source(here::here("scripts/02_build_datasets_for_report.R"))
# source(here::here("scripts/03_similarity_metrics.R"))
load(here::here("proc_data/datasets_for_report.Rdata"))

# Set theme for plots
theme_set(theme_minimal(base_size = 12, base_family = "Poppins"))
theme_update(
  plot.title.position = "plot",
    plot.title = element_text(face = "bold", margin = margin(b = 10)),
    plot.margin = margin(10, 20, 10, 20),
    plot.background = element_rect(fill = "#D3D2F9", color = NA),
    legend.position = "none",
    axis.title.x.bottom = element_text(color = "grey60", size = rel(0.7), hjust = 1, margin = margin(t = 5), face = "bold"),
    axis.title.y.left = element_text(color = "grey60", size = rel(0.7), hjust = 1, margin = margin(r = 5), face = "bold"),
    axis.text = element_text(size = rel(0.75)),
    strip.background = element_rect(fill = "#807FFF", color = "white", size = 1.5),
    strip.text = element_text(color = "white", face = "bold"),
    panel.grid.minor = element_blank(),
    panel.spacing.x = unit(2.5, "lines"),
    panel.spacing.y = unit(2.5, "lines")
)

2 Methodology

  • We started by taking a random subset of 60k keywords from DataforSEO´s 354M large US keyword database. The subset contained an equal set of keywords with monthly search volumes of 500-1000 , 1000-10000 and 10000-100000, respectively.
  • We used DataForSEO´s SERP API to populate the top 30 search results for each keyword across 4 search engines.
  • We start from a dataset containing 7,606,414 observations, where one observation is a search result for a given search engine
  • After we drop a handful of ill formatted keywords, filter to keep only organic results and remove observations featuring a missing keyword we are left with 7,154,169 observations
  • After we join our data to our database of keywords we are left with 7,011,038 observations
  • After removing keywords that are not present for all 4 search engines we are left with 7,010,575 observations
  • After removing keywords for which we find a missing URL we are left with 6,976,526 observations
  • After removing keywords for which we find a duplicate set of keyword_id, search_engine, rank_group we are left with 6,520,886 observations
  • After removing observations featuring a rank over 30 we are left with 6520838 observations
  • Overall we are left with 55,979 keywords

Due to technicalities fetching data with the API, 3% of our search results among the top 30 results of our search engines could not be identified, i.e. we miss observations.

We believe this shouldn’t impact sensibly our learnings, and since removing all keywords showing an anomaly would drastically affect the sample size we build artificial observations for these cases so we end up with 6,717,480 observations, amounting to 55,979 keywords times 4 search_engines times 30 results.

3 Analysis

3.1 How similar are the rankings between search engines?

We define similarity between two search engines as the fraction of results (identified by url) that we find in both top 10 results of the pair.

We see below that on average 70% of top 10 results on yahoo can be also found in the top 10 of Bing, making them the most similar pair.

Google is very dissimilar to other search engines, especially Yahoo, its shares only 29% of top 10 results wit the latter. It is most similar to DuckDuckGo, with a similarity indicator of 31%.

se_list <- c("Google", "Yahoo", "Bing", "DuckDuckGo") 

similarity_by_se_10 <- 
  similarity_by_se_10 %>% 
  mutate(
    search_engine = if_else(search_engine == "DuckDuck", "DuckDuckGo", search_engine),
    search_engine2 = if_else(search_engine2 == "DuckDuck", "DuckDuckGo", search_engine2)
  )

df <- 
  crossing(se_1 = se_list, se_2 = se_list) %>% 
  left_join(select(similarity_by_se_10, -descr), 
            by = c("se_1" = "search_engine", "se_2" = "search_engine2")) %>% 
  left_join(select(similarity_by_se_10, -descr), 
            by = c("se_1" = "search_engine2", "se_2" = "search_engine")) %>% 
  mutate(
    similarity = coalesce(similarity.x, similarity.y),
    se_1 = factor(se_1, levels = c("Yahoo", "Bing", "DuckDuckGo", "Google")),
    se_2 = factor(se_2, levels = c("Yahoo", "Bing", "DuckDuckGo", "Google")),
    include_exclude = case_when(
      se_1 == se_2 ~ "exclude",
      se_1 == "DuckDuckGo" & se_2 == "Bing" ~ "exclude",
      TRUE ~ "include"
    ),
    label = ifelse(is.na(similarity), NA, glue::glue("{round(similarity * 100, 0)}%"))
  ) %>% 
  select(se_1, se_2, similarity, include_exclude, label)

p <- 
  ggplot(df, aes(x = se_1, y = se_2, fill = similarity)) +
  geom_tile(color = "#D3D2F9", size = 2.5) +
  geom_tile(data = filter(df, include_exclude == "exclude"), fill = "#D3D2F9") +
  geom_text(data = filter(df, include_exclude == "include"), 
            aes(label = label), size = 7, family = "Poppins", fontface = "bold", color = "white") +
  scale_x_discrete(limits = c("Yahoo", "Bing", "DuckDuckGo")) +
  scale_y_discrete(limits = c("Google", "DuckDuckGo", "Bing")) +
  scale_fill_gradient(low = "#C2C2FF", high = "#7426da", limits = c(0.2, 0.8), breaks = c(0.2, 0.5, 0.8), labels = scales::percent_format(accuracy = 1), name = "Similarity") +
  guides(fill = guide_colorsteps(title.position = "top", title.hjust = 0.5)) +
  labs(
    title = "Yahoo, Bing and DuckDuckGo give very different results",
    x = "",
    y = ""
  ) +
  theme(
    panel.grid = element_blank(),
    plot.title = element_text(size = rel(1.6)),
    axis.text = element_text(size = rel(1.2), face = "bold"),
    legend.position = c(0.85, 0.8),
    legend.direction = "horizontal",
    legend.key.width = unit(10, "mm"),
    legend.key.height = unit(2, "mm")
  )

ragg::agg_png(here::here("plots", "plot_01_plot_ranking_similarity.png"), width = 8, height = 8, units = "in", res = 320)
print(p)
dev.off()