Major update: More precise Mangools Keyword Difficulty

Category: KWFinder

Keyword difficulty has been an important metric and a vital part of KWFinder since the beginning. Now, we have updated its algorithm to make it even more precise.

Welcome to the age of Mangools Keyword Difficulty 2.0!

Why would we do this?

We are really proud that KWFinder’s keyword difficulty (KD) has been considered (one of) the most precise difficulty metrics on the market, because of the highest correlation with real rankings in Google SERP.

Many reviews and analyses went deep into the numbers and the algo we’ve developed proved to be one of the most accurate ones when compared to Ahrefs, Moz, SEMrush, LongTailPro and others.

Check the reviews by Moz, Authority Hacker or Authority Website Income, just to name a few.

Thanks to this, there was no reason to change or update the metric at all for a long period of time.

The first thought about a possible update of the KD algorithm in KWFinder came to our minds when we noticed that Moz has updated their metrics.

The original keyword difficulty

The original calculation was based on the Moz metrics:

  • Domain Authority (DA)
  • Page Authority (PA)
  • MozRank (MR)
  • MozTrust (MT)

This is a simplified version of the algorithm:

First, we calculated the overall Link Profile Strength (LPS) of each URL on the 1st SERP based on the metrics mentioned above. Then, we made an average of them. The final number represented the keyword difficulty.

Moz metrics changes

In May, Moz announced that they have improved the calculation of their metrics. What has changed? Well, a couple of things:

  1. The backlink index got bigger and now it’s updated more regularly (in the past, it could take more than 40 days to index a new backlink)
  2. Domain Authority and Page Authority metrics are updated more often
  3. Moz got rid of MozTrust and MozRank metrics

What happened with KD?

One would think that more advanced input metrics mean more advanced KD. Or at least, that’s what we thought at first. (Un)fortunately, there were more and more questions abouth the “keyword difficulty jumps” so we decided to analyze the changes in detail.

The topic was widely discussed on forums too. This is one of the comments:

blackhat world feedback

Here, we would like to express our deepest gratitude to all our users for their feedback. It was them who pointed out that the changes were deeper than we thought and made us look at what happened. Thanks for that!

What we found out

The main problem was a significant jump in PA values for URLs that had a very low PA before. The biggest increase concerned the PA values between 20 and 40, often jumping to 50 or 60.

As a result, their LPS increased and so did the KD metric of the keywords concerned. The “easy” keywords in the so-called green zone suddenly reached the “possible” or “hard” levels.

Here’s a quick comparison of how the difficulties changed in a randomly picked niche:

Keyword KD before Moz update
KD after Moz update
cheap foosball table 28 52
sportcraft foosball 25 52
foosball coffee table 27 46
carrom foosball table 25 41
foosball players 25 41
warrior foosball table 24 41
foosball table corner ramps 19 39
halex foosball table 27 39
foosball table cover 26 39
carrom burr oak foosball table 22 37
highland games foosball table 20 36
coin operated foosball table 28 34
foosball table parts 26 31
good foosball table brands 28 25
top foosball table brands 27 25
Average KD 25 38

I could continue with many other “low-difficulty” niches, the numbers would be more or less the same.

Link Profile Strength vs. Keyword Difficulty

Now, there is one super important question:

Does increased LPS of some websites on the 1st SERP automatically mean that it’s more difficult to get on the 1st SERP?

No! And that means the KD as a metric that estimates how difficult it is to start ranking on the 1st SERP, should not be influenced by the changes of some of the websites so easily.

The magic of Google algorithm is that thanks to the relevance, websites with very low authority are able to outrank authoritative websites. In other words, the big players won’t always be in front of the underdogs because they may not be that relevant.

Doing an average of LPS values on the 1st SERP, especially with the increased PAs, is no longer a clear indicator of how hard it is to reach the 1st SERP.

Moreover, Moz got rid of MozRank and MozTrust, so having an algo based only on two metrics wouldn’t make any sense. That’s why we decided to add the Citation Flow (CF) and Trust Flow (TF) by Majestic to the calculation. And we got rid of MR and MT.

A lot of tests and analyses were made to mix the inputs so they would correlate with organic search results as the KD used to before.

Keyword Difficulty 2.0

Thanks to the Moz update, the LPS itself became more precise and up-to-date. However, it didn’t reflect the overall keyword difficulty so well.

That’s why the KD is no longer an average of LPS values on the 1st SERP. The LPS is far more sensitive to changes in metrics and their weights. From now on, you will probably see higher LPS scores when compared to the overall KD.

Keyword difficulty reflects how hard it is to get to the 1st SERP, not how hard it is to rank first.

Although the old KD formula did the same, now, we have decided to highlight this even more by giving more weight to the websites with lower LPS when calculating the final keyword difficulty.

That being said, it’s totally natural when a website with LPS 20 outranks one with LPS 60 and that’s exactly what the updated keyword difficulty focuses on.

New ranges

Based on the changes, we have also updated the ranges. It reflects exactly what the updated algo is about. Originally, there were 8 ranges.

Now there are 6 ranges with less colors so it is easier to distinguish between the values.

But most importantly, since the KD estimates how hard it is to get to the 1st SERP, the “green zone” shifted to lower values from 0 to 29 (compared to 0 to 39 before).

New keyword difficulty range

Keyword Difficulty caching

We deployed the new KD algo on 25th September 2018. Due to this change, all previously calculated KD values are now outdated.

To indicate it, we have decreased the opacity of the values and added “N/A” in the tooltip.

KWFinder keyword difficulty old cache

From now on, all KD values that were calculated more than 30 days ago, will have decreased opacity as well.

What’s more, you can see the exact date when the KD was last updated in a tooltip. Just hover over the KD value.

last updated keyword difficulty in KWFinder

To recap:

  1. Due to the Moz update, the old KD calculation could indicate harder difficulty when compared to the reality
  2. We updated the metrics used in the KD formula – it is now based on Moz (DA, PA) and Majestic (CF, TF) metrics
  3. It is important to remember that the main purpose of the KD (both old and new) is to estimate how difficult it is to get to the 1st SERP, not how difficult it is to rank first
  4. To highlight this even better, KD is no longer the average of LPS values on the 1st SERP, but it gives more weight to the URLs with lower LPS
  5. All the cached KD values that were calculated more than 30 days ago have lower opacity and the exact date when it was last updated in a tooltip

This is a comparison of the old algorithm before the Moz update, the old algorithm after the Moz update and the brand new keyword difficulty:

Keyword KD before Moz update
KD after Moz update
New KD
cheap foosball table 28 52 34
sportcraft foosball 25 52 34
foosball coffee table 27 46 28
carrom foosball table 25 41 23
foosball players 25 41 23
warrior foosball table 24 41 27
foosball table corner ramps 19 39 20
halex foosball table 27 39 16
foosball table cover 26 39 28
carrom burr oak foosball table 22 37 18
highland games foosball table 20 36 19
coin operated foosball table 28 34 21
foosball table parts 26 31 23
good foosball table brands 28 25 19
top foosball table brands 27 25 19
Average KD 25 38 23

We are looking forward to your feedback

We will keep checking the results and listening to your feedback. Don’t hesitate to contact us if you have any concern or input that would help other users to understand the keyword difficulty.

Loading Disqus...