AHREFS Crawling issue - resolved their end but still not working

Question

I checked with AHREFS and they have said Cloudflare has made the changes required to allow their IP to crawl our site. However we received the following:

Hello Daniel! We have added new IPs so it looks the audits fail because we can't access your site from those IPs. You need to reach out to your hosting provider and ask them to whitelist this IP range 202.8.4x.x.xx
You can find individual IPs for this range here: https://api.ahrefs.com/v3/public/crawler-ips/

We are worried that this may be stopping Google search console crawling our site as well.

Oleg Isonen · Answer

First of all ahrefs is not google, different bots.

Oleg Isonen · Answer

Second if they got updated their IPs then cloudflare wouldn't be blocking them, we aren't doing anything manually ourselves.

Oleg Isonen · Answer

They are basically telling you to reach out to cloudflare

Oleg Isonen · Answer

I personally can't verify the IPs cloudflare has submitted for their bot on cloudflare https://radar.cloudflare.com/traffic/verified-bots

Oleg Isonen · Answer

but also we never had that issue in the first place, we are also using ahrefs

Oleg Isonen · Answer

Are you using webstudio cloud?

DanielCMGCAI · Answer

Yes I understand they are different bots, just saying if it effects AHREFs it could also effect other crawlers. I notice I am not the only one with this issue.

https://discord.com/channels/955905230107738152/1321099396951310409

Have you tried a side audit recently? See if you get the same issue

DanielCMGCAI · Answer

And no we are hosting directly with WebStudio

Oleg Isonen · Answer

We use it all the time

Oleg Isonen · Answer

We need to get a clear answer from them if the problem is with the ip range of verified bots or something else, because how can we not have that issue then.

@John Siciliano did you see this issue on any other site that is not webstudio.is?

Oleg Isonen · Answer

@DanielCMGCAI just to be sure, can you pls screenshot for us what you are seeing on ahrefs?

Oleg Isonen · Answer

could also effect other crawlers. Not the google bot. Noone ever blocks this one 🙂

John Siciliano · Answer

No but Idk of any other platforms that use the Cloudflare Verified Bots list.. Shopify uses Cloudflare Idk if they use the list (I like using the list for the record)

Oleg Isonen · Answer

No I mean any other site on webstudio that is not webstudio.is website itself that also has this problem

Oleg Isonen · Answer

It can't be that only a few specific sites are affected, it must be either all of them or none

Oleg Isonen · Answer

Unless it's a user setting in the cloudflare admin panel that admin in their cloudflare account can change.

John Siciliano · Answer

I can confirm another site is failing too. I noticed they just up the attached about Hostinger

Oleg Isonen · Answer

I think I have an idea: maybe ahrefs is using different ips fr different customers, so it was affecting only some customers which were crawled from some new ips

Oleg Isonen · Answer

that's why we have only a few reports and we never had a problem ourselves

John Siciliano · Answer

Yes possibly but every site I have in Ahrefs is failing. I'm in the US though

Oleg Isonen · Answer

Ah so there are other cases, that's what I was asking about. And these sites worked before, right?

John Siciliano · Answer

I've only been talking about Ahrefs Site Audits. Yes these sites worked before

Oleg Isonen · Answer

Then I am confident this is nothing we did or need to do. It's 100% between them and cloudflare.

Oleg Isonen · Answer

Maybe they bought a new IPs range and migrating something and cloudflare didn't get the range yet.

Join the Webstudio community

AHREFS Crawling issue - resolved their end but still not working