How to stop moz dotbot from accessing your website


DotBot is Moz's web crawler, it gathers web data for the Moz Link Index

Dotbot obeys robots.txt rules before accessing your host machine so the easiest way to stop dotbot is by adding robots.txt rules that limit dotbot activities

To forbid a directory let's say "login" add;

User-agent: dotbot
Disallow: /login/

Upon reading and parsing directives above moz dotbot won't dare access you sites login sub directory in it's craw routine

To forbid an entire website's access include directives below;

User-agent: dotbot
Disallow: /

Alternatively you can limit crawl rate by adding directives below "time is probably in seconds"

User-agent: dotbot
Crawl-delay: 10

I've attached an nginx log it's a trail left by dotbot along with it's ip and moz support e-mail address;

216.244.66.194 - - [19/Mar/2020:15:16:29 +0000] "GET /index.html HTTP/1.1" 200 13433 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"

Comments

Popular posts from this blog

How to get user's ip address in JavaScript

How to make a static http server in nodejs using express

Array concat method in JavaScript

What is 'this.' keyword in JavaScript

How to create promises in JavaScript