How to stop moz dotbot from accessing your website

DotBot is Moz's web crawler, it gathers web data for the Moz Link Index

Dotbot obeys robots.txt rules before accessing your host machine so the easiest way to stop dotbot is by adding robots.txt rules that limit dotbot activities

To forbid a directory let's say "login" add;

User-agent: dotbot
Disallow: /login/

Upon reading and parsing directives above moz dotbot won't dare access you sites login sub directory in it's craw routine

To forbid an entire website's access include directives below;

User-agent: dotbot
Disallow: /

Alternatively you can limit crawl rate by adding directives below "time is probably in seconds"

User-agent: dotbot
Crawl-delay: 10

I've attached an nginx log it's a trail left by dotbot along with it's ip and moz support e-mail address; - - [19/Mar/2020:15:16:29 +0000] "GET /index.html HTTP/1.1" 200 13433 "-" "Mozilla/5.0 (compatible; DotBot/1.1;, [email protected])"


Popular posts from this blog

How to set content disposition header for nginx server to force content download

Animated daylight on HTML5 Canvas

How to make a static http server in nodejs using express

How to make a Snake Game using JavaScript and html5

JS13K Preparation