How to stop moz dotbot from accessing your website

DotBot is Moz's web crawler, it gathers web data for the Moz Link Index
Dotbot obeys robots.txt rules before accessing your host machine so the easiest way to stop dotbot is by adding robots.txt rules that limit dotbot activities
To forbid a directory let's say "login" add;
User-agent: dotbot
Disallow: /login/
Upon reading and parsing directives above moz dotbot won't dare access you sites login sub directory in it's craw routine
To forbid an entire website's access include directives below;
User-agent: dotbot
Disallow: /
Alternatively you can limit crawl rate by adding directives below "time is probably in seconds"
User-agent: dotbot
Crawl-delay: 10
I've attached an nginx log it's a trail left by dotbot along with it's ip and moz support e-mail address; - - [19/Mar/2020:15:16:29 +0000] "GET /index.html HTTP/1.1" 200 13433 "-" "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.o…

How to set content disposition header for nginx server to force content download

Nginx (pronounced as engine x) is an HTTP server, reverse proxy server, as well as a mail proxy server
Nginx is well known for its high performance, stability, efficiency, rich feature set, simple configuration, and low resource consumption.
content disposition headers direct web browsers to save content instead of attempting to render it
To set content disposition open your desired server configuration in "/etc/nginx/sites-available/"
within your configuration add code below within your server {block}

location ~* (mp3|ogg|wav)$ {     add_header Content-Disposition "attachment"; }
mp3, ogg and wav are example file extensions matched by regular expressions rules
Test configuration by acessing mp3, ogg and wav files from your webserver
Alternatively you can force custom filenames as shown below "$1" substitutes filenames sent to clients. As an example its value corresponds to the requested file's extension
location ~* (mp3|ogg|wav)$ {     add_header C…

Reasons why you shouldn't consider hosting on github pages

Github pages is a service for deployment of static websites normally software documentation and blogs
Github pages is available on all public github repositories and on premium accounts it extends to private repositories as well.
So what's not to like about it. hmmmm think about it. that's too good to be true It's not a secret that "the only free cheese is served on a mouse trap"
1. "Downtime due to maintenance" Every now and then github pages websites undergo an annoying and imaginary maintenance that causes undesirable user experience especially for professional websites.
2. "Slow deployment"
I've been using github pages to host my static website with less than 10,000 pages deployment process knocked my entire site offline for 10 mins again that's unnecessary downtime sometimes it extended to days
3. "uninvited cloners"
So you put your site out there few days later 50 clones uninvited unwelcome and unnecessary
4. "Limi…

How to set content disposition headers for express nodejs apps

In express nodejs apps you can force the user agent "browser" to rather download content instead of displaying or attempting to render given content within the browser.

In this example assuming you're using express.static to serve content pass an Object as second argument to express.static function

An Object with configuration parameters; const config = {
setHeaders: res => res.set('Content-disposition',  'attachment')

In your code replace "path/to/media/" with the path pointing to the static content.

app.use('/media/', express.static('path/to/media/', config))
To test try accessing content at /media/*

HTML minifier module for Nodejs

HTML minification is a process involving striping off unneeded, abundant or otherwise unused HTML attributes and elements without affecting the webpage appearance.minification is useful for statically genetated 'serverless' web application webpages as it delivers easy to digest minimal webpageshtml-minifier is free and open-source npm nodejs module written in JavaScript for minifying HTML documents, as compared to similar nodejs html minification modules html-minifier proves to be pretty fast, flexible and efficienthtml-minifier module can be used to minify HTML code programmatically or alternatively you can minify HTML code using global commandline moduleNpm module InstallationTo install module in your terminal typenpm i html-minifierTo install commandline version of html-minifier in your terminal type npm i -g html-minifierusage:
html-minifier [options] [files...]options:
Strip HTML comments--remove-empty-attributes
Remove all attributes with whitespace-onl…

Open-graph meta data tags introdution

Open-graph elements are meta tags that contain extra meta data of a webpage open-graph is used by Facebook to create website cards for urls shared on it's platform  cards contain website image thumbnail and as well as title, description, type and url open-graph meta tags have a property attribute containing a property prefixed by "og:"For example;<meta property="og:image"  content="image-url">

Solutions to blocked URLs on Facebook

Facebook allows sharing links on it's platform but in order to protect it's users from accessing URLs containing malicious content Facebook on a regular scraps content on shared URLs effectively raising an alarm if content shared goes against Facebook community standards or in short it's content is forbidden on the Facebook platform,Forbidden content may include potentially malicious crafted phishing websites or websites spreading malware among othersFacebook may as well ban content flagged spam either reported or flagged by Facebook moderating algorithm "Facebook sheriff"You wouldn't want to find your website 'spam-list handed' you should avoid thatThere's no way to redeem a website flagged as a blocked website on Facebook except other non viable alternates such as completely changing your domain or using another sub-domainFacebook may as well ban content containing banned URLs in the meta open graph attributesTo resolve open-graph sharing relate…

How to remove null elements from array in JavaScript

Every now and then arrays normally tend to have null or otherwise empty elementsArray filter is a specialized JavaScript function for filtering array elements found to be matching a given criteria for example;const arr = ['max', 'duke']In order to remove duke from elements of the array you can do it as follows;arr.filter(name => (name == 'duke' ? false : true))// returns ['max']You can filter empty and false elements as well, using a similar syntax as done above;const arr = ['max', null, 'duke']arr.filter(name => name)
// ['max', 'duke']

How to strip out html using regular expression in JavaScript

Regular expression isn't a very viable solution for manipulating html due to the complexity of html syntax in the browser it's recommended to create dom elements, appending html source code and then manipulating it using built-in dom querySelectors that has few drawbacks especially for environments without a dom take for instance nodejsYou can use regular expression replace function as;const str = "<b>John swana</b>"str.replace(/(<([^>]+)>)/ig, "")// John swanaAnd that's it

Es6 new features summarized round up

Escmascript es2017 or better known as es16 is the edition after the famously popular es15, In this edition there are only two new additional features the exponention operator and Array.prototype.includes() functionexponention operator** Operator is a short hand method used for raising numbers to their powerIts syntax is similar to native operators such as '+' Addition and '-' (minus) for subtractionExample usage
Initially to raise a number to the power you would need to use math library pow function as;Math.pow({number}, {factor})Since es2017 you can simplify the operation above down to simply as;{number} ** {factor}Example as;5 ** 2
// Giving you 25Array.prototype.includes()Array includes is a feature similar to a pre existing function array.indexof() as compared to array.indexof() array.includes() function notably returns a true  boolean on matched value, and false if value not matchedExamples on how to use 'array.includes()' functionThe first argument is the…