How to test Stop Web Crawlers on your website

In order to test to see if Stop Web Crawlers is working on your website and to ensure that it is protecting your website from a specific Referrer spam bot.  You will need to download a Browser plug-in or application like:

 

Postman

Postman is the swiss army knife of API tools, allowing you to design, build, test, document and monitor your services, all in one place. You will need to download Postman, it used to available as a Chrome browser plugin, but now it is a stand alone application which requires download.

If you are going to be developing or exploring API’s to use, it is a really handy tool to get to know. It is also a free tool!

Testing Stop Web Crawlers with Postman is extremely easy.

postman

  1. Enter Your URL
  2. Select Headers
  3. Enter Referer
  4. add a domain you would like to test to block
  5. click send
  6. Check the result. You should see 403 Forbidden

 

 

HTTP Requester

Our personal favourite is Http Requester due to its ease of use and friendly user interface. However most REST clients work in much the same way.

The Instructions below will guide you through making use of Http Requester . Complete the form as below:

Http Request Configuration

Once complete click GET

Test Referer spam result

 

Gary Woodfine

Founder & CTO of threenine.co.uk. Experienced full stack software developer well versed in delivering web & mobile applications utilizing cloud architectures. Areas of speciality include cross platform development in .net , PHP, JavaScript.

  • Julius Musembi

    Tests on my website show your plugin is not protecting me. What should I do to make it work?

    • Julius,
      Please let us know what your Url is?
      We can take a look for you.

      • Julius Musembi
        • Hi,

          We just conducted a test using httprequester on your url supplied and all is working as expected.

          We see the issue you’re recieving with Postman , it appears that
          postman now requires to use Postman interceptor. You should ensure to
          install and enable interceptor on your Postman in order to be able to
          access the restricted headers in postman

          • Julius Musembi

            Thank you so much!

π
%d bloggers like this: