>>
Site Map
>>
Forums
>>
Security Issues
Forum module - topics in forum:
Security Issues - Get help in securing your PHP-NUKE Installation.
msnbot
Currently, I am being crawled by msnbot constantly on my site. whenever I do searches on MSN for my website, it shows my site with contents of the shoutbox I have as the description. Is there some form of robot we can use to prevent them from crawling everything on my site? I would just like my website description to read what my site is about. Not the contents inside it. It seems to like to crawl the forums and my shoutbox alot. If not, can I halt these crawlers?
phpNuke 7.6 - 2.9 Patched
Sentinel 2.2.0
First make sure you have a robots.txt it should be in root.
If you have one make sure it contains the following text:
| Code: : |
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Disallow: admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/ |
If you don't have one put the above in a text file, name it robots.txt and upload it to root (the same place your modules.php file is).
Yes it does
then try adding this to the robots.txt:
| Code: : |
| Disallow: msn-bot |
So it looks like this:
| Code: : |
User-agent: Mediapartners-Google*
Disallow: msn-bot
User-agent: *
Disallow: admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/ |
Click here for a tutorial on creating an effective robots.txt file
