HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS, which is now in beta. ocPortal 9 will be superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


Robot.txt and adding robot messages to the page-header

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#68677 (In Topic #14581)
Avatar

Honoured member

Using robot.txt lines and adding robot messages lines to the page-header to protect against unwanted crawlers

Hi,

Sorry I bump in with this, but I have little time, did a search what give three messages with the use of 'robot' in it, but nothing about robot.txt.

I bumped this morning in a list of 26 unwanted robot crawlers on our sites where I have something to do about.
I learned how to make robot.txt and a lot of what to put in it to prevent robots to use stuff, that is when they are decent robots.
Some robot messages have to be placed in the headers of pages, and because there are a lot of pages and there are dayly created new ones, I asked myself if oCPortal has some option for it to use lines in the header like <meta name="robots" content="nofollow"> and <meta name="robots" content="noarchive">?

Also the login has a scrambled picture with text and numbers to type in to check for human or not. Is that enough and secure to keep robots out as one can only enter if logged in?

I had one robot in for more then an hour and I don't like that.

Thanks in advance for the reactions,
Puran

21-7-1955 - 24-10-2011 I hope she rest in peace and I hope to see you soon.  Puran
Back to the top
 
Posted
Rating:
#68678
Avatar

Community saint

Just add your meta data to HEADER.tpl and use "website cleanup tools" to flush the cache. As all pages are dynamically generated they will automatically pick up the new header when the robots next visit.

Your login page captcha (the "scrambled picture with text and numbers") should protect your site from 'decent' robots.

Do you have a Samsung Galaxy S / Galaxy S II ? If so, why not check out my ScreenFree FM Radio .
Back to the top
 
Posted
Rating:
#68679
Avatar

Honoured member

Thanks, that will help me further.

Puran

21-7-1955 - 24-10-2011 I hope she rest in peace and I hope to see you soon.  Puran
Back to the top
 
There are too many online users to list.
Control functions:

Quick reply   Contract

Your name:
Your message: