HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS, which is now in beta. ocPortal 9 will be superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


Robots.txt

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#96289 (In Topic #19126)
Avatar

Well-settled

Hi,
In the error logs of my host I have the following logs:

[Wed Mar 20 17:22:28 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/404.shtml
[Wed Mar 20 17:22:28 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/robots.txt
[Wed Mar 20 17:17:48 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/404.shtml
[Wed Mar 20 17:17:48 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/robots.txt
[Wed Mar 20 17:17:40 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/404.shtml
[Wed Mar 20 17:17:40 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/robots.txt
[Wed Mar 20 17:16:27 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/404.shtml
[Wed Mar 20 17:16:27 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/robots.txt
[Wed Mar 20 17:15:07 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/404.shtml
[Wed Mar 20 17:15:07 2013] [error] [client 5.9.7.208] File does not exist: /home/xxxxxxxx/public_html/robots.txt


According to OCportal Security tutorial if I add a robots.txt to my root directory I believe at least some of those "errors' will go away.

2 Questons:

1.) I have a re-direction for my index.php to the sub-directory of my install.  Do I put robost.txt in my "installed" sub-directory or still put it in root?

2.) Should I be concerned about these log statements are they hints of something else going on?

Thanks in advance.
~larryg1957

Back to the top
 
Posted
Rating:
#96293
Avatar

Community saint

1) robots.txt always goes in the root of the domain/subdomain.

2) No need to be concerned about these at all, its just search engines like google looking for any special instructions from your site on what it can and can't crawl. And if you add a robots.txt then those 404's will also disappear as they are being generated because you have no robot.txt file.

Do you have a Samsung Galaxy S / Galaxy S II ? If so, why not check out my ScreenFree FM Radio .
Back to the top
 
Posted
Rating:
#96294
Avatar

Well-settled

temp1024 said

1) robots.txt always goes in the root of the domain/subdomain.

2) No need to be concerned about these at all, its just search engines like google looking for any special instructions from your site on what it can and can't crawl. And if you add a robots.txt then those 404's will also disappear as they are being generated because you have no robot.txt file.

Thx. Temp1024.  That's kinda what I thought.   Thanks again for the info.
Back to the top
 
1 guests and 0 members have just viewed this: None
Control functions:

Quick reply   Contract

Your name:
Your message: