HTML Logo by World Wide Web Consortium (www.w3.org). Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS, which is now in beta. ocPortal 9 will be superseded by Composr 10.

Head over to compo.sr for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.


Need some Google advice

Login / Search

 [ Join | More ]
 Add topic 
Posted
Rating:
#74230 (In Topic #15455)
Avatar

Community saint

In Webmaster Tools, I originally set up the account at domain.com. Unfortunately, Cloudflare requires the www. prefix so I set up www.domain.com. In my settings, I have set the preferred domain as www.domain.com.

The domain.com site has a bunch of 401 errors from when the site was protected with a password. I figure these will rectify themselves over time but some of those URLs are ending up as 404s. I know why it's happening but I have no clue how to fix it. If I click one of the 404 links, I end up directed to

Code

http://www.domain.com/http%3A//www.domain.com/guidelines.htm
which is indeed not available.

This seems to have something to do with the www redirection in .htaccess. At first, I thought it was because I left out the "Rewrite Base /" but, after including that, I still have the same problem. Everything works properly if I type domain.com into the address but it seems to have a problem any time the URL is for a specific page.

This is the section of my .htaccess that deals with the www redirect:

Code

RewriteEngine on

#Redirect to www
#RewriteBase /
rewritecond %{http_host} ^domain.com [nc]
rewriterule ^(.*)$ http://www.domain.com/$1 [r=301,nc]

I've removed the RewriteBase" directive because I am not sure if it will cause problems with ocPortal's existing rewrites (and it doesn't seem to make a difference anyway)

So, my questions are:

  1. Is it safe (or necessary) to specify the RewriteBase?
  2. What must I do so that the rewrite rule applies to more than just the main domain?
  3. How do I eliminate the addition of the same domain that is presently occurring?

If anyone can help, I would really appreciate it.

Bob


Last edit: by BobS
Back to the top
 
Posted
Rating:
#74247
Avatar

Community saint

I think I have this working properly now. This is the current code for the redirect:

Code

#Redirect to www
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\.domain\.com [NC]
RewriteRule ^(.*)$ http://www.domain.com/$1 [L,R=301]

I don't really understand why this works - apparently the difference between having L instead of NC in the RewriteRule.

But it has introduced an issue with my link to the large images in my galleries. This is what was working before the updated rewrite code:

Code

{$COMCODE,[url="View large image" target="_blank"]{$BASE_URL*}/{$REPLACE,/catalogues/,/catalogues/large/,{FIELD_5_PLAIN}}[/url]}

This now results in a 404. I'm guessing it has something to do with the {$BASE_URL}.

Any suggestions?

Thanks for any help.

Bob
Back to the top
 
Posted
Rating:
#74250
Avatar

Community saint

Changing from nc to L, I'm surprised you aren't having more problems.

From memory, the L means that its the last rule so when it matches that line it will not process any more RewriteRules. The short URLs would not work any more (as your rule is right at the top).

Sorry I don't have time to look into it in more detail for you.

Do you have a Samsung Galaxy S / Galaxy S II ? If so, why not check out my ScreenFree FM Radio .
Back to the top
 
Posted
Rating:
#74262
Avatar

Community saint

Hmmm…strange that it is working given what you say. I think I will move that code to the bottom of the rewrite code just as a precaution.

I wonder if the problem with the large image link being broken is a result of the L being included in the redirect. I plan on working on that today.

I'm just stumbling about trying to get my site properly indexed by Google while not breaking anything. They finally added 8 (out of 243) pages this morning.

Thanks as always for your help.

Bob
Back to the top
 
Posted
Rating:
#74279
Avatar

Community saint

I got my "View large image" link working. Turns out it had nothing to do with the URL redirection; I just forgot to adjust the link when I added another field.

I've moved my redirect code to the bottom of the rewrite stuff so that it is the last item. Evrything seems to be working fine. I guess I'll know in a few days if Google starts cleaning up its 404s.

Bob
Back to the top
 
Posted
Rating:
#74281
Avatar

Community saint

The top is the right place for something that will affect every single URL.

If there are any other rewrite rules with L, then you can't just place the new rule at the end because it will end when hits any of them, and you want to add www to every URL.

Do you have a Samsung Galaxy S / Galaxy S II ? If so, why not check out my ScreenFree FM Radio .
Back to the top
 
Posted
Rating:
#74283
Avatar

Community saint

Almost all the rewrite rules include L which makes me wonder if that really does mean "stop processing".

I have found lots of examples of what the redirect should look like but I havenot yet found something to explain what those letter mean (NC,L,QSA - not to hard to figure out that the R means 'redirect').

The funny thing is, everything on the site seems to work regardless of whether the rule is at the top or bottom. I also question whether there is any real benefit in adding the "RewriteBase /" as things seem to work without it.

The scary part is that I really am not sure what I am doing and though everything I've tested so far has worked (even what I thought was broken above), that certainly doesn't mean that everything, in fact, works.

Where would you put the wwwredirect? Top or bottom?

Bob
Back to the top
 
Posted
Rating:
#74287
Avatar

Community saint

temp-

I moved the www redirect back to the top and checked it by using the problem URLs in Google webmaster tools. It seems to work fine/

Because the L means that additional rules are not processed, I removed the 'L' flag (leaving just the R=301) and tested against the same URLs in Webmaster Tools. The result was 404s because URLs were built looking like this:
 

Code

http://www.domain.com/http%3A//www.domain.com/lifescapes.htm

So I put the 'L' flag back but worry that there could be problems with something else down the road.

The only other option I can think of is putting the www redirect at the end of the rewrite rules with only the 'R=301" flag.

I found a good source explaining the flags at Apache Rewrite Cheatsheet | BorkWeb.  I think you might have posted this the other day but I couldn't find the file on my desktop or the post.

This regex stuff is just plain confusing to me.

Bob
Back to the top
 
Posted
Rating:
#74288
Avatar

Community saint

BobS said

Almost all the rewrite rules include L which makes me wonder if that really does mean "stop processing".

It means stop processing further rewrite rules IF this rewrite rule is matched.

The fact that most rules have L simply means that most URL are fully resolved in only a few rewrites.

but I havenot yet found something to explain what those letter mean (NC,L,QSA - not to hard to figure out that the R means 'redirect').

Sorry, I assumed that you had found at least some info on syntax. Here is a summary:

    * C (chained with next rule)
    * CO=cookie (set specified cookie)
    * E=var:value (set environment variable var to value)
    * F (forbidden - sends a 403 header to the user)
    * G (gone - no longer exists)
    * H=handler (set handler)
    * L (last - stop processing rules)
    * N (next - continue processing rules)
    * NC (case insensitive)
    * NE (do not escape special URL characters in output)
    * NS (ignore this rule if the request is a subrequest)
    * P (proxy - i.e., apache should grab the remote content specified in the substitution section and return it)
    * PT (pass through - use when processing URLs with additional handlers, e.g., mod_alias)
    * R (temporary redirect to new URL)
    * R=301 (permanent redirect to new URL)
    * QSA (append query string from request to substituted URL)
    * S=x (skip next x rules)
    * T=mime-type (force specified mime type)

That list was taken from one of my bookmarks:  URL Rewriting for Beginners - Added Bytes by Dave Child

The funny thing is, everything on the site seems to work regardless of whether the rule is at the top or bottom.

Pick a few random links on your site, and try each of them with and without a www in from of them.

The reason it "seem" to work may be the fact that the first time you get to your site the www rule is matched, and you get to www.yoursite.com. From then on all subsequent links you click will inherit that www, so the www rule is not matched again.

I also question whether there is any real benefit in adding the "RewriteBase /" as things seem to work without it.

Don't know about the specifics of that one.

The scary part is that I really am not sure what I am doing and though everything I've tested so far has worked (even what I thought was broken above), that certainly doesn't mean that everything, in fact, works.
:lol:

Where would you put the wwwredirect? Top or bottom?

You want to ensure that every URL has www, so it should be at the top. Its just a straight hierarchy. WWW affects most links so it goes on top.


Do you have a Samsung Galaxy S / Galaxy S II ? If so, why not check out my ScreenFree FM Radio .
Back to the top
 
Posted
Rating:
#74289
Avatar

Community saint

Temp-

Thanks for the list of flags. It is more complete than the one I found. I also added the above link (as well as the two about regex you gave me the other day - I think I bookmarked those but I was doing some bookmark housekeeping the other day and managed to toss away some stuff I did not mean to).

I've been testing by using the 404 links in the non-www domain in Webmaster Tools. Most of those failed because the URL was written as

Code

http://www.domain.com/http%3A//www.domain.com/lifescapes.htm
without the L flag on the rewrite.

I am going to test some more as you advised.

This whole thing would not have been necessary (and I would have been able to use my preferred non-www domain) except that the Cloudflare CDN requires the www prefix to work. I have yet to set that up but I wanted to get my domain/URLs sorted right away for the search engines.

Thanks so much for your help.

Bob
Back to the top
 
Posted
Rating:
#74290
Avatar

Community saint

Well, I am sure something is going to bite me in the *ss down the road but things look fine from my testing.

Rather than using random URLs, I used the Crawler error lists from Webmaster Tools making sure to clear my browser cache between each URL.

First, I used the 16 404s and many of the 115 401 errors (being careful to check URLs both with and without query strings) from the non-www domain; in every case except where a file no longer existed because I deleted it, these non-www addresses resolved to the new www address.

I then did the same using the 12 401 errors and the single 404 error from the www-domain crawler list. Again, everything worked.

I think I will just check these lists every day for the next month or so. If I don't run into any problems by then, things should be working fine or, at least, it will be an odd occurrence that has trouble.

Now, I just need to figure out if I can use ocPortal's redirect feature to redirect from some old forum sub-categories to their parent category.
I decided to change the forum structure so I just want to dump people into the parent where they can make a new choice. Fortunately, there were no posts in the forums involved that were crawled.

Bob
Back to the top
 
Posted
Rating:
#74299
Avatar

Community saint

It sounds like you just had some bad luck with the timing of your redirect changes and when google actually visited your site.

I can't help you with the ocPortal redirects. The one time I tried using it ages ago I got nowhere. :( .

Do you have a Samsung Galaxy S / Galaxy S II ? If so, why not check out my ScreenFree FM Radio .
Back to the top
 
Posted
Rating:
#74301
Avatar

Community saint

temp1024 said

It sounds like you just had some bad luck with the timing of your redirect changes and when google actually visited your site.
It didn't help that that was the day I was so frazzled by my site being offline and I uploaded the sitemap before removing the directory password. Another doofus moment.

I can't help you with the ocPortal redirects. The one time I tried using it ages ago I got nowhere. :( .
Hey, for once we both got to the someplace!   :lol:

Bob


Last edit: by BobS
Back to the top
 
Posted
Rating:
#74305
Avatar

Community saint

BobS said

This whole thing would not have been necessary (and I would have been able to use my preferred non-www domain) except that the Cloudflare CDN requires the www prefix to work. I have yet to set that up but I wanted to get my domain/URLs sorted right away for the search engines.

I'm thinking about using CloudFlare also, so I'm interested in knowing how this works out for you.

Steve
Back to the top
 
Posted
Rating:
#74307
Avatar

Community saint

I'll post as I implement it.

But, for starters,make sure for primary site is at www.domain.com. The Cloudflare proxies work only with the www.prefixed domain (which I hate because it makes my long url even longer and less memorable).

Bob
Back to the top
 
Posted
Rating:
#74308
Avatar

Community saint

I had set my site up without the www so now I'm letting you work out the bugs   :thumbs:   before I go down that route of changing everything over to use CloudFlare.   ;)

Steve
Back to the top
 
Posted
Rating:
#74324
Avatar

Community saint

Steve-

The one point I would make is that if you know you will be using Cloudflare because you have a lot of static content (for me, large images, a video and,soon,some audio), the you should get your site switched over to the www sooner rather than later so that the search engines grab the right url.

Of course, if your site is already existing and well-indexed, yo may take a bit of a roller-coaster ride as the SEEs sort things between the two domains. Google has a setting to choose which is the preferred domain but I haven't found anything similar in Bing.

Bob
Back to the top
 
Posted
Rating:
#74345
Avatar

Community saint

I'm not in a big hurry to make the switch. If it happens in six months, then that would be ok for me.

Two months ago I started transforming a forum based site into a content based site, so right now the majority of my content is forum, with the peripheral content slowly growing.

A quick single-word search just now on Google put me at about 30 from the top and a two-word search put me at #1.  :cool:   I don't have a whole lot of competition for my type of website, so I'm not too focused on Google rankings at the moment. .  :lol:

Anyway… I'm not in a big hurry to change things around. My members are from all over the world and I just want to use CloudFlare to help speed up page loading for those members outside the US.

Steve
Back to the top
 
Posted
Rating:
#74383
Avatar

Community saint

Well, the rewrite to the www.domain.com/xxxxx.htm seems to be doing its job. I dropped from 115 401s in Webmaster Tools for domain.com to 70 with no increase in 404s (and those are all dated prior Sept 1 so they will hope duly be dropping off, as well).

The Webmaster Tools for www.domain.com shows them holding steady with 6 URLs that are 401 to guest, 1 404 URL that should resolve and the 4 404s that are legitimate because I changes the forum category structure. I need to get those last for ironed out.

Bob
Back to the top
 
Posted
Rating:
#80734
Avatar

Community saint

sholzy said

I had set my site up without the www so now I'm letting you work out the bugs   :thumbs:   before I go down that route of changing everything over to use CloudFlare.   ;)
It looks like Cloudflare now supports non-www sites, as well. I'm thinking about changing my preferred domain to the non-www version and making the necessary adjustments but I am at a critical point in terms of site ranking and do not want to jeopardize my current ranking right now so I will need to think this through.

Bob
Back to the top
 
1 guests and 0 members have just viewed this: None
Control functions:

Quick reply   Contract

Your name:
Your message: