Cloudflare CDN

BobS said

I didn't enter all the Pingdom addresses since I was only having issues with these three but it won't hurt to grab all the Pingdom addresses and place them here.

Stupid question. How did you know it was just these addresses causing a problem, and where do you get all of Pingdom's IPs?

I'm currently trying to get my head round basic webmaster stuff.  Can anyone recommend any good, simple, up to date general web amdinistration documentation?  (I'm technical, just not done anywebsite administration before.)

Thanks!

I found the three problem addresses while checking my Apache error logs - they were getting a 403 due to other rules in my .htaccess file.

I found this list of addresses for Pingdom just now that is probably reasonably accurate but it is closing in on a year old.

LVTC (Living Vicariously Through Corey):LVTC - Living Vicariously Through Corey » .htaccess file to allow pingdom servers

Just to make sure I would do a whois lookup at (you can do this on domaintools.com) on each of the addresses to confirm that they are Pingdom addresses. I just noticed that the 141.101.70.127 does not belong on the Pingdom list - it is part of a Cloudflare block of IPs.

Bob

Thought I'd report back on my experiment of moving my country blocklists from my .htaccess file to Cloudflare. Cloudflare does not actually ban all IPs from the country but requires that the user enter a CAPTCHA which I have set to remain valid for 45 minutes.

So far this has worked well. When the blocklists were in my .htaccess (where they were incomplete) I could count on getting hit by a spammy address from China, South Korea, Vietnam or Thailand at least once a day. Since moving the country blocklists to Cloudflare, I have not had a single spammy address from these countries while average users who pass the CAPTCHA are now able to browse. I am sure some are put off by the CAPTCHA but before this change, they mostly got a 403 error unless it was an IP from an address block not in my blocklist. I recognize that some of this may be pure luck in that spammers were not hitting my site as hard but Cloudflare indicates that they have stopped 4-6 "threats" each day. I am pretty convinced this is  working and so plan to move my Soviet Bloc countries blocklist from my .htaccess to Cloudflare.

I still have some blocklists of known exploited servers in my .htaccess file. I should be able to remove these under the assumption that Cloudflare is also aware of these addresses.

I have not yet decided how I will handle single IP address bans. I may leave those in my .htaccess file or I might ban them in Cloudflare - my guess is that Cloudflare processes this information faster than my server plus I have the benefit that they do not "pollute" server or ocPortal statistics.

Thinking of the future, should the spammer database feature request ever get sponsored, I hope that Chris uses Project Honey Pot as the source of bad IP information as it seems to work really well in Cloudflare. I would hope that any IPs banned in ocPortal would feed that information back to Project Honey Pot so that such bans are included in future results for other Project Honey Pot users.

The features I have mentioned here are all available in the free version of Cloudflare.

Bob

I thought I would report my results from the second week of mu site using the Cloudflare proxy. Although the results are not as dramatic as the first week, they are still pretty impressive - saving about 53% of bandwidth and 59% of requests to my server. I consider this a success.



You will also notice that a number of threats were stopped, mostly known threats from Project Honey Pot including spammers and some zombie botnets. In addition, the mustardy color of the pie chart reflects the Asian country blocks I moved from my .htaccess file to Cloudflare where bots did not successfully enter the CAPTCHA. I am sure some real people are put off by the interstitial CAPTCHA page but, for most, it beats getting the 403 error page they were getting. The CAPTCHA provides a space for visitors to ask to be whitelisted.

I considered that Asian country blocks experiment successful enough to have expanded it to the former Soviet Bloc countries today. This is going to greatly reduce the amount of time I was tending to keeping my country blocklists up-to-date and looks to provide the same or even better security against threats.

Just a reminder that these features are all accomplished with the free version of Cloudflare.

Bob

BobS said

Cloudflare does not actually ban all IPs from the country but requires that the user enter a CAPTCHA which I have set to remain valid for 45 minutes.

I've also been using this feature successfully and find it to be much fairer than total ban of a country's or region's IP. If someone were to ban my IP in the standard way through .htaccess, the entire country of Aruba would be banned. Actually, this is alarmingly happening and caused by indiscriminate methods of protection.

Kudos to Cloudflare and those like you Bob, willing to investigate and try something outside the box.

Again, thanks for sharing these experiments and results.
Jean  

Thanks, Jean. I can't code, I can't skin but I can test things out and report.

I'm still tinkering with settings and have an idea for boosting performance that i need to think through and experiment. I've set up a Cloudflare proxy for my test site which is a clone of my live site so I can afford to experiment. That said, RocketLoader appears to not work with the v8 version of ocPortal, apparently due to the JS changes. RocketLoader is still in beta so it's not too surprising but I'll be dropping Cloudflare a bug report today on this issue.

Bob

A quick update on my experiment. I have entered the former Soviet Bloc countries into my Cloudflare blocklist. This had probably had only a small impact on my site since I was blocking many, if not most, IPs from these countries in my .htaccess file. I no longer need to worry about new IPs being assigned or individually updating my .htaccesss file with the CIDRs for these countires. Plus, if someone from a blocked country properly enters the cAPTCHA, they are granted access to the site. My logs indicate that this is already working and, so far, these visits have not been in any way malicious. I'm posting the security graph for the past 24 hours to demonstrate what Clouflare's country blocklist and browser integrity checks are catching. Remember that both these features are available in the free version of Cloudflare.



I've also signed up for the Pro plan. It's a little pricey at $20 a month but I want to see if the advertised additional performance benefits improve my site loading time. I'll be turning on features gradually over the next week and seeing what improvements I can eek out. Cloudflare reps have indicated that they will likely be offering a discounted annual subscription for the service in addition to the current $20 monthly subscription in the future. Frankly, the savings in time for me maintaining the country blocklists in my .htaccess file are worth at least $10 a month to me.

The Pro plan promises prioritization on Cloudflare's network and a Website Preloader which is a pre-fetch that tracks your most often accessed assets and updates this information twice daily in the Cloudflare cache and then delivers these to the visitor's browser in the background. It also includes "Advanced security which is described as "Our Advanced Security is like a web application firewall (WAF) that exceeds OWASP standards. It runs in real-time, preventing automated attacks, SQL injection, XSS javascript injections and a host of other real-time POST attacks." There are also some beta Apps which will be available only to Pro users: Mirage which promises to resize images within context prior to delivering them (this does not work for me) and Polish which allows you to set a compression level for your images. The latter feature holds some interest for me since I purposefully included large images on my site so that visitors would have adequate resolution to print the images at 300dpi for an 8x10 output. I am hoping that I can optimize my images for display at 80% while delivering the full-size file as a download.

I am really impressed with what the free version of Cloudflare offers; going "Pro" will have to really provide significant improvements for me to be impressed.

Bob

Another quick update on my experience with Cloudflare which has been generally positive.

After adding the former Soviet Bloc countries to my Cloudflare blocklist (removing them from my .htaccess file), I have seen a further reduction of visits from spammy IP addresses. Ironically, I actually have more visitors from these countries now once they enter the CAPTCHA but they appear to be legit (not spammers).

My bounce rate continues to drop and the average  time per visit is increasing - these are likely just the result of the bad actors never hitting my server.

I am running into some rough edges but they all seem to be with features clearly labelled as beta. I report the issues and hope they will be addressed when these are properly released.

One issue I have discovered (via Google Webmaster Tools) is that I am suddenly getting 404s on a few URLs that are appending the beginning of content after the .htm suffix. I am wondering if it might be the auto-minifying of HTML but Cloudflare tech support doesn't think so. I am pretty sure it is related to Cloudflare since it just started happening and is on my live site (not yet upgraded to v8) which has been very stable otherwise. I am running a link checker against the site now to see what I find but I think this is ultimately going to be a Cloudflare issue.

Bob

I've found a couple of areas I would like to see improved in Cloudflare.

First, the "Alert Panel" contains a list of all IPs which have been challenged for which the CAPTCHA was not successfully entered. These IPs stay on the list until the CAPTCHA is correctly entered. I have suggested to Cloudflare that they should consider aging the IPs off the list based on Project Honey Pot's rule which is that if there is no negative activity for an IP for a 90-day period, they no longer consider the IP a threat. This will make the list more manageable while providing true current threat information since PHP is constantly updated with new and renewed bad IPs.

The second issue is that I would like to see the ability to add an IP directly to the Alert (Threat) panel instead of blocking it which would then use the above aging process. One thing I dislike about banning IPs is that the IP you ban today will be used by someone's grandma in six months and she shouldn't have to deal with the repercussions of something that happened ages ago.

Cloudflare has added both of these suggestions to their discussion list, so here's hoping.

I have also requested that they consider the ability to block by user-agent like they do with countries (which just forces a CAPTCHA). PHP has recently started to catalog bad user-agents so this would be a good integration between Cloudflare and PHP.

Overall, this is really a very good product and people might want to take  look at the free version which has many of the important features like edge-node caching and the automatic checking of Project Honey Pot for bad IPs. If you have a Cloudflare icon in your cPanel, setting it up is relatively easy and you can turn it off and on from that panel. You might have to change an "A"  record to a "CNAME" in your DNS Zone editor but your host should help with that.

Check it out.

Bob

Well, here are the results of my third week with Cloudflare. The bandwidth save dropped under 50% but this was probably the result of running a link checker through the Cloudflare proxy - it is doubtful that they had all my links cached. The number of requests that did not need to hit my server (especially under the extreme stress brought by the link checker) is still quite impressive.



It is also impressive to see that so many "known threats" are being caught before they ever hit my website. Blow is the threat alert panel which shows all IPs which were issued a CAPTCHA (rather than being blocked) but failed to answer the CAPTCHA successfully. Note that all except the first challenge for India were initiated using the Project Honey Pot data and whatever heuristics Cloudflare applies. The entry from India show that the CAPTCHA was successfully entered and that the user left a message asking me to grant access. The IP was granted access for 45 minutes (my setting) as a result of answering the CAPTCHA. I am still determining what to do with this IP - although there are currently no threats from this particular address, virtually the whole range of addresses from this IP address block are lit up as threats. I may go ahead and dismiss the challenge and see if the IP is once again challenged.



I do want to point out again that these features are available in the free version of Cloudflare, so you might want to give it a go and see if it helps improve your site performance and security.

Bob

Here is a link to Cloudflare's hosting partners page: https://www.cloudflare.com/hosting-partners

If your host is listed on this page, there is a good chance that you will find an easy-to-use Cloudflare option in your cPanel to make setting your site up on Cloudflare pretty much painless, including setting up the basic options. From there, you need only further configure your site t the Cloudflare site for the other options presented.

Bob

So, I haven't written about CloudFlare in a while and thought I would provide an update. Note that I am using the "pro" version of CloudFlare which includes some additional benefits like prioritization on their network. What really prompted me to write this is that I have noticed a significant difference between my test site (which is not on CloudFlare) and my live site. The difference in page loads is significant even allowing for the prioritization. Of course, my site is probably a near-perfect match for CloudFlare with my large images now cached on their edge nodes instead of coming from my site. Just to give you an idea of its effectiveness, here are my CloudFlare stats from the last 30 days; note the number of requests and bandwidth saved. Also notice the number of threats caught.


Even though ocPortal's recently coded anti-spam feature would catch much of this (and more), I am going to continue to use CloudFlare for the following reasons:

1. I don't get a lot of relevant traffic from Asia, Eastern Europe and Africa. CloudFlare allows me to "block" these countries before they ever hit my server. But what it really does is issue a CAPTCHA so that bots are stopped but humans can get through. Since we also have human spammers, this is where ocPortal's new anti-spam feature will kick in by checking the a different spam database which will catch most everything that needs to be dealt with.
2. CloudFlare keeps track of problem user agents and blocks these automatically. These are usually rogue bots/crawlers that provide no benefit.
3. CloudFlare injects my Google Analytics code into my large images served up "raw" with no HTML. This provides more accurate reporting than having GA only account for pages that include the tracking code. Yeas, I could add the code to the image pages but why bother, especially since the whole point of these pages is to serve up these very large images without any distractions – just the image itself.
4. Since activating ScrapeShield, I have had fewer scrapers taking my content and the hot-link protection also seems to be working well.

Just a note that all of the above features are available in the Free plan. I am pretty impressed by what they
offer for free.

CloudFlare does have some issues which they are working on including improved threat reporting. There are occasional issues where going into "Development mode" does not bypass the cache - they are aware and are working on this.

Overall, I consider this one of the best investments of my time and efforts because it is providing improved performance and better security each and every day – even if I were just using the Free Plan.

Bob

Issues with my host this morning reminded me of one other benefit of using CloudFlare – their "Always online" option which attempts to present as much of your website as possible. There are obvious limits here such as when you need to log in or otherwise reach the database, but it seems like it serves many pages properly.

This feature is also available in the Free Plan.

Bob

BobS said

Issues with my host this morning reminded me of one other benefit of using CloudFlare – their "Always online" option which attempts to present as much of your website as possible. There are obvious limits here such as when you need to log in or otherwise reach the database, but it seems like it serves many pages properly.

This feature is also available in the Free Plan.

Bob

Yes, you are right Bob. My sites seemed to work fine while on CloudFlare after my own problems with the server. It is only after I disabled CloudFlare and deleted my templates_cache that things got ugly. This system can be a useful temporary solution to prevent user panic

As you can tell from my posts, I am pretty much sold on CloudFlare. It does add a layer of complexity and you have to wonder if that is what might be causing issues, but as you suggest, you issues arose after disabling CloudFlare.

I can think of few free services that provide as much as does CloudFlare. While I am sure that they will continue to add more to the Pro Plan to make it more attractive, they are continuing to add to the Free Plan as well.

Bob

Another example of why I like using CloudFlare:


Note that this was over just a six-hour period (less than 24-hour granularity is a Pro feature). These threats never touched my server. My traffic has dropped a bit since before I used CloudFlare but so has my bounce rate – from 50-60% down to 10-15%. I consider this pretty good for such a specialized site.

Bob

Bob,

Do you have any idea how Cloudflare might work for a less photo-intensive site?  My (still very young and incomplete) site is pretty much all text at the moment, and not static content at that (wiki and forum posts, mostly wiki).  But I am on a shared hosting solution for the moment so I'm wondering if there is a more aggressive caching option that might be advantageous for a site like mine that would performance.  Thanks

Hey D-Train-

I run my site with the "aggressive" caching level. I am not sure how much benefit you would see but you would save on any theme graphics/icons and, of course, the javascript and CSS. You can also safely engage the auto-minify for JS, CSS and HTML. I'd stay clear of Rocket Loader as I've had some issues when I've tried using it (it loads the JS asynchronously and I sure wish it would work and probably could if I could tag the resources to bypass).

Your biggest benefit might be in the security area, although that can spill over to performance if you are getting a lot of hits from bad bots. CloudFlare could keep these from ever hitting your server. Since the weak link in most shared hosting solutions is the shared database server, this could be significant if you have a lot of bogus traffic.

Since it costs nothing for their fairly full-featured "Free" plan, I'd give it  try and let it run for a week or so so that the cache fills.  If there is no benefit, you can always turn it off.

Bob

Bob, I added the free Cloudflare to my site to give it a try and I was curious about something.  The Dashboard reports page views from "Known Threats", but am I missing something?  I haven't seen anything that says they are actually blocking access from those threats, just that they identified the source.

I also find it funny that I getting more threat readings from Canada (of all places) than all other countries combined.  But admittedly, I just started the service a couple of days ago and my site is still very new.  Not like yours.

Thanks for you continuing updates in this thread.  It's been interesting.

The "Known threats" are stopped dead in their tracks without any further reporting in the "Threat control" page.

I have Canada as the argest offender a number of times although typically the USA, Russia, China and other expected counries are among the leaders.

Bob