Sponsorship for feature tracker item #290 - Spammer database

Thanks, Chris. I look forward to seeing what you think.

Bob

Ok I've just waded through several forum topics, a few dozen websites, and the tracker issue .

I will try and summarise what is out there, and what I propose, and how long each component would take. I'm going to do that here, before moving it all back to the tracker.

I've taken 3 hours going over all this already, for what was going to be a 3 hour task, but I'll still meet the very simplest form for the 3 hour quote (discussed below). Of course, anyone else may contribute code here if they have the skills, it does not have to be sponsoring ocProducts.

Solutions out there:

1. Mollom
2. Akismet
3. Project Honeypost / HTTP:BL / Bad Behaviour
4. CAPTCHA
5. Heuristics (e.g. blocking anything that submits a form too quickly, or did not seem to use the site properly, or is not apparently running Javascript, or is using what is recognised as a disposable email address)
6. Honey pots
7. Ham/spam bayesian filtering
8. stopforumspam/anatoa/fspamlist/spam-busted block list
9. Mass blocking
10. Manual moderation (active, or queue based)
11. Flood control

1/2 are run by commercial companies, although there are free versions. As far as I am concerned they can be ignored, because they aren't a true representation of how to best do things:

• there is a concentration on web services, because it's a business model to them – by promoting an active content filtering methodology using proprietary algorithms, it justifies their business. I actually don't think that is in any way necessary given the alternative approaches, and I think it has some real disadvantages (notably creating dependencies and complexities).
• a lot of the proposition isn't directly in the proprietary filtering, it's actually in creating even something as simple as a CAPTCHA – a lot of software doesn't come with it. So it's a kind of "all in one", 50% of which overlaps with what ocPortal already does.

I like the look of 3. "Bad behaviour" is a nice open source package that implements HTTP:BL and some heuristic checks. This could be implemented as an addon, in 4 hours work.

We obviously have 4 already. Eventually the new style CAPTCHA that currently is only available by turning on a hidden option will be on by default. It is working stably. We just haven't gotten around to promoting the hidden options to proper ones yet. The problem with CAPTCHA is that real humans are used to spam sites. For 1 hour we could make that a proper option sooner and also add a privilege to bypass the CAPTCHA (i.e. don't just show it to Guests, show it to newbie/unvetted registered users too – a good way to limit spam by those manually creating accounts, or people signed in via Facebook!)

5 is pretty lame, it can easily be beaten by a spammer – it only works so long as what you are doing is obscure enough, and that'll never last.

We could implement our own honey pot (6) in ocPortal, and tie it into the hackattack alert system. However there'd be no guarantee it would be hit before spam comes through, so it wouldn't really help a great deal. I think that's best forgotten.

I don't like 7. We're not restricted to the automated mechanism of the e-mail protocols, we can create whatever blocks and intercessions we want without having to fall to the last resort of implementing keyword analysis (which is not a winnable battle anyway).

I really like 8. That could be done in the 3 hours which is the original proposal, for one of the services.
stopforumspam seems to be the best. The rest I think rely on a human component in identifying spammers, than sharing that data, or they just have smaller databases. I don't think a human component would work reliably, I suspect a spammer would hit tens of thousands of sites successfully before they ever got reported.

Regarding '3' and '8' we would probably want it to be configurable which of the following events it happens on:

• All page views
• Registration and posting and trackbacks
• Registration and trackbacks only
• Trackbacks only
• Never

We could also make an option whether fails on this work to remove the bypass-validation permissions regardless of permission settings on the Guest group, for 1 hour.

I am not a fan of '9', it seems very draconian.

We can do '10' already with permissions, but it'll annoy users and waste time.

For '11', we do have this, but I don't like it so much. I don't think flooding is much of a problem, and flood control can lead to irritation to user. I'll say how we could deal with what (in my opinion) is a better way of dealing with that below (easy mass delete).

A quick review of why we care about bots:

1. Spam
2. Bandwidth use
3. E-mail address harvesting
4. Hacking site

1 is the main thing we are talking about here.
2 is dealt with if we do the 'all page views' blocking discussed above.
3 is a non-issue, ocPortal protects emails
4 is dealt with already in ocPortal

More stuff that could be done…

We could add in additional virtual topics:

• all posts by a specific member
• all posts by Guests
• all posts by an IP address
• all unvalidated posts

Which would allow for mass deletion (0000405: Search feature for super admins\admins and moderators - ocPortal feature tracker).
This would be 3 hours.

We could integrate the punishment actions (ban etc) into the delete post form. i.e. combined punish & delete. This would be 1 hour.

We could spend some time ensuring that we log the registration IP for a member permanently (probably storing the stats entry against the member when they register even though they are not technically logged in). This would be 1 hour. The lookup user features are already fairly good I think, you can see what accounts use an IP, and what IPs an account uses.

There's also this issue regarding ideally we would log both proxy and connection IPs and display both: 0000418: Dual IP address logging and checking - ocPortal feature tracker

---

Phew. I think that covers all the bases:

• Detecting spammers early on, regardless of CAPTCHA (Bad Behaviour or Stopforumspam or both)
• Enhance CAPTCHA, deploy it more widely
• Efficiently dealing with spam
• Slightly better reporting tools

Chris-

Thanks so much for taking the time to analyze this properly.

I currently make use of HTTP:BL through the CloudFlare service and it is very good at catching at least several problems each day, so I am a strong proponent of number 3.

I also use StopForumSpam to look up problem IPs. It provides the means to look up by IP, username or email address and is thus the most comprehensive.

I am not familiar with the StopForumSpam API, but Project Honey Pot returns the number of days since the submitted IP was last used maliciously. I like this since it allows IPs to be aged off any ban list as spammers move on to new IPs.

For me, I think the ideal solution would be a combination of 3 and 8 with the site  owner able to specify a cutoff number of days after which an IP would be removed from the list. I suppose for the sake of thoroughness, there should also be a permanent ban option.

I've been meaning to go through my ban list to see how current some of the early IPs are and whether they still represent a current threat. I'll try to get to that this week just to get a sense of how important aging of bans is.

Bob

Chris-

Can you please complete this thought:

Regarding '2' and '8' we would probably want it to be configurable which of the following events it happens on:
All page views
Registration and posting and trackbacks
Registration and trackbacks only
Trackbacks only

We could also make an option whether this

Also, is that supposed to be "'3' and '8'" or is '2' correct?

Thanks.

Bob

Ah, yes -- post updated.

Summary…


FEATURE: stopforumspam
TIME: 3 hours
PLAN:

• Add config option to ocPortal for storing API key (optional)
• Add config option for when to enable checks (All page views OR Registration and posting and trackbacks OR Registration and trackbacks only OR Trackbacks only OR Never)
• Add option on punishment form to report user to stopforumspam (requires API key)
• Implement checks
• Update default privacy policy to mention use of this service, but only show when enabled in configuration

TRACKER: 0000290: Spammer database - ocPortal feature tracker



FEATURE: stopforumspam & validation
TIME: 1 hour
PLAN:

• Add config option to determine whether stopforumspam bans a user, or whether it just knocks out their bypass-validation permissions; perhaps tied into confidence level
• Implement

TRACKER: 0000432: stopforumspam & validation - ocPortal feature tracker



FEATURE: "Bad behaviour" addon
TIME: 4 hours
PLAN:

• Do a straight integration of "bad behaviour" to ocPortal, release as addon
• Extend it a bit to hook into the same config options and behaviours that the stopforumspam feature does

TRACKER: 0000433: "Bad behaviour" addon - ocPortal feature tracker



FEATURE: Improve CAPTCHA
TIME: 1 hour
PLAN:

• Make enhanced CAPTCHA a proper option
• Make avoiding CAPTCHA a permission so that you can turn it on for non-Guests if you wish

TRACKER: 0000434: Improve CAPTCHA - ocPortal feature tracker



FEATURE: Virtual topics for mass moderation of spam
TIME: 3 hours
PLAN:

• New virtual topic: all posts by a specific member (linked from member profiles)
• New virtual topic: all posts by Guests (linked from Guest posts)
• New virtual topic: all posts by an IP address (linked from Guest posts)
• New virtual topic: all unvalidated posts (Linked from member bar, but only visible if there are any)

TRACKER: 0000405: Search feature for super admins\admins and moderators - ocPortal feature tracker



FEATURE: Punish & Delete
TIME: 1 hour
PLAN:

• Integrate punish feature into post deletion form, so you can moderate more efficiently

TRACKER: 0000435: Punish & Delete - ocPortal feature tracker



FEATURE: Keep track of IPs at joining
TIME: 1 hour
PLAN:

• Retroactively reassign Guest browsing page hits to a member, when a user becomes a member

TRACKER: 0000436: Keep track of IPs at joining - ocPortal feature tracker



FEATURE: Make proxy/client IPs part of the UI and database structure
TIME: 3 hours
PLAN:

• Update database structure to report both
• Most things that use IP addresses must be updated to show both, and explain the difference

TRACKER: 0000418: Dual IP address logging and checking - ocPortal feature tracker

Now I need sponsor(s) to decide how I should proceed . Currently the stopforumspam one is fully sponsored, but given the new ideas here I am considering that pending confirmation.

Personally I think 'stopforumspam' is the core thing for people. It was essentially what was originally discussed, and HTTP:BL ("Bad behaviour" addon) is already available via CloudFront and overlaps quite a bit with it.

Chris Graham said

Now I need sponsor(s) to decide how I should proceed . Currently the stopforumspam one is fully sponsored, but given the new ideas here I am considering that pending confirmation.

Personally I think 'stopforumspam' is the core thing for people. It was essentially what was originally discussed, and HTTP:BL ("Bad behaviour" addon) is already available via CloudFront and overlaps quite a bit with it.

I rethought this and arrived at the same solution.

CloudFlare currently offers a free service which would allow anyone to sign up and gain the benefits of HTTP: BL. It also has the additional benefit of occuring before the IP ever hits your server. In light of this, it seems silly to pay to have this done specifically for ocPortal unless CloudFlare becomes a paid-only solution with no alternative free services.

I check problem IPs using StopForumSpam, so this seems like the logical next step and, to me, the only question is whether or not we also add validation.

We could always add the additional options over time, but my feeling is that being proactive and heading off the spam is the most important issue. Also, it is important to feed banned IPs back into the system.

Bob

Chris-

Hate to be a pest but I think this point is unfinished:

FEATURE: stopforumspam & validation
TIME: 1 hour
PLAN:
Add config option to determine whether stopforumspam bans a user, or whether it just knocks out their bypass-validation permissions; perhaps tied into confidence level
Implement

TRACKER: 0000432: stopforumspam & validation - ocPortal feature tracker

Also, can you describe how you would implement a confidence level check?

Bob

Simple, stopforumspam provides a confidence quotient, and we have a config option that specifies a number relating to those quotient values. Above the number, we block, below it, we just remove validation.

I think adding the confidence level is a good idea, especially in light of the fact the stopforumspam deems it important enough to return the result. It really is part of their whole approach in dealing with spam.

I'll provide half the funding for this – 3 support credits. We can have the stopforumspam with validation if someone else will provide the other 3 support credits for this feature.

0000432: stopforumspam & validation - ocPortal feature tracker

Bob

BobS said

I'll provide half the funding for this – 3 support credits. We can have the stopforumspam with validation if someone else will provide the other 3 support credits for this feature.

0000432: stopforumspam & validation - ocPortal feature tracker

Bob

I'll take up the slack on the remaining three credits - I think I have enough in the piggy-bank. If I haven't, I'm sure Chris will let me know pretty soon!

I've dropped CloudFlare in any case. I found it to be too aggresive for my needs, especially since ocPortal was fending off the 'pimples' without any outside assistance. Also, CloudFlare became a bit of a nightmare when attempting to upgrade the site. Putting it in 'Developer Mode' wasn't the solution as it still needed to clear its caches, and I really didn't like to have to wait for their servers to catch up when I was ready to go!

 

Thanks, Fletch. Let  me know if you have success in sponsoring the three credits – I keep getting errors.

As far as CloudFlare, they need to do some work on the "Development mode" as far as I can tell. However, the option to toggle (or pause) CloudFlare off in the cPanel CloudFlare plug-in seems to work quite well. Still, if you don't need it, you don't need it.

Bob

BobS said

Thanks, Fletch. Let  me know if you have success in sponsoring the three credits – I keep getting errors.

Me too!

APPLICATION ERROR #203
A number was expected for amount.

I'm sure Chris will get around to fixing it.

 

Chris-

Regarding the basic stopforumspam addon: is there any provision for feeding manually banned IPs back to stopforumspam so that we are not just leeching off the resource? If so, we will need to be able to store the API key for each site.

Bob

yes, being dealt with

BobS said

 Let  me know if you have success in sponsoring the three credits – 

Success!

 

Ahh…success for me too. Another feature item fully-sponsored by the community. It's great to be a part of ocPortal's improvement and growth.

Bob

BobS said

Chris-

Regarding the basic stopforumspam addon: is there any provision for feeding manually banned IPs back to stopforumspam so that we are not just leeching off the resource? If so, we will need to be able to store the API key for each site.

Bob

Chris-

I think this got passed over. Can you comment on this please?

Bob

Yes, that was what my vague "being dealt with" reply was for .