Sponsorship for feature tracker item #290 - Spammer database

I will provide one hour (6 credits) for this project or one-third the total cost.
0000290: Spammer database - ocPortal feature tracker

This project has at least two other members interested so, hopefully, they can each make the commitment for 6 points (just over US$50) which would fully fund these project.

Bob

I am rethinking my support for this feature. Although my "dynamic" IP address never seems to change, other's will.

The biggest problem I see is with the increased use of mobile devices. The IP reported is going to be the IP address for apiece of equipment at a cell tower. You really can't ban that IP address because someone completely different will be using that same IP minutes later, especially if you are bouncing from tower to tower.

Bob

BobS said

The IP reported is going to be the IP address for apiece of equipment at a cell tower.

I highly doubt that, its bound to come from a centralised pool.

You really can't ban that IP address because someone completely different will be using that same IP minutes later, especially if you are bouncing from tower to tower.

The IP addresses will be session based at a minimum, meaning that you will keep the same IP address for the life of a session, and will not get a new one every time you bounce between towers. If you did you would kill anything that is currently downloading, and would kill any authenticated logins and web site sessions.

In other words the net would be effectively unusable.

So, my specific points might be wrong but I believe the point of my argument holds: You can't really ban IPs on mobile networks because there is a very loose association between a handset/user and the IP address. This seems to work similar to old modem pools in the dial-up days. The IP you block for abuse is just as likely to prevent some very nice people from accessing your site.

I know that this can also be a problem on other broadband connections, but my personal experience tells me that it is less so. I moved to my current location over 18 months ago and I have had the same IP address during all that time. Prior to that I had the same IP address for 6 years in spite of network build-outs. I have never paid for a static address, it just seemsthat the leases are long enough that, even when I go away for 4 or 5 days with all my equipment turned off, I get the same IP when I turn everything back on.

My real point to all this is rethinking how effective automatic blocking could be. If you could block a username or email address, you might have half a shot (well, not really since this is often forged or stolen), but blocking IPs automatically seems like it might well be counter-productive for many of the same reasons Fletch mentioned in another post.

What made me take notice of this is a very large influx of mobile traffic over the past few weeks. I get some IPs that are identified as spammers but I would not be locking out a handset but an IP address that gets re-used as needs be.

If someone could assure me that this is not the case, thenI would likely continue my sponsorship of the feature. Otherwise,it just seems like a bad idea for a world that is increasingly connected over mobile devices.

Bob

Innocent IP's get blocked all the time and not just on mobile devices but let me tell you the trade off is worth it when the spammers start targeting your site. Innocent people who are interested enough in your site that were blocked by someone else' misuse can always contact you to have their IP's white listed again. I have had this happen on several sites I run. It is no biggie but the piece of mind is!

BobS said

You can't really ban IPs on mobile networks because there is a very loose association between a handset/user and the IP address.

I suspect that the association is a lot tighter then you might think.

While I said that "the IP addresses will be session based at a minimum", it more likely then not that it will be kept alive and linked to your account a lot longer (No doubt shorter then a fixed line account, but not totally disposable). There are administrative overheads (both network and account related) with allocating/releasing IP addresses so they would want to minimise those.

This seems to work similar to old modem pools in the dial-up days.

On the face of it yes, but I'm sure IP address management strategies have changed much since.

I know that this can also be a problem on other broadband connections…even when I go away for 4 or 5 days with all my equipment turned off, I get the same IP when I turn everything back on.

And with mobiles the session/connection can still be active even if you are not *using* the internet.

Mobile OS' and/or apps do all sorts things in the background when you are *not using* the internet actively, so it can easily be connected for days/weeks on end.

My real point to all this is rethinking how effective automatic blocking could be…but blocking IPs automatically seems like it might well be counter-productive

Then maybe only auto-block for a definable short period. Maybe say 3 days for first offence (for the given IP address), 7 days for second, and permanent for third.

What made me take notice of this is a very large influx of mobile traffic over the past few weeks. I get some IPs that are identified as spammers but I would not be locking out a handset but an IP address that gets re-used as needs be.

Then you have to ask yourself what is doing more damage to your site, the spamming, or the potential access problems or new/existing users.

Also, one important thing to remember is that just because you see a mobile user, does not actually make them a mobile user. I use my mobile internet most via wi-fi to my fixed line, and not via my mobile carrier.

Duck said

Innocent IP's get blocked all the time and not just on mobile devices but let me tell you the trade off is worth it when the spammers start targeting your site. Innocent people who are interested enough in your site that were blocked by someone else' misuse can always contact you to have their IP's white listed again. I have had this happen on several sites I run. It is no biggie but the piece of mind is!

Good point

temp1024 said

Also, one important thing to remember is that just because you see a mobile user, does not actually make them a mobile user. I use my mobile internet most via wi-fi to my fixed line, and not via my mobile carrier.

I meant to mention that in my post. I am making the distinction based on provider - that is., MetroPCS and Verizon Wireless are definitely cell providers whereas an iPhone or Android device on Cox or Comcast is using wi-fi. I'd have no reservations about blocking a mobile device based if they are coming from a fixed location.

I like the idea of the progressively long bans, even for fixed IPs. I wonder how much that would add to the cost.

Bob

BobS said

I am making the distinction based on provider - that is., MetroPCS and Verizon Wireless are definitely cell providers whereas an iPhone or Android device on Cox or Comcast is using wi-fi.

But how reliably are you getting carrier info?

I would suspect not much at all and would probably be part of their design to begin with. If not I would be happy to modify what they build to allow those options as it is really minor work in comparison to building the the rest of it from scratch.

temp1024 said

But how reliably are you getting carrier info?

It's obviously hard to say but when the provider is Metro OCS, Sprint PCS or Verizon Wireless, I feel fairly confident that the information is correct (this is based on the information from Infosniper) since it is pulling that information from the ARIn records. On the other hand, ATT does not seem to have segmented their IP block assignments in the same way. I also see other wireless carriers like Vodaphone, Orange and I think I've seen 3.

However, using the progressive-time ban would actually make it very easy to have the same behavior for both landline and cellular connections.

Bob

Hi guys,

It's been my observation that spammers use toss away IP numbers to spam with because those numbers are the most likely to get banned, and as a result only temporarily slow down the spamming.

I had the opportunity to sit and monitor activity (registrations, logins, etc.) on a busy forum over about a 6 month period and was able to manually capture IP numbers being used during the registration process. It was very boring, but insightful.

In my observations I've notice as many as three different IP numbers being used to create an account and start spamming: one number to register with, another to activate the account, and a third to do the spamming. The first two are much more difficult to detect and thus less likely to be blocked.

When I started noticing the same IP numbers (about 5 or 6) being used to create multiple "sleeper" accounts and I banned those numbers, the spam started to drop also. I believe I was able to cut the head off the snake.

We need a way to tie the spamming IP number with the IP's used during account creation and activation to know which IP number to actually block or ban. On an very active site, seeing a high number of account creations (not confused with account usage or activations) from the same IP could indicate a spammer is busy creating "sleeper" accounts, allowing them to rest for a period of time before being activated (from a different IP). Even after activating them, spammers will allow the account to rest again until they feel its safe to use it to spam (either sig links or posts).

If you have a busy site with many unverified accounts, you either have sleeping spam accounts or high number of guests that were not all that interested in completing the registration process. On my site after 5 days of not being verified I delete those accounts.

Once I'm able to be in a position to help sponsor this, one thing I definitely want added is the ability to capture the IP's used during registering, activation, and logging in. Another is the ability to limit the amount of time for an unverified registration and to automatically delete it after said time period.

Sorry for the long post.

sholzy-

Those are some good observations which would make this a much more complicated change.

The ability to automatically delete unverified accounts would be fairly straight-forward and it is not unusual that account verifications are limited to a few days, so I think that would be good to include.

How the rest would be implemented is what would interest me. temp's suggestion could at least slow them down a bit. Spammers tend to go the easy route so that helps but I think it is also important to check against a spammer database.

Ideally, it would be good if you could reprocess the list against the spammer database since, as suggested, spammers use throwaway IPs and what is worth banning today may not be next year (this assumes that IPs do not live on the spammer database indefinitely).

I'd like to here what Chris had in mind when he made the 3 hour quote. It sounds to me like we may be increasing the work (and cost) to get what we are after.

Bob

BobS said

sholzy-

Those are some good observations which would make this a much more complicated change.

The ability to automatically delete unverified accounts would be fairly straight-forward and it is not unusual that account verifications are limited to a few days, so I think that would be good to include.

This would be a separate addition somewhere else, not the spammer database check.

BobS said

How the rest would be implemented is what would interest me. temp's suggestion could at least slow them down a bit. Spammers tend to go the easy route so that helps but I think it is also important to check against a spammer database.

I think I would have to disagree that spammers tend to go the easiest route. I think they go they go to great lengths to protect their "business" and use automated software to help with that. Remember, they're making lots of money or they wouldn't do it. That's why I think a lot of them will use multiple IP address to carry out the registration process. The more layers they can use to isolate them from a spammers database the better they are able to keep their business going.

Take for instance, if a spammer uses an internet account (maybe a hijacked computer) to initiate account registrations all day long by automated software, there is really nothing that would get his IP banned. Next he hands off the list of sites (along with usernames and passwords) he registered at to another member of his team, and a different IP address, to process the activations of each one, again, nothing there to get that IP banned. At this point, neither IP address would be recored by the forum so they would be isolated from getting banned and they could probably carry on all day long without anyone even suspecting them.

Once that is completed the list is handed off to someone who has a list of throwaway IP addresses used for the actual spamming. These three people may be the same person. One "kingpin" could keep several underlings busy using this method.

Now, if that same spammer used the same IP do all three steps, then most likely he would need to change IPs everyday (maybe even several times a day), probably not something he wants to waste his time on. (just speculation)

It's better to cut off the snake's head than to cut off tiny pieces from the tail.

I personally wouldn't even auto-block an IP number unless it hit high on the spammer database check and it hit my site several times. I think I would block by email addy or username first. The ability to deny or ban throwaway email address domains (like 10minutemail, mailinator, tempinbox, etc.) would help cut spammer registration a lot also.

I've noticed on my old forum spammers tended to register using email domains in groups. They would use, for example, Yahoo for awhile, then use hotmail, then on to another. I would issue a temporary 30 day registration only ban on the popular email domains like Yahoo or Gmail because most of my legitimate users used those type of email domains to register so I couldn't do a permanent ban on those. A lot of the spammers used email addys hijacked from normal businesses which makes it so much easier to issue a permanent ban.

I also noticed many usernames were a subset of the email address. Some of them even used the same username but placed periods (.) at different places to throw off automatic spam detection.

BobS said

Ideally, it would be good if you could reprocess the list against the spammer database since, as suggested, spammers use throwaway IPs and what is worth banning today may not be next year (this assumes that IPs do not live on the spammer database indefinitely).

Correct. Spammers wear out an IP number after a time and move on to the next one on their list.

BobS said

I'd like to here what Chris had in mind when he made the 3 hour quote. It sounds to me like we may be increasing the work (and cost) to get what we are after.

Bob

The ability to capture the IP numbers during registration and during activation would probably increase the cost a small amount.

Since I moved my site to ocPortal I've really have not had much of a problem with spammers. I kept getting hit from three IP ranges (182.177.*.*, 182.178.*.*, 110.36.*.*) from the Punjab region in Afghanistan. Once I banned those IP ranges about 3 months ago, I've had maybe 5 times a spammer registered and tried to add spam links to their profile. I've disabled the ability for new registrations to modify their profile until they reach the next level.

Well, I m blocking the former Soviet Bloc countries and Asia with the exception of Japan. My blocklist continues to grow as I don't have every CIDR for these countries but the problem addresses are getting lower at the cost of cutting off a huge part of the world's population. I'm fine with this for now.

I haven't had much of an issue with Africa yet but I could ban that without much concern. I can't ban most of South America as the artist's and his paintings are pretty well-knownin many South American countries.

My biggest problem lately has come come addresses in the US which is why a solution is still intriguing to me. I'd like to "cut them off at the head" as this is what I've been doing manually and an automated solution incorporating the best ideas discussed here should be a real benefit. I don't want to restrict new members from o]posting nor do I want to validate their posts. I have way too much other stuff to do an my competition has just come back to life.

Bob

I've noted this in the issue tracker item but thought I would mention it here. If this feature is done, it would be a good idea to incorporate support for Project Honey Pot (www.projecthoneypot.org) both for screening and for feeding banned IPs back to the project so that their database continues to grow.

One advantage of this approach is that ocPortal could determine when the cut-off for bad IPs is, thus allowing for aging of compromised IP addresses. Project Honey Pot currently keeps all known issues "on file" although a huge percentage of those have no activity after a few weeks as spammers and other troublemakers move on to a new IPs.

Bob

Some interesting statistics from Project Honey Pot:
https://www.projecthoneypot.org/statistics.php

Bob

Well, I was looking around the Project Honey Pot site and came across the API to implement their Http:BL service.

From their overview:

For many years email recipients have benefited from the use of various DNSBLs in the fight against spam. Through efficient DNS lookups, mail servers are able to check individual connecting clients against various black lists. This provides mail servers with the ability to decide to how client requests are handled from hosts based on individual black list criteria. Hosts are able to decide to block requests, allow requests, or perform extra spam filtering scrutiny to messages from hosts based on results from black lists lookups.

Http:BL is similar, but is designed for web traffic rather than mail traffic. The data provided through the service allows website administrators to choose what traffic is allowed onto their sites. This document describes how to integrate with and take advantage of the http:BL service.

https://www.projecthoneypot.org/httpbl_api.php

Hopefully this is useful to Chris or to someone who wants to create an add-on to address this issue.

Bob

Chris-

Do you have any thoughts about using Project Honey Pot as the basis for spam/bad actor quarantines? Would using this service with it available API impact the quote at all?

0000290: Spammer database - ocPortal feature tracker

I am particularly interested that PHP would be used for black lists, and that IPs banned in ocPortal are fed back to PHP. Also, it is important that an option exist to age bans to some user-determined criteria (I think PHP returns the most recent activity for an IP) but there should be a function that would recheck PHP to see if existing bans still belong on the list.

Interested in your thoughts.

Bob

(Question noted - when I come back to this I'll look over things and assess where we're at - but it'll take some research as I'm not familiar enough with these service(s))