The battle against spammers

Those of you who have been on our community for a while might have seen a huge increase in spam posts on the /talk forums over the year. After a bit of thinking I’m assuming we ended up in some black-seo spam software’s database. We’ve apparently solved the issue by integrating support for Akismet and StopForumSpam.com in the forums. So far this has worked great, but I’ll talk more about this below.

This whole thing got me interested into understanding how spam software works, who pays for that kind of stuff, etc… What I discovered after just a few hours of research is, to be honest, a bit scary. If you own a blog or a forum, hopefully this post will be useful for you.

Captcha and anti-bot questions are useless

First of all, it is important to understand that spam software has cracked all known Captcha techniques on most forum platforms. This is particularly true for popular forum systems such as phpBB, and this includes all “are you human” types of questions. The people at phpBB are basically in denial of this, and still recommend to use the Q&A plugin to stop bots from registering, but I’ve seen with experience that this doesn’t work.

There are two types of anti-bot questions: questions that google can answer (“What is the color of the sky?”), and questions it can’t (“type the 4 numbers in this sequence: ab4d56g7s”). phpBB’s recommendation is, of course, to go with the latter, since spam software already queries google to crack the “too easy” questions. But what I’ve learned by visiting some public knowledge websites for wannabee spammers (they call themselves “internet marketers”) is that advanced spamming software defeated most of this already (I won’t name any of those, don’t want to give free advertising to such a tool). My guess currently is that some of that software provides huge databases with answers to the subscription questions for hundreds of thousands of websites (I can’t be sure if this is some collaborative work by several spammers who share the work and benefits of that database, or if they pay for the service, or if it’s a trick involving fake porn websites to get random people to break the questions for you for free,…). And from what I could see with my own experimentation, these lists are probably updated extremely regularly, so that even changing your Q&A question every day doesn’t help if your forum’s become a target.

phpBB default tools are not adapted to efficiently prevent bot registration

The typical tools provided by phpBB to get rid of bot accounts are not adapted to the expertise level of modern spam tools. Ban by email doesn’t work as nowadays it is extremely easy for these tools to register many accounts on sites such as gmail or hotmail. Gmail itself allows you to virtually create as many email addresses as you can by just adding “.” characters wherever you want in your email address. So if you own a forum, you will see lots of spam email addresses looking like “jas.on.wit.ten.ab.c0.0.1@gmail.com”. It is basically impossible to stop that with the default tools in phpBB unless you entirely ban registrations from gmail.com.

Banning usernames is also, obviously, useless, since spammers come up with random ones. Finally, IP ban has proven to be majorly ineffective since the spamming software uses an army of proxies, with many of them being either infected computers, or computers in China and Russia where hosting is cheap and providers don’t seem to worry too much if you’re running some illegal spam business.

The conclusion is, phpBB is perfectly ineffective for stopping spam at the moment. WordPress doesn’t have the same issue (or not at the same scale) because it integrates with Akismet, a database that has a constantly updated list of spam keywords and urls. This blog doesn’t require any form of registration, and yet has close to no spam.

Akismet and StopForumSpam to the rescue

I think phpBB should have Akismet integration by default, but it is not the case so we had to install a MOD to handle this (AntiSpam ACP). This worked great, and started stopping the spam posts, moving them into the moderation queue. But that was also too much work, as nobody wants to manually review hundreds of spam posts per hour and ban the offending bots (we have almost 1 bot registration attempt per minute as I type this).

So in addition to Akismet, we enabled support for StopforumSpam.com at registration (from the same mod). Basically, any user trying to register with an IP, a username, or an email that was recently flagged as a spammer on that collaborative database, will be rejected from our site. This, in combination with Akismet, as far as I can tell, has stopped 99% of the bot traffic on our community, and our moderators can finally start to breathe (I still have to add that MOD on the wagic forums which are in a terrible state right now…). You’ll note that it’s nothing more than what we have with phpBB (IP/username/email bans), except this time, the ban is proactive and relies on a collaborative database, which allows us to automatically ban the bot before it even registers.

Winter is coming

Does it mean the war with spam is over? Absolutely not. First of all, the spam tools I’ve found are constantly evolving to take into account the latest antispam techniques. I really want to say that every single forum owner on this planet should add StopForumspam support to their registration system, but then I also know this would make it the next major target to defeat for spam software.

But don’t be fooled, they are already working on it. On the spammers forums I visited, people are regularly discussing ways to trick services like Akismet or StopForumSpam, simply by getting their IPs and or urls “unflagged” using various tricks. For example one of their techniques for Akismet is to create dozens of bogus wordpress blogs, post their spam comments there, and mark is as “Ham” (false alarm spam) until Akismet moves them away from the spam list.

Spammers also leverage the StopForumSpam website, by adding the list of top StopForumSpam contributors (here) to their own “blacklist” in order to not make it to the spammers database(by never accessing the known StopForumSpam contributor websites). Some of them even use the StopForumSpam list of spammers IPs/emails/usernames as a way to reverse-find unsecure forums where spammers are active, and where their own spam will go through.

In parallel, as I mentioned above, Spam software “companies” are already working on systems that will automate all of this, to guarantee Akismet and StopForumSpam won’t be as effective in the near future.

And even if techniques like StopForumSpam stay effective, even if it reduces the overall amount of spam on your own forum, the money to be made by those techniques is so huge, that it just means the clever spammers will get even richer, with the decrease of competition for them (which somehow doesn’t make me feel good). Some of these people already laugh at systems like StopForumSpam, claiming it’s easy to defeat, and since I can myself see many ways this could happen, I’m inclined to believe them.

But I digress, we solved our current issues with spam on /talk, so enjoy it while it last :)

  1. Jd8531’s avatar

    You forgot to put an Insert more tag :p

    Reply

  2. rafael707’s avatar

    im tired of random post about NFL jerseys or UGS BOOTS, etc…

    Reply

    1. wololo’s avatar

      Well, hopefully now they’re all gone, but don’t hesitate to let the mods know if you find more.

      Reply

      1. rafael707’s avatar

        will do :D

        Reply

    2. dm646’s avatar

      its across all forums sadly..some bots are also selective on where they post..most of the spam on the forum *if you click on my name* is in the psp section of the forum spam still gets into other sections but not as bad as the psp section and its mostly NFL jerseys heh..but also sometimes gibberish, ugs boots, or just a simple hello and good bye

      Reply

      1. dm646’s avatar

        or what really annoys me is when spam spams in a necro topic.. lol

        Reply

  3. e’s avatar

    Please help us putting it in the wagic forums

    Reply

  4. Get rick quick!’s avatar

    Hey guys you wouldnt believe how easy it is to make money buying/selling stock! Using the tools i found on…
    Haha just kidding! Spam is really annoying and if people actually fall for such tricks they deserve to lose their Money.

    Reply

    1. Tupac’s avatar

      Thug Life

      Reply

    2. >_>’s avatar

      You sir, deserve an epic BANHAMMER delivered to your ball sack. <_<

      Reply

  5. psn representative’s avatar

    less crap, more exploits

    Reply

    1. rafael707’s avatar

      nah..

      Reply

  6. Jd8531’s avatar

    Sadly its the internet, someone will always find a way around. Thanks for cleaning the forums though :)

    Reply

  7. Mr. Shizzy’s avatar

    Yes. Spam drives me INSANE!! lol

    Thanks wololo for cleaning up /talk as well as taking the time to post the very informative article. :D

    Reply

    1. wololo’s avatar

      Let’s not forget to thank the moderators, they are the ones who have suffered the most from the spam situation :)

      Reply

  8. zoraktorok’s avatar

    heh, fighting spam is like sony fighting this community… ironic it is, but what can you do? They find a loophole, shut em down… wait for the next opening and do it again. Thankyou for dealing with the headache on both sides wololo!

    Reply

    1. zoraktorok’s avatar

      well, thankyou to all the mods you have inlisted as well.

      Reply

  9. Dutt’s avatar

    Good news and good work wololo.

    Reply

  10. nakano’s avatar

    Thanks wololo, this was an interesting post!

    Reply

  11. svenn’s avatar

    Its not as bad as you think it is, Google is getting smarter, people are getting smarter (arguable). So the target is a lot smaller == less money.

    Its like hackers and “protectors” you can’t 100% protect, but if nobody cares, nobody breaks your console.

    The porn idea, is actually pretty genius, its even cheaper then Indian low-payed jobs.

    Reply

  12. Leires’s avatar

    Now if only they had a way to battle trolls..Your blog is rather full of them, Wololo :x

    Reply

  13. Leires’s avatar

    er..didn’t mean the angry face, meant : x ** as in ‘hushed face’

    Reply

  14. braveheartleo’s avatar

    “Winter is coming.”

    Reminded me of the Game of Thrones TV Series, and is the common phrase of folks from the House of Starks in Winterfell.

    *patiently waiting next year’s season :) *

    Thank you Wololo for sharing this information.

    Reply

    1. Turd Sandwich’s avatar

      It is the Game Of Thrones…

      Reply

    2. hammer’s avatar

      I’m waiting for the next book in the series, but at the rate G RR Martin is going the tv show will catch up to the books

      Reply

  15. Dario’s avatar

    wololo = sony behing the scene

    Reply

  16. gwenavirre’s avatar

    I use http://areyouahuman.com on my site and it seems to greatly deter spambots

    Reply

    1. Deagle275’s avatar

      Are you a spambot?

      Reply

  17. WolfRamiO’s avatar

    you have spammers in your forums because phpbb is like a 15 years old forums software.

    Reply

  18. Thorwak’s avatar

    I manage a mail server for ~1000 users, and I use various well-known anti-spam services block lists as “weights” in mail rules. I can’t take their lists for The One Truth (this creates false positives since clueless admins easily end up in those lists temporarily), but they way in as one of several factors. This catches well over 90%. Still, that means 100-1000 spams that would get through every day in my case, which is of course still a disaster.

    The key I have found is to come up with a couple of your own scripts/rules/factors that take care of the rest. And then, DON’T SHARE THEM! They are easily defeated if studied, and could probably be defeated by analysis (bruteforce trial-and-error), but since a minor target like our mail server is of little interested, this has worked wonders. I’m proud to say currently about 99.98% of the spam is rejected, and AFAIK not a single false positive after years of use. Every now and then I tweak them a tiny bit when something manages to slip by the checks by pure luck several times, but this is really minor work.

    It’s doable, not even that hard, and your users will love it :)

    (In a way it’s “security by obscurity”, but as far as spam goes this is definitely good enough.)

    Reply

  19. Ivo’s avatar

    For remotejoy on psv… Do not try psplinkusb or do … But try psplink and have it setup for nethost … Try it on a psp first to get it going then launch it via vhbl or cef and on pc use nethost and pspsh instead of usbhostfs … Anyway then load remotejoy prx and maybe loadvsh … Have phun.

    Reply

  20. Turd Sandwich’s avatar

    This site is just weak and shitty that’s all. It’s like a cardboard box filled with styrofoam pretty much garbage.

    How about you find a more interesting yet stable website?

    Reply

    1. n1nurt4’s avatar

      I’m sure wololo would love to have your expertise on the matter. Why don’t you send him a pm with your ideas?

      Reply

    2. wololo’s avatar

      Turd Sandwich, I’ve been managing this site by myself for the past 4 years despite having a pretty busy personal and professional life, I started from nothing and 4 years later this site is *the* reference in psp and vita security.

      Think what you want, but I’m pretty sure I don’t need your opinion on what makes an interesting website :)

      Reply

      1. dm646’s avatar

        i happen to think this website and the content is very interesting and helpful :)

        its a lot of work to manage a site a lot of work to code them too heh

        Reply

      2. Turd Sandwhich’s avatar

        I can revamp this whole website into something extraordinary adding awesome features and what not. So whadda say? I been designing websites since the age of 14.

        Reply

        1. wololo’s avatar

          I say being a designer and having useful content are not the same, so you are kinda changing the subject. What type of features do you have in mind?

          Reply

          1. Telgar Drakore’s avatar

            Im not sure if he meant the actual building blocks of the site themselves or the sites contents. O_o.

  21. jd20dog’s avatar

    why do i feel this is the same story when it come to adds and my add blocker???

    Reply

    1. wololo’s avatar

      Yeah, it’s probably a similar situation, although I would say ad networks that try to bypass ad blockers are bad ones… I’m expecting google adsense for example to respect people’s decision when it comes to blocking ads. but I might be wrong

      Reply

  22. Deagle275’s avatar

    New way of finding spambots ;) , just do like this:
    Wrait thou trhird lutter ov Sssprinkel
    Correct Answer : R , because Sprinkle ;)
    And yes, this is just a joke, but perpharps it works ?

    Reply

  23. trecenters’s avatar

    No one besides me bought any asbestos jerseys?

    Reply

    1. wololo’s avatar

      Nah, I’m more into viagra myself

      Reply

      1. dm646’s avatar

        what good is the viagra without the porn spam!

        Reply

  24. Mary Alice’s avatar

    Being a member board mod from RPGamer, I can say that spammers are sneaky little cockroaches; we’ve been fighting a war against them for about 2 years now. Their fav trick now (because they can’t post threads(You have to make so many posts to make a thread if you are new) and most of them are blocked by a board plugin that uses the two things you mention) is to make spam profiles. They put spammy things in sigs and member messages (we use Vbull btw) and in their profiles.

    Luckily it’s only 1 a week now. Only thing we’ve been able to do is clear out the spam and ban. I suggest you keep your eyes open for that kinda bull in case it shows up.

    Reply

  25. ivo’s avatar

    great site btw

    here my runtvtime.sh script i would like to see a php imageslider from
    ./runtvtime.sh
    #!/bin/sh
    echo
    tvtime &

    sleep 4

    echo
    arecord -D hw:1,0 -c 2 -r 32000 -f S16_LE -t wav | aplay – &

    this is on ubuntu
    allso try remotejoy-sdl and watch tvtime pic in pic on as many clients as u wish

    Oh please
    How do i create a php script from an application virtual frame buffer ?

    thanks anyway :)

    Reply

  26. ivo’s avatar

    hello,

    let me rformulate

    how do i export a pc app or any app standard IOs android pc mac linux win osx towards a psp and just one app (being tvtime xor xbmc xor mythtv or yavdr or any ?.? ? or the whole desktop ?

    ifso are there then allso desktop apps for psp ?

    Reply

  27. ivo’s avatar

    i know of one way between colinux or any ubuntu and xming-w32 x server for windows.

    by doing export DISPLAY:=ip:port of xming or any x server
    then launch the app with tvtime&

    Reply

  28. ivo’s avatar

    i mean to ask simply for psp and ps3 … on ubuntu ps3mediaserver tvtime bindings with audio :)

    bye nd thanks

    Reply

  29. Haze7’s avatar

    People in 2nd world and 3rd World countries(also in Ch.ina) actually have jobs solving massive amount of Captcha. They get for as low as $0.25 for an entire week of captcha solving. And in their economy its a huge number. So I don’t see anyway around that except a Captcha solving limit(And if there was I myself would be locked out of that website).

    Reply

Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>