Notes on recent spammy ads as well as the recent forum upgrade and migration

Users who are viewing this thread

Taylor Love

Taylor Love
Staff member
Administrator + Developer
Messages
108
Reaction score
50
Points
28
Location
Everett WA
Website
www.linkedin.com
Hello everyone. I wanted to take a moment to address some concerns that have been brought up recently about the spammy ads and notifications, and I also want to explain one of the reasons for our recent forum migration and upgrade.

We are committed to providing a secure, safe, and helpful experience to all visitors to the site.

The ads:
Members were reporting some "inappropriate" ads and notifications being sent from the terrylove.com domain. This was never our intention. It was not approved. Neither I nor my father support those types of ads, or using notifications for spam. Notifications should be reserved for giving you updates relevant to content you are actively engaged in (your posts, your inbox, your profile, or when someone replies to you). Also, you can opt out of these notifications at any time, or pick and choose how or IF you want to be notified at all. If you previously blocked notifications due to spam, know that it is once again safe to re-enable notifications from terrylove.com.

Assessing the damage:
Research into the file system revealed that a vulnerability was exploited a long time ago that allowed third party attackers to modify backend code and add additional files to the system. The files I could find included a file manager, some obfuscated instructions, and code that would include JavaScript from third party locations. The JavaScript (at the time of me finding it) was being used to display the hacker's own ad content, and also to abuse the notifications system to send advertisements through the notifications channel. Here's the thing. The site is pretty big. We didn't notice that files had been modified until the attackers began sending ads through the notifications system. Which means... the attackers may have grabbed user passwords. I never found any evidence of the attackers actually doing that, but that doesn't mean they weren't able to do it with the level of access they managed to get. Everyone should be changing their passwords here, and they should also change their passwords on any other sites that share the same password as well. I'm sorry we didn't catch this sooner.
TL;DR - spammy ads, backdoors on server, and possible password privacy breach

What has been done about it:
1. Fresh installs on everything. (New Server, new OS, new user accounts, new databases, new website slots, and new files)
2. All files in the site are now tracked by private version control. Whenever any file is modified, we see it and then choose to approve or reject the change.
3. Went through every single JS and HTML file on the server to hunt for and remove unexpected content.
4. All executable files either removed, or checked

It's difficult to be sure you've removed everything that needs to be removed from a compromised system. Even one missed file could be enough for re-entry. So it's really important to be very thorough and careful. Rather than try to remove affected files and risk leaving one infected file in, I went with a different approach. Fresh installs. Fresh installs everywhere. We got a second server going. Newest operating system, newest versions of software, fresh slate of system users, fresh databases, fresh set of db users. Basically a brand new sandbox incase anything might have broken out of the old sand box (why take any chances at all?). We then removed all backdoor or modified code we could find from the older server (You should never trust that you've found everything. Always assume something was overlooked. Just in case). Then I set up a private version control system to make sure any file changes from now on would have to be approved or rejected. Nothing will get by the filter without at least being noticed. And that's good. It's what we want. No changes except expected changes. Next we went through the files and started moving over static content. Static content is safe because it does not execute anything on the server or on the client. Images, movies, sound files, etc. After this was done, I went through every single JS and HTML file on the server to hunt for and remove unexpected content. Thankfully I was able to use some full text search tools because there is just a massive amount of files to be searched through. I basically looked into any file with a hint of JavaScript or possible external content in it. Believe me, it took some time. Worth it though. Just to be sure. Deleted pretty much any file that could execute on-server after that, so long as it wasn't in the forums directory. After this was done it was finally time to take a look into moving the forums over.

My goodness.
1. Upgrade older forums to be on latest version.
2. Fresh install of forums on new server.
3. Commit fresh install to source control.
4. Paste old (and upgraded) forum over the new forum, and check the diff.
5. Discard any changes that aren't expected or needed.
6. Verify all attachments are in a safe format. (Not executable on server, no php files, just a bunch of data files.)
7. Check old database for any evidence of script tags or tampering.
8. Switch DB

And then that's it. That's what happened. That's what was done to fix it and clean up the server. It feels pretty thorough, but if you see something that might have been missed feel free to send me a PM and I can look into it. Thanks.

What you should do now:
Change your password here, and anywhere else where you might be re-using that same password. Also, if you disabled notifications from terrylove.com because of the incident, go ahead and re-enable notifications again.


Thanks.
Taylor Love
 
Last edited:
Top
Hey, wait a minute.

This is awkward, but...

It looks like you're using an ad blocker. We get it, but (1) terrylove.com can't live without ads, and (2) ad blockers can cause issues with videos and comments. If you'd like to support the site, please allow ads.

If any particular ad is your REASON for blocking ads, please let us know. We might be able to do something about it. Thanks.
I've Disabled AdBlock    No Thanks