What's That Noise?! [Ian Kallen's Weblog]
Sunday April 13, 2008
Speaking of Upgrades
UPDATE (2008-04-14):In his posted a responding to the Security Focus alert, Matt Mullenweg noted the wp-pro mailing list as a resource for people who need to find consultants to help maintain their installations. This is great to know.This old server I've been running my stuff on is really long in the tooth and I hate it (the CPU is ancient and RedHat 9 sucks but at the time so many years ago, it was my best option). So I'm migrating to a new host, faster CPU, more RAM, newer OS, new software installs (no more Apache 1.3, g'bye old chum) ... we can rebuild it, we have the technology. I'm not going to post any WordPress updates for a while. If you're one of the folks out there who need help upgrading I know there are folks like The Friendly Webmaster who are available to consult. Unfortunately, I don't know of any WordPress equivalent to Six Apart's Professional Network but I'll be happy to post pointers to you if you're a consultant who can help people out with their WordPress situations. Of course, watch for updates from the WordPress Blog and follow the forums for updates.
Original post follows:
Meantime in Technorati's crawl data, the rate of WordPress site compromises hasn't really changed, there's a ton of WordPress installations that are being taken over. I've also been reading a lot of conflicting data points on the web and in email exchanges. Furthermore, I recently heard from a WordPresser that some of my information is wrong (though specifics were sparse) and I'd like to get whatever clarifications or corrections are necessary. Hopefully I'll hear back, I have no interest in posting inaccurate information; if/when I find out where it's wrong, I'll update here.
So for now, I'd like to thank my friends at ServePath for setting me up for the migration. I'll be working on moving my goods to some shinier digs and forgoing posting any more findings about WordPress for the time being. Peace out.
wordpress blogging security technorati apache linux redhat sixapart
( Apr 13 2008, 10:13:20 PM PDT ) Permalink
Saturday April 12, 2008
Is WordPress the new Outlook?
UPDATE (2008-04-14): Matt Mullenweg has posted a response to the Security Focus alert, he says it's bogus. I agree that a security alert needs to include more specifics about how an exploit is applied. I'm hoping now that either the author of that report steps forward with details or invalidates the whole thing. I'm disowning the post below (yet) but clearly people are talking about and need to reckon with the facts.
Original post follows:
More WordPress security concerns have come to my attention and it reminds me of the days 5 or 10 years ago when every other day seemed to bring a new exploit with Microsoft's IIS web server, Exchange, Internet Explorer or Outlook. I recall having a conversation with an analyst at the time, we concluded that Outlook wasn't just a chunk of swiss cheese security holes, it was a virus platform. I'm starting to arrive at the same conclusion about WordPress, given the procession of security issues that have come to my attention.
This latest one seems to affect all versions of WordPress (2.3.3 and 2.5 users, you're not safe). I'd seen a report about it here, which lead me to an analysis posted few weeks ago. I've seen a number of blogs with those symptoms (though they were older and I'd assumed they'd fallen victim to the XML-RPC exploit). Assuming this is the same issue, Security Focus says all versions are vulnerable (there's a long list of vulnerable versions and an empty space under "Not Vulnerable:", bad news 
). And there's no patch under the "Solutions" tab. Ugh!
My estimation of WordPress is falling through the floor, maybe it's the Microsoft of blogging platforms. If WordPress doesn't respond soon with an aggressive trustworthy blogging response soon, Technorati may have to quarantine indexing all WordPress installations. Sux.
wordpress blogging security technorati spam microsoft outlook
( Apr 12 2008, 04:07:58 PM PDT ) Permalink
Friday April 11, 2008
WordPress Pandemic Chronicles - 2008-04-11
I found this post about 3rbsmag from the other week that provides some details of a particular WordPress attack interesting. Technorati is still seeing a steady flow of hacked blogs showing up in Technorati crawls. The ones that we can identify as symptomatic of the compromise aren't getting their crawls processed. Some bloggers have noticed that upgrading to WordPress 2.5 is an effective way to clear up those crawl obstacles. It seems like the word is getting out there, but there's still hundreds of vulnerable blogs being compromised every day. Some other WordPress blogs that I've noticed that have upgraded in the last few days include
I didn't post stats last night 'cause my macbook got mad at me for having too many Firefox tabs open, it staged a late-night revolt (it crashed) so I just called it a night. To catch things up, here's latest snapshot of the trailing 90 days of WordPress updates handled by Technorati:
| Version | Count (in thousands) | Change |
|---|---|---|
| 2.3.3 | 238 | 0 |
| 2.3.1 | 149 | -3 |
| 2.3.2 | 141 | -3 |
| 2.5 | 105 | +12 |
| 2.2.2 | 76 | 0 |
| 2.2.3 | 71 | +1 |
| 2.0.1 | 59 | 0 |
| 2.1.2 | 34 | -1 |
| 2.2.1 | 35 | 0 |
| 2.2 | 29 | -1 |
By the way, when I'm being good about posting links and dumping browser tabs, you can spot what I'm reading here. If I'm not posting to this blog, I might be posting links there.
wordpress blogging security technorati spam
( Apr 11 2008, 11:12:03 PM PDT ) PermalinkFear, Uncertainty and Disinformation About The WordPress Exploits and Spam
I've seen a few ill-conceived suggestions that the measures we've taken at Technorati to suspend updates of blogs that appear vulnerable are coercive and should be countered. Let's just put this nonsense aside. When the XML-RPC exploits first caught my attention in February (two months ago), I was seeing five or ten, sometimes a few dozen blogs per day popping up on our radar with severely unusual publishing characteristics. I talked to Niall and Matt about it, learned about the hole that 2.3.3 fixed and posted about it on the Technorati blog urging bloggers to Patch or Upgrade Your Wordpress Installation, Now.
- Fear: Being New Doesn't Make WordPress 2.5 More Secure
- This is Dubya-esque illogical FUD. Nobody ever said "new release"=="secure". The thinking there is: Even if there aren't known exploits of 2.5 but there are of the legacy releases, you should still fear the devil you don't know more than the one you do. Which is unabashed crap. In the case of WordPress, "old release"=="insecure" evaluates to true. Period. Hundreds of blogs or more are proving it everyday.
-
- Uncertainty: WordPress 2.5 is "broken"?
-
Thousands of blogs are upgrading everyday without a hitch. If the WordPress developers broke backwards compatibility for your particular plugins and themes, there are reportedly patches for the other major code-lines in WordPress:
From what we can tell, the patched releases for the 2.0.x and 2.1.x code lines have had statistically insignificant adoption, which is why we're just suggesting that people upgrade. As far as API compatibility goes, this sounds like a problem that needs to be taken to the WordPress community for resolution. Bloggers should weigh the value they're getting from incompatible plugins against the impact of getting hacked.Code Line Patched Release 2.0.x 2.0.11 2.1.x 2.1.3 2.3.x 2.3.3 - Disinformation: Technorati is "dropping" un-upgraded blogs
- We're not "de-listing", "dropping", "disappearing" or anything of the sort. One commenter went so far as to post his own made-up statistics, that we're dropping "85-90% of the blogs published on" WordPress. Totally not the case, the truth is that blogs that are symptomatic will not be updated, they will grow stale in our index until they cease appearing symptomatic. The number of crawls effected are significant but percentage-wise, in the single digits. Taking advice to remove or put misleading generator tags and other "counter-measures" is actually counter-productive. If the suspension evaluation is defeated, and the crawl gets processed, an exploited blog will likely fall into our splog classification systems, mis-flagging it and, in that case, it really will be disappeared. Why do we allow this to happen? Here's a fact that is known to few who don't work on search systems or who aren't spammers: legitimate blogs get disowned and taken over by spammers all of the time. This happens with lapsed domain registrations, deleted blogger blogs (blogger's URLs get recycled), and so forth. Spammers love to get established URLs 'cause they often have page rank and other goodies associated with them. However, once a blog starts publishing spam links, all of the major link processing systems will classify it as a splog, the value of the URL diffuses and degrades; eventually dropping out of searches.
I usually restrain myself from responding to trolls but the impacts we're seeing on the blogosphere are too important to let the fallacies and fear mongering go unchallenged. Don't pay attention to those who are trying to profiteer, making hay about Technorati being "bullies" or trying to "tell people how to blog." That's just outright nonsense. Techorati is not doing anything coercive at all, it's protecting the community by quarantining the infected. Technorati is simply suspending updates on the hundreds of blogs that are popping up as being vulnerable and appearing symptomatic of being hacked. Technorati is a small company seeking to be of service to a very large community. Amidst that community, a lot of bad actors (not the Keanu Reeves kind) are expending considerable effort to hijack the fundamental currency of the real time web: time and attention. We would be remiss if we didn't expend our efforts to thwart them.
wordpress blogging security technorati spam fud
( Apr 11 2008, 10:33:17 AM PDT ) Permalink
Thursday April 10, 2008
Trustworthy Blogging
The WordPress hack pandemic continues. Sampling the data from Technorati's crawler, I'd estimate there are at least 2500 blogs that did not get updated in our index in the last 24 hours due to being compromised. So while Rome is burning, the WordPress developers continue their violin serenade; the WordPress front page and blog still has nothing new posted alerting the vast majority of WordPress users how vulnerable they are. There's a huge, escalating problem for their community but instead the site is just the usual marketing fluff. It's really past time for the WordPress developers to exhibit some leadership. If Bill Gates can get off his butt to prioritize security, you'd think these dudes could. OK, here we are six years later; I never believed the "trustworthy computing" crap from Microsoft but at least they said something. What we're sorely missing from WordPress is trustworthy blogging.
Check your WordPress blogs and check your friend's. If you're not sure how to talk to your friends about it, perhaps these tips on How To Stop a Friend From Driving Impaired might help:
Seriously folks, send them to the WordPress post about the vulnerability.
- Be proactive. Don't wait for them to get around to realizing that they have a problem
- Politely, but firmly, tell them you cannot let them drive home because you care. Direct them to upgrade wordpress quickly (YMMV with those instructions).
- Drive your friend home. Upgrade their blog for them if they're too lame to do it.
- Call a cab. Tell them to shutdown their blog and use Facebook instead.
- Have your friend sleep over. Sex sells.
- Take the keys away. Help them migrate to Movable Type.
- Whatever you do, don't give in. Kick their asses.
read the original list
We at Technorati have discussed resumption of indexing vulnerable WordPress installations but treating all of the links like nofollow links. This might cause more misunderstanding about the issues than we currently have but it's worth consideration.
By the way, Google's Matt Cutts posted a nice write up with some basic security measures WordPress users should take, Three tips to protect your WordPress installation. These steps won't help you if you're WordPress installation is running a vulnerable version but they won't hurt. I disagree with Matt's recommendation to remove the generator tag - rather than removing it, I would recommend advertising that you're using a secure version of WordPress (2.0.11, 2.1.3, 2.3.3 or 2.5).
wordpress blogging security technorati spam
( Apr 10 2008, 02:33:42 PM PDT ) Permalink
Wednesday April 09, 2008
WordPress Pandemic Chronicles - 2008-04-09
I've been acting on the assumption that WordPress 2.3.3 was a "safe" release. I certainly hadn't spotted any hacked blogs using 2.3.3 but poking around, I find these reports of compromised 2.3.3 blogs:
Folks need to keep getting the word out: friends don't let friends run vulnerable installations of WordPress. In the meantime, here's latest snapshot of the trailing 90 days of WordPress updates handled by Technorati:
| Version | Count (in thousands) | Change |
|---|---|---|
| 2.3.3 | 238 | -2 |
| 2.3.1 | 152 | -1 |
| 2.3.2 | 144 | +2 |
| 2.5 | 93 | +7 |
| 2.2.2 | 76 | +1 |
| 2.2.3 | 70 | +3 |
| 2.0.1 | 59 | 0 |
| 2.1.2 | 36 | -1 |
| 2.2.1 | 35 | 0 |
| 2.2 | 30 | -2 |
wordpress blogging security technorati spam
( Apr 09 2008, 11:40:45 PM PDT ) PermalinkStature In The Blogosphere (Or Open Source) Is Not Immunity
When I was comparing notes with Kevin Burton, it looks like we each independently found the same A-lister (who shall remain nameless here) that had fallen victim to the WordPress vulnerability on a secondary blog. I think we each independently had passed a "heads-up", I know I was in touch with this blogger a few times in the last two weeks about it. The blog has since been taken down (the URL redirects to a different blog and that redirect target is not vulnerable). This phenomenon is hitting blogs up and down the blogosphere's power curve -- it's neither the A-listers nor the Z-listers who are targetted. Any old vulnerable WordPress installation will do. And as can be seen in the metrics I've posted recently, the number of potential targets is vast.
Bokardo had fallen into the link-spam hole in Technorati's system because of spam defacement (I've since corrected the flagging, we're indexing Bokardo again). Ironically, the same day that Bokardo posted about being zapped in the Google index, the Google Webmaster Central Blog posted My site's been hacked - now what? which details the process of getting out of their purgatory. Unlike the aforementioned A-lister's silence on the matter, Bokardo author Joshua Porter posted about it, to which I say, "Yay, brother!" His case clearly illustrated the basic point: if you haven't upgraded your vulnerable WordPress installation, you're operating an insecure wiki -- any jackass with the exploit can re-write your pages (and worse). And they will.
Shift gears. I've been participating in online community on The WeLL for almost 14 years (yea, I'm paleolithic but I'm young at heart). One of the central ethical underpinnings on the WeLL is YOYOW: You Own Your Own Words. Other people can't quote/repost your words outside of the system without your permission and you need to be responsible for the things you say. In that spirit, I suggest that quality open source projects should adopt a collective You Own Your Own Code ethic. If you release code for other people to do great things with, mazel tov! But take pride in your products by keeping that usage fulfilling and secure. Where are the WordPress folks in getting the word out about the hack pandemic? Why isn't there a Big Red Banner on wordpress.org alerting people to the hazards of not upgrading? Waxing on about all of the groovy features in v2.5 is fine but really, they should be shouting: URGENT! YOUR INSTALLATION WILL BE HACKED UNLESS YOU UPGRADE TO ONE OF THIS FIXED RELEASES OR APPLY A PATCH. It's not like they don't know, both Kevin and I have talked to WordPress developers and posted very publicly about what's going.
Perhaps if Bokardo or the aforementioned A-lister migrated to Movable Type or some other platform and trumpetted about it, WordPress-land would hear the message. Instead of urging people to upgrade, maybe we should be urging them to migrate.
wordpress kevinburton blogging technorati opensource responsibility movabletype spam security
( Apr 09 2008, 10:37:03 AM PDT ) Permalink
Tuesday April 08, 2008
State of the WordPress Pandemic
I've been conversing with Kevin Burton about the WordPress pandemic. We're in agreement that the WordPress community's response to this security issue has been excessively lax. Most of the feedback I've received about yesterday's crawler changes have been supportive; folks generally want more hygienic social media. Kevin is also implementing a change to block spam-infected blogs from Spinn3r's crawls. We're both going to be keeping tabs on this. I'll be developing metrics on the blogs that Technorati is not indexing when they appear symptomatic so that the efficacy (or not) of yesterday's changes are measured. In the meantime, here's an updated trailing 90 days of WordPress updates:
| Version | Count (in thousands) | Change |
|---|---|---|
| 2.3.3 | 240 | +3 |
| 2.3.1 | 153 | -1 |
| 2.3.2 | 146 | 0 |
| 2.5 | 86 | +8 |
| 2.2.2 | 75 | 0 |
| 2.2.3 | 67 | 0 |
| 2.0.1 | 59 | 0 |
| 2.1.2 | 37 | 0 |
| 2.2.1 | 35 | +3 |
| 2.2 | 32 | +2 |
Some of the feedback that I've heard from bloggers that haven't upgraded is that the upgrade is a big PITA. Some have asked me for referrals for WordPress consultants to help them get their theme and plugin data rolled forward to a newer version. If anybody has suggestions about where to find reputable consultants knowledgeable about WordPress, please blog about it. If you link to this post and you're not using a vulnerable version of WordPress, I'll even find it on Technorati 
wordpress blogging security spinn3r technorati spam
( Apr 08 2008, 11:53:20 PM PDT ) Permalink
Monday April 07, 2008
The WordPress Security Cancer
The blogosphere has had its share of maladies before. Comment spam, trackback spam, splogs and link trading schemes are the colds and flus that we've come to know and groan about. But lately, a cancer has afflicted the ecosystem that has led us at Technorati to take some drastic measures. Thousands of WordPress installations out in the wilds of the web are vulnerable to security compromises, they are being actively exploited and we're not going to index them until they're fixed.
We know about them at Technorati because part of what we do is count links. Compromised blogs have been coming to our attention because they have unusually high outbound links to spam destinations. The blog authors are usually unaware that they've been p0wned because the links are hidden with style attributes to obscure their visibility. Some bloggers only find out when they've been dropped by Google, this WordPress user wrote
My 2.2 installation was being hacked into and spam hidden links dumped into index.php. I didn't notice until google decided to ban me (they have now reincluded my site).
read it
To their credit, the WordPress developers have been fixing the issues. They released v2.3.3 in February and patches for older releases to thwart this exploit. More recently, they released v2.5, which in addition to having the flawed XML-RPC code fixed, boasts a number of new features. But from what I can tell, despite brisk uptake many blogs remain obliviously vulnerable and the occurrence of compromised blogs seems to only be accelerating. As of today, here is the count of blogs running WordPress installs that have pinged Technorati in the last 90 days:
| Version | Count (in thousands) |
|---|---|
| 2.3.3 | 237 |
| 2.3.1 | 154 |
| 2.3.2 | 146 |
| 2.5 | 78 |
| 2.2.2 | 75 |
| 2.2.3 | 67 |
| 2.0.1 | 59 |
| 2.1.2 | 37 |
| 2.2.1 | 35 |
| 2.2 | 30 |
So at Technorati today, I posted that we deployed an update to the crawlers to abort the crawl if the blog appears to have symptoms of being compromised. We'll probably rescind this measure when the number of vulnerable installations in the distribution above looks a little better (some of the false positives I've found are patched but still have unusual metrics associated with the crawl, so they look fishy). However for the time being, these are just creating a lot of noise and instability in our systems and enough is enough. If you're running an old WordPress installation and you're not getting indexed, stop what you're doing and upgrade. Just Do It. The docs on the WordPress site seem to cover what you need to know and the WordPress Forums should help fill in the gaps.
Digging through the lore, it looks like there have been a procession of security problems with WordPress installations:
- wp-forum
- There's the 'WP-Forum Plugin for WordPress "user" SQL Query Injection Vulnerability' advisory from French Security Incident Response Team in January.
- theme distributors
- WordPress theme author Derek Punsalan advised 'Do not download WordPress themes distributed by 3rd party sites' last November.
Using Technorati membership information, I have personally contacted several hundred of bloggers about this issue. These have included blogs with no authority as well as blogs belonging to A-listers. Many have been grateful for the heads up but none (that I have spotted) have posted about this issue. The blogs that are unclaimed are SOL, I don't have any way to reach them (without groping around their site to find a contact email, though I've done a little of that too). Kevin Burton has made a public plea, Anyone Want to Help Fix these Compromised Wordpress Blogs? One blog that did break the silence (Deep Jive Interests) did so in response to tweets about the issue that Kevin's been facing on TailRank.
But is outreach to bloggers going to be enough to stop the spread of this cancer? Probably not. I think the best way to get the word out is to spread the word, tell bloggers you know to post about it. For their part, what I'd really like to see from the WordPress folks (and all blog CMS developers) are
- Automated updates -- I understand that automating upgrades my be problematic when there are database schema changes and such required but installing security patches should be an option in the administrative console
- Security check services -- Bloggers who are uncertain of their blog's vulnerability should be able to authenticate (via OpenID) that they are the author and have their blog sniffed for security holes. OK, this won't work for old versions that don't support OpenID or if, heaven forbid, the OpenID libraries themselves are compromised but I think you get the point. If it can be sanely checked, check it.
wordpress xmlrpc security technorati spam
( Apr 07 2008, 10:23:44 PM PDT ) Permalink
![[Valid RSS]](/images/valid-rss.png "Validate my RSS feed")