Sucuri, the Hack, and the Lessons Learned

We believe in an open, safe web.

Open means a web without secrets, where communities rule, where nothing is kept hidden, where security vulnerabilities are discovered, fixed and discussed without any fear. Heck, we even believe “Hackers are the internet’s immune system“.

Safe means a web where good hackers discovering vulnerabilities can safely get in touch with companies, disclose exploits and get rewards and credits, after letting companies fix their mistakes in a reasonable amount of time.

Are we sure we are all aiming for this in the WordPress community?

Responsible disclosure, SaaS vs self-hosted

It’s common practice among software security circles to disclose bugs privately with software companies, then get a reward, credit and the possibility to write about it, given a reasonable amount of time to fix it.

You see, it’s all about a reasonable amount of time.

In a typical scenario a “Software as a service” company is able to fix the vulnerability in no more than a week, generally a day, and after that let the world know. This works because when the exploit becomes public the software has been fixed definitively, the company has full control over it. The process also ensures companies don’t get away by hiding the vulnerability.

We don’t have that control. WordPress is a self-hosted platform running self-hosted plugins. There’s no way to get in touch with users and let them know about a security release. That’s why timing is important.

Sucuri contacted us on June 16th about a vulnerability they discovered in our plugin. We reacted fast and fixed the vulnerability 2 days later and released a new version after a few days.

On releasing MailPoet 2.6.7, exactly the same day, Sucuri wrote a blog post about disclosing the vulnerability, effectively giving no time to users to upgrade their MailPoet version. You probably already know the rest of the story.

Is less than 24 hours a “reasonable amount of time” in the self-hosted software world to let most users upgrade their plugin before being flooded by the always hungry mass of WordPress hackers?

By waiting a few days, or maybe even weeks, before posting a detailed technical disclosure, 70% of MailPoet users would have been on the latest MailPoet version, Sucuri would have had their credit and MailPoet its shame. Users would have been protected since day one after the exploit.

Of course this means fewer hacked websites.

Are we sure we are all aiming for an open, safe web in the WordPress community?

DISCUSSION

    Author’s gravatar

    Frankly, I’m happy for ANY security alarm that tells me when my site, or those I maintain is exposed.

    Yes, as the creator of the software, I can understand why you don’t appreciate being exposed before you were ready, but the “Hundreds of sites” that needed painful cleanup would have appreciated the warning even earlier.

    Lastly, things that are countable, and web sites are, take the adjective “fewer” rather than “less.” Fewer people, not less people. Fewer websites, not less websites.

    Author’s gravatar

    I agree with your last comment, Bob.

    Author’s gravatar

    Hi Bob,

    I completely agree with you, users should be warned immediately. We sent a security email immediately after the release, and users received the upgrade notice in the WordPress control panel immediately.

    There’s a difference between warning users and disclosing a 0 day vulnerability to the entire world on the same day of the bugfix release.

    There’s isn’t a single case of exploiting this vulnerability in the wild before the Sucuri blog post, here’s why we think that “warning earlier”, like you wrote, would have been better.

    Fixed “fewer/less”, thanks for that!

    Author’s gravatar

    I love MailPoet, it’s a impressive plugin and service, and I love Sucuri too, they,re my favorite security service for my WP stuff.
    Nothing is perfect, and while all the infections are sad I don’t see a massive bad feeling towards MailPoet or Sucuri – it’s curious that many people that don’t use MailPoet seems to be infected, so the flaw must be common to other plugins?
    I agree the Sucuri’s timing in the flaw announcement was very short, Sucuri must rethink this to the next cases.
    Credit for both, no shame at all.

    Author’s gravatar

    It was a question that I had in mind… I do not understand why these guys involved in security have quickly published a way to hack a website ?….
    It is like they have some interest in their business…

    Author’s gravatar

    They sell website cleanup services, so it kinda make sense for their business to publish findings so quickly. More sites hacked more money for them to make.

    Author’s gravatar

    Probably, Sucuri gained many more clients announcing the details so early…

    Author’s gravatar

    Anybody knows of a tool that can clean the sites/files/databases etc ?

    Author’s gravatar

    How can you protect yourself if you do not know you are vulnerable? It is VERY important that you know ASAP, unless the vendor agrees to clean and rebuild your site for you… ;)

    Author’s gravatar

    Hi Coleen,

    We completely agree, users should know as soon as possible.

    You, as a MailPoet user, have been notified immediately within the WordPress plugins panel and by our security email.

    The problem here is letting users know as soon as possible and then, when a good amount of users upgraded, let the world know.

    It’s about giving just enough time to most of the users to protect themselves before being attacked.

    Author’s gravatar

    My host shut-down my site, and I have version 2.6.8, which is supposedly the “safe/patched” version. According to the host, all of my files have been corrupted, and I suspect that my database was also infected. Basically, my site is a total loss due to the simple fact that MailPoet didn’t bother to test their code before releasing. I can rebuild, but I’m not looking forward to it. Thanks a whole f’ing lot.

    Author’s gravatar

    Hi

    This is Tony, CEO at Sucuri. My response was too long to post here, so I decided to share it on our blog: http://blog.sucuri.net/2014/07/responsible-disclosure-sucuri-open-letter-to-mailpoet-and-future-disclosures.html

    All the best,

    Tony

    Author’s gravatar

    Hi Tony,

    I know Daniel, I know about OSSEC and I personally think he’s a super talented security researcher we should thank for OSSEC.

    I’m at MailPoet as a software engineer and system administrator on the SaaS projects we run and I’m involved in security researching, enjoying it as you do.

    I also agree with you that this discussion can’t be completely understood if we don’t know about the history of bug disclosures and how the community arrived to the “Responsible Disclosure” concept.

    What we were really trying to say is: Are a few days of “upgrade window” really that bad?

    Let me quote the Mozilla security policy here:
    https://www.mozilla.org/en-US/about/governance/policies/security-group/bugs/

    “Before making a security bug world-readable, please provide a few days notice to the Mozilla security bug group by sending email to the private security bug group mailing list.”

    “Please try to be understanding and accommodating if a Mozilla distributor has a legitimate need to keep a bug in the security-sensitive category for some reasonable additional time period, e.g., to get a new release distributed to users.”

    The process here is that Mozilla, for example, get notified about a vulnerability, fixes a bug on Firefox, sends out a new version and after a few days, when they see a good amount of users upgraded, makes the vulnerability public.

    Users are immediately notified about the upgrade, this is absolutely not about keeping vulnerabilities hidden, and it has nothing to do with brand.

    Do you think we can agree on something like this?

    Author’s gravatar

    I believe Sucuri acted responsibly and ethically in this situation. They did not discuss the vulnerability until it had been patched, and even then withheld full exploit details. They are not responsible for the actions of malicious actors in a situation in which they ethically brought information to you in an effort to protect your users. Raising awareness about a patched, critical vulnerability as soon as possible is the right thing to do.

    Author’s gravatar

    Hi Michael,

    We all agree here that raising awareness about a critical vulnerability as soon as possible is the right thing to do.

    But what does “as soon as possible” mean?

    Is it really that bad to give users an “upgrade window” of a few days where they would be warned to upgrade immediately?

    Here are a few lines from the Mozilla disclosure policy:

    https://www.mozilla.org/en-US/about/governance/policies/security-group/bugs/

    ” Before making a security bug world-readable, please provide a few days notice to the Mozilla security bug group by sending email to the private security bug group mailing list.

    Please try to be understanding and accommodating if a Mozilla distributor has a legitimate need to keep a bug in the security-sensitive category for some reasonable additional time period, e.g., to get a new release distributed to users. “

    Author’s gravatar

    “Is less than 24 hours a “reasonable amount of time” in the self-hosted software world to let most users upgrade their plugin before being flooded by the always hungry mass of WordPress hackers?”

    Yes it is. Because hackers are diffing changes anyway, and know about any vulnerability within the hour after a patch is released. Even windows updates – which are binary and thus very hard to read – are dissected by hackers to find vulnerabilities that are not documented anywhere.

    Author’s gravatar

    Hi Dave,

    Yes, of course, experienced hackers are diffing changes.

    But they are a very low number.

    The majority of WordPress hackers know about the vulnerability from blog posts and then try the exploit, even to just see if it works.

    Are really that bad a few days of “upgrade window”, considering that users are being warned immediately?

    Author’s gravatar

    Instead of complaining about the already well established procedures within the responsible web security arena for disclosures of software vulnerabilities, why not avoid stupid mistakes to begin with?

    Your software vulnerability wasn’t some obscure issue that was hard to avoid. You relied on an authentication verification that could have been better written by a developer who spent 10 minutes reading the WordPress codex. Using the is_admin() function? Seriously?

    Quit complaining, suck it up, and move on. Your clients who know nothing about web development may think little of this, but the web developers among us know that this hack was pathetic, and complaining about this disclosure is a distraction. Spend your time and energy shoring up the security in your software, not whining about being called out on your poor code.

    Author’s gravatar

    This comment finally hits the nail on the head. Take the high road, you will feel better. Quit letting a 3rd party ‘rent space in your head’ and move forward. Don’t let ego or the persistence of a particular point of view repeated ad nauseum take center stage. Keep your side of the street clean, stop blaming others and feel the joy of letting go. Those who appear as ‘enemies’ are often our best friends, as they reveal how we perform under duress. We end up, when time has passed and water has passed under the proverbial bridge, thanking our ‘opponent’ for making us stronger and live to feel the voice and vibe of blame disappear. There will be no need to defend yourself, nor try to convince the world that you are right. Do you want to be right or do you want to be happy? In the end, your customers are more important — find ways to put them first, truly and not as mere ‘marketing’ — and your success will be boundless.

    Author’s gravatar

    Hi Andrew,

    We all agree about the vulnerability.

    I know very well the established procedures, and I also know that there are companies that act within the limits of these established procedures but give a few days of “upgrading time” before disclosing.

    It’s not about being called out, as we said, security researchers are an immune system, they force us to do better.

    It’s just about the time between a security release and a blog post. That’s it.

    Author’s gravatar

    Sucuri never published an exploit script; you should be more upset with metasploit than anyone. They were the first ones that provided a working exploit for this vulnerability as far as I know.

    Sucuri acted responsibly. They let you patch and publish an update, and even then, they did not disclose enough information or a proof of concept to formulate an attack.

    However, using their information, and the metasploit module, I was able to make WAF rules to prevent the majority of my customers from being hacked. Had they waited I would be dealing with many more hacked sites and servers. Without the responsible disclosure, protecting my customers ASAP would have been much harder.

    Author’s gravatar

    Seriously, fixed the issue in 2 days but released the patch after “few days” (15 days) by adding some shiny new features and improving some other???

    and in the changelog: “Fixed security issue reported by Sucuri”.

    Come on guys, you can at lease say that it is a critical bug in your change log. That would be a better warning.

    Author’s gravatar

    I’d agree with mailpoet. Vulnerabilities that are live like this need a measured communication response. Not a blast email to everyone at the start.
    1. Whitehat email release. To those that are fixing
    2. Greyhat email release. To those that need the fix
    3. Blackhat email release. The rest of the world hackers included.

    Vulnerabilities happen. You need to give the patchers the best timeframe and the hackers the worst time frame. WordPress has a whole very large community of users that update plugins every 6 months or so. The securi release only gave me two days to fix everything. That was ok to me. But it also gave the hackers a jumpstart on anyone that was slower. I don’t think that is a good security protocol for the internet.

    Author’s gravatar

    Trying really hard to control my sarcasm reflex here…

    The unfortunate reality is that there is no way to notify the good guys without notifying the bad guys. But the bad guys often keep vulnerabilities and exploits to themselves. This is why it’s important to get the details out to everyone as soon as possible–because as soon as a patch for a security issue has been made available, the bad guys will work on an exploit. And I mean in minutes or hours, not days.

    When the good guys have the relevant information about the vulnerability, they can prioritize the installation of the patch according to their own estimation of risk.

    Author’s gravatar

    Btw, I haven’t used MailPoet before, but now that this is out there, I am more inclined to use it in the future, not less. Now I know that people are looking at the code for security holes. And I know that, whatever the situation, patches are likely to be quick. But even if I had been using it, I would not have been compromised. Security models should assume failures like this and beg the question “what if?”

    It is also nice to see this discourse being civil without people resorting to name calling, even though this is an emotional situation.

    Author’s gravatar

    Security researchers who report security issues responsibly do sometimes give some lead time to allow users to patch. In some cases, a researcher will allow a few weeks, or a month to go by without publishing information on it.

    That is kind of an extra courtesy though, and not something that vendors should count on.

    I’d just be glad that it was disclosed responsibly in the first place to allow a patch to be created, and not just posted by someone to full-disclosure or sold as a 0day etc.

    Author’s gravatar

    Lesson learnt. My website is completely down in 2 days, and the last 3 weeks (4 July – 30 July) is the darkest weeks ever for my website.

    I am glad you have updated your plugin, keep up the good work.

    I also learn that I must create my own backup file, clean backup file, so in case something like this happen again, I am more than ready than now.

    Cheers guys, and keep up your good work.

    Regards,
    Batam Dine Team

    Author’s gravatar

    This is the same crap about that cookie gig a few months ago ~ Blame that “one girl” (forgot her name, & not looking it up). No, blame WordPress. No, blame _________________.Bleh, bleh, bleh (a.k.a. It gets O-L-D!!)
    Look, MailPoet was notified from Sucuri, MailPoet has the revenue, the wherwithall, the capacity, and they have clearly been told – BEFORE THE ACTUAL EXPLOIT WAS DETAILED – how to fix their problem.
    When someone (like Sucuri) hands you the problem and the solution on a silver platter, then waits …patiently, for hours, and hours, and hours, for someone over at MailPoet to push out an update, that doesn’t exactly give MailPoet much room to whine!

    Plugin developers should reach out to the WordPress Core Security Team, as they have the ability to trigger critical plugin auto updates for specific vulnerable versions of a given plugin.

    The update could have gone out in a timely manner, but MailPoet dropped the ball. Plain and simple. Let’s not attempt to put fluffy butterflies, sugar-coating, and get the blame game shenanigans going again for this problem. @Sucuri’s job is security. They did their job. MailPoet didn’t. End of Story.
    -or-
    More simply put:
    If a plugin author or anyone else on the innerwebs simply cannot take responsibility for their own code’s security, then perhaps they should STOP writing code and hire the folks over at Sucuri to write it for them.

    PR tip: Perhaps something along the lines of “Thanks Sucuri. We submitted the patch to the WP Security Team and all is updated.” The “deflect and redirect” approach is NOT earning any brownie points for MailPoet.

    Author’s gravatar

    Hey Brad, I’m not sure that this exists:

    “Plugin developers should reach out to the WordPress Core Security Team, as they have the ability to trigger critical plugin auto updates for specific vulnerable versions of a given plugin.”

    Please let us know if you know something about it, because we don’t.

    Author’s gravatar

    One of the key features with WordPress 2.7 was automatic updates. It also includes the ability for the folks that run WordPress.org to trigger auto updates of plugins, if they’re aware of a critical security vulnerability with a wide enough reach.

    If in doubt, reach out to any of the Project Leads or folks whose pictures you see on the credits screen, or just email security@wordpress.org

    For the Jetpack security update in April, we worked closely with them to arrange a rapid deployment of secured versions, and to the best of my knowledge, not a single site was hacked through our plugin.

    The key is to view this as an ecosystem, and to work with the ecosystem as a whole.

    Author’s gravatar

    What George told you in the previous comments. It’s always a good idea anyway to talk to some of the core devs when stuff like this happens. They’re often very willing to help.

    Author’s gravatar

    You know what would be great? Is if someone came up with a plugin to clean the malicious php code, search the database for hacks, and delete any possible back doors. Because of this whole mess I have lost confidence in WordPress and myself as a web designer.

    If sucuri and mail poet would spend more energy in fixes and less time debating what should’ve been done, there’d be a lot more happy users out there!

    Author’s gravatar

    So you’re basically handed a HUGE security issue and a fix and then want to complain about it? I don’t know, when someone is doing my job for me, I may say Thank you instead.

    You didn’t even properly disclose the update. As a WordPress Administrator for many sites, I have a lot on my plate and unfortunately I don’t get around to all plugin updates as timely as I’d like to. But if I’m aware of a major security issue in a certain plugin, I’m able to prioritize and update immediately. But with how you handled it, how would I be aware of this update? Why didn’t you contact WordPress or hosts? Find a way to spread the word to get this updated faster?

    What I find concerning is that your team didn’t catch this huge security issue. But someone who is skilled enough to catch it, did and even disclosed a fix for you. What it sounds like this is about is pride. You made a mistake, handled it all wrong, and someone who was trying to help you had to disclose it properly and got the word out because you didn’t. You titled this blog post “lessons learned” but not once in the post did you take responsibility for anything you did or the security issue that you created- instead you chose to blame someone who helped you.

    I’m really disappointed in how you’re handling this. I do have several client websites using your service, but will be switching them over immediately…. because it doesn’t even sound like your team learned from this so it won’t happen in the future.

    Author’s gravatar

    Tish: I love you, and I want to bear your children.
    BEST RESPONSE EVER!!!!!!!

    Best lines from Tish:
    “What it sounds like this is about is pride.”
    -then-
    “I’m really disappointed in how you’re handling this.”
    -followed by-
    “switching them over immediately”
    Tish = NAILED IT!!!!

    Author’s gravatar

    Tish, your concerns are quite accurate. I’ll give you more information, so you can make a better judgement of our position.

    This is not about ego, or pride. Security issues happens to everyone, even the best. We’ve had other security patches in the past, and worked with other parties to resolve them.

    In the majority of cases, security issues are fixed discretely, and mostly go unannounced. Depending on the severity, they’ll pop up in a change log, or not. On every occasion, we’ve taken these very seriously.

    We thank Sucuri for having found the issue, and alerted us.

    On this occasion, Sucuri announced the vulnerability on the same day we released the patch. This was the first time this happened to us.

    When we released our fix, we:

    – contacted hosts
    – repository admins
    – we sent 3 emails to our users
    – we used Twitter and Facebook

    But all of that is simply not enough. We can’t contact everyone, because of the nature of the distribution model of free plugins.

    The majority of users are left for themselves. We’re talking about more than half of users who won’t upgrade within a month.

    Thousands of our users got hacked within 2 weeks of the announcement.

    We believe today, in the light of events, that delaying Sucuri’s announcement might have helped our users’ security.

    On a final note, every project has a disclosure policy. Mozilla, Google, Facebook. Each of them address the critical aspect of making a vulnerability public, or not, in due time, often case by case.

    WordPress doesn’t have any recommendations in that regards. This is one of the reason we decided to bring this topic to public debate.

    Thank you for participating!

    Author’s gravatar

    Its true that WordPress doesn’t have any recommendation. But you should have that disclosure policy right!!!!

    At least, you should have asked sucuri to delay the announcement explaining why you think it is important.

    You can’t just blame sucuri for your wrong doing.

    Author’s gravatar

    Kim,

    Thank you so much for you response and additional information.

    Security issues do happen for everyone which is unfortunate. I do appreciate you guys taking security issues seriously and I know that a lot of work goes into maintaining your plugin.

    But I still feel that there is room for improvement when it comes to communication and getting the word out.

    I find it concerning that not security fixes for your plugin are not properly disclosed and go unannounced. I do feel more comfortable having my clients using plugins that do properly alert me to security issues, so that I am able to protect my clients. Maintaining and protecting their websites is my job and when I don’t have enough information to do my job properly, they start to lose faith in my awesome website ninja skills. Which is not good for me and super not good for you when I stop installing your plugin and getting my clients to buy your license. haha ;)

    So while you did take measures to try and alert others, at the end of the day it was not enough.
    I am thankful that I did hear of this from Sucuri when I did. From the way I see it, they helped you get the word out.

    Does it suck that the word got out to jerk hackers who exploited your situation too? Yep. But from where I stand, they were going to find out about it anyways (those jerks always do)… at least this way more administrators heard and were able to stop it from happening to their websites. Sucuri delaying their announcement would’ve only given administrators LESS time to update and get ahead of this. This situation could’ve been MUCH worse.

    What I really would like to see is the conversation shifting from placing the blame on a security company to how your company is changing the way they handle security issues. Like having others audit your code to prevent this from happening again. Better disclosing security issues, etc.

    All the best!
    -Tish

    Author’s gravatar

    Hello. I think I should say something. I am not defending Mailpoet, but simply want to give you a glimpse from my case.

    Before I got this problem, I don’t install Sucuri, I don’t even know Sucuri exist (maybe you will call me “what a noob!”). I rely my security on other plugin.

    When the Mailpoet plugin is available, I have no time to update my plugin until 2 days later. I got Mailpoet email, and I can show it to you if you want it. In 2 days, my website has been infected. Only 2 days, but it was a nightmare.

    Now, because I don’t know Sucuri, I don’t read their blog post, but some hacker did read the blog. Now if only Sucuri delayed their blog up to one week, the hacker maybe didn’t know the security hole, and I still had time to update my website.

    Everything is connected, and as the developer, I think it is normal and humanly to express our discontent about something is not working as our expectation.

    Well, I hope you understand what I talking about, English is not my first language.

    Thanks!
    Aji

    Author’s gravatar

    Hi Aji!

    I’m so sorry to hear your website was hacked. I’ve had that happen to one of my clients once and it was not fun at all.

    I can totally see it from your point of view because I was there at one time.

    But I really appreciate how WordPress.org handles security issues. For example a core update was available this week due to a security issue (be sure to update your core ASAP if you haven’t already)

    As soon as the update was available, they wrote a detailed blog post and got the word out. Could a hacker read the blog post? Yeah… but that risk outweighed the need of getting the word out to administrators to update asap. They didn’t wait a week or a month and I don’t think plugin authors should either.

    I’m not putting down MailPoet- they have a good product one that I felt comfortable recommending to clients at one time. But the way they handle security releases and disclosure of them could use some work. That’s all.

    All the best!
    -Tish

    Author’s gravatar

    Well job folks.
    I appreciate your way to analyze the facts. There is something wrong in this “mors tua vita mea”, we have to deal with, every single day.
    Resist, it’s a great plug-in.
    N.C.

    Author’s gravatar

    I love MailPoet and I’m so glad to have found this wonderful plugin. I think you got the job done and fixed the issue so I’m happy. Keep up the good work and don’t sweat small stuff. You will never be able to please everyone. Just keep doing your best and focus on your goals.

    Author’s gravatar

    This is turning out to be a long thread. Is it time to throw in a Nazi comment yet? :)

    Author’s gravatar

    The problem with this kind of eventualities is that people loose trust and confidence…

    My site was hacked via the mailpoet plugin since march. I´ve only discovered the threat last week when my server was blacklisted for sending spam.

    I had to clean up the whole mess and restore an old backup of my site (all backups after march were compromised : ( ) which I´m still struggling to get it work like before, not to mention that I loose lots of content from march…

    Now, I´m really not sure if I will keep using Mailpoet or look for another provider/solution like mailchimp or aweber…

    I don´t know if I´m paranoid, but I´m scare to install the MailPoet plugin again after what happened. It´s kind of a trauma…

    What if there are still vulnerabilities in this plugin?

    I don´t have trust and confidence with Mailpoet now…and that´s the problem

    Author’s gravatar

    If you step in poop on your way to school you have two choices, go home and arrive late, mess up the class and have everyone think you are an annoyance or finish the day with the stink following you. People will be upset with you either way. But by the end of the day the smell won’t be so bad. Most of the poop will be gone and you can easily clean your shoes when you get home.

    Moral of the story?

    A little bit of poop isn’t going to change anything unless you make a big deal out of it. Carry on, work on creating awesome plugins!

    Author’s gravatar

    Mailpoet is my one of the best plugin for WordPress also sucuri.

Comments are closed.