It’s been a warm summer evening in ancient Greece…
Actually, it was a cold snowy morning in modern Midwest, when my phone rang a bell of an email arriving. The email (forwarded to me by my colo service) stated literally the following:
A public NTP server on your network, running on IP address XXX.XXX.XXX.XXX, participated in a very large-scale attack against a customer of ours today, generating UDP responses to spoofed “monlist” requests that claimed to be from the attack target.
Last time my websites got hacked occurred years ago, and this particular server of mine was never hacked before (spitting over the left shoulder, knocking the wood, and blessing the fact my cat is red, and not black), so I just logged onto my little colocated server, made sure my NTP service is not running (is not installed on that box at all, to be precise), and thought it might be some sort of mistake. Talked to the colo support, they seemed to be as puzzled as I am, because the complaint was not confirmed the next day, and they decided to just close the ticket.
A week later though, the story repeated itself, but it was much worse this time: I’ve got a call from the colo services informing me that I over-used my monthly traffic quota (which is huge, BTW, and I never used even 10% of it), which makes me in their debt for over $1000. At the same time, the ticket mentioned above was re-opened, because my colo services received another furious email from the same source.
It’s a totally different story how I found my way out of it, but I have to mention that my colo company really did they best to help me with it.
But I still had a hacker attack to investigate and to protect myself against.
The original complaint was about my NTP server, specifically (presumably) its “monlist” option. The NTP attack started to spread around Dec. 2013, and still seems to be around (read here).
A recommended cure for that is to upgrade NTP server to the most recent version, which doesn’t have that “monlist” option.
Well, here is the thing: I don’t have NTP server running at all. I can’t totally rule out the possibility of my Ubuntu account being compromised, but the logs didn’t show anything of this nature.
It probably was a lucky guess which allowed me to fix the problem in no time… I could have been spending 4 hours on it as well. What got compromised is my IPMI account (remote management for Super-Micro computers). I’ve found that I can’t login into it, plain and simple. Luckily, resetting your IPMI can be done with
ipmitool which can be found in standard Ubuntu repository:
modprobe ipmi_devintf ipmitool -I open user set password 2 ADMIN
And if loading just that module is not enough, you can load more related modules which probably were not loaded by default:
modprobe ipmi_msghandler modprobe ipmi_devintf modprobe ipmi_si
and than call ipmitool…
Then I simply had my password reset and disabled IPMI NTP entirely.
I went and checked my colo traffic repots…
Holy smoke! And of course, it wasn’t the IP my box is sitting on. It was my IPMI IP.
Frankly, I never can understand the trend of adding features just for the sake of adding features… Exhibit A: NTP service as part of my remote computer management. There must be some reason network people and system administrators want this feature, but to me it just looks like another potential vulnerability.
BTW, this DDoS attack seemed to be really global:
Here you can read more details about what really happened. I wonder how many computers were hacked around the world…