We recommend that you use Amazon EC2 to reboot your instance instead of running the operating system reboot command from your instance. If you use Amazon EC2 to reboot your instance, we perform a hard reboot if the instance does not cleanly shut down within four minutes.
AWS - Assume Weak Stability
With me never shutting up about it, who could forget my Amazon Web Services disaster? If you’ll recall, I was testing AWS to see if I might find it useful and I was hit with a $6700 infestation of Chinese bitcoin miners almost immediately. This was thanks to Amazon creating a pricing system that is pretty stupid for large spenders and evil for regular folks. It also showed us that Amazon is pretty stupid about sending confirmation emails when a customer fires up maximum VMs from China hours after the account is created in California. If you disagree with that, then you must accept that what "maximum VMs" means is stupid - either way too high or too low. It also showed us that, because of Amazon’s exceptional value to criminals, search engines can no longer be trusted to do, well, anything.
Those were all good lessons and I’m glad I learned them in a setting where I was educating myself with a controlled test. Last night, however, I learned an extremely valuable AWS lesson in a production situation. Before getting to that, I need to get a bit defensive for a moment. I feel like with my Amazon problems the answer is pretty clear to most people: this kind of stuff happens to idiots; if it happened to you, you’re an idiot. But this is not always so. For example, although I was catastrophically burned by my experience with AWS when I first tried it, I consider it a success to have found such a failure while evaluating the service for exactly that kind of problem. In the same spirit, although I suffered a catastrophic loss last night with AWS, it wasn’t a completely catastrophic loss. By design. There is no way I would put all my eggs in the AWS basket and that is today’s lesson.
I have a client who has a lot of sensors out in the world and the sensors report their sensings which are very valuable. We primarily have the sensors report back to a local system that we physically control. We thought, wouldn’t it be good if we could have a completely redundant system where the sensors could also report? (I’m told that when this was presented to a group of climate scientists as "in case extreme weather takes out all of Southern California" no one considered that possibility hyperbole.)
That seemed like a good job for AWS. And, in fact, it really has been. For almost no cost I have single-handedly implemented a complete replication of our ability to receive all of this data. If the primary receiver dies, this can be an invaluable fallback.
Moving forward in the story, on Christmas Eve (good timing!) I got an email from Amazon saying, that within a fortnight the EC2 VM I was using was going to be rebooted after enjoying nearly a year’s worth of continuous operation reliably receiving millions of sensor readings. This explains the likely cause though truly competent VM managers should not have this problem. But ok. Fine.
When the time came, I kept an eye on things to restart the listening server because I actually had never really prepared for that. When I first set it up, it was as a pilot test and then it worked so well that we never touched it. So sure, definitely properly establishing things should no longer be neglected. What was odd though was that the reboot time came and went and I saw that the VM did not actually reboot. Ok. I figured that maybe they warned me in case they couldn’t avoid a reboot, but apparently it was unnecessary. Lesson one: when AWS tells you they’re going to reboot your VMs, they may, in fact, not reboot them.
Months later, I got another email warning that they were going to reboot my VM. This time they did! The VM was left running fine with a reset uptime. That was ok, but I was going to need to set things up so that my server software was started automatically on boot. Last night I was navigating that hornet’s nest and I had created scripts that could start and stop the service and I felt like if the machine were rebooted, it would come back and resume service without missing a beat. What did I do next? Relax and congratulate myself on a job well done? Of course not, the last thing I was going to do, I was hoping, was to test it.
sudo shutdown -r now which is the proper Linux way to
reboot. I was logged out of my session. I waited a couple of minutes
and tried to log back in… And I couldn’t. Waited a bit more. Nope.
Getting nervous, I fired up the web console. The console showed I had
no instances running. None even existed. My security group and key
pair were gone. Everything about that VM, had just disappeared. I then
spent the next hour or so recreating this VM from back ups.
So rebooting was stupid of me, right? Well, maybe not. This morning I
fired up a test VM, logged in and did the exact same thing with
shutdown -r. And within a minute it was back up and accessible. I
shutdown -h and the machine still remained in my web
console instances display ready to be reactivated (though in that case
it got a new IP number).
Looking into it I did find some AWS documentation that says this.
Ah ha. They conveniently fail to mention that if you do not follow
this suggestion, your entire setup may get nuked. Lesson two: even if
shutdown command sometimes does work, you should never use it.
I hesitate to recommend removing it outright since who knows if it
gets used by the AWS automagical system? I’ll probably just make an
shutdown to prevent it from ever being used
Finally we come to the important point of this story. Lesson three: Assume Weak Stability. But that’s ok. At the top of my RAID notes, I say this.
The uncertainty is not if your hard drives will fail. The question is when will they fail? The answer is, at the absolute most inconvenient moment.
And so it is with AWS. The trick is to make the most inconvenient moment not that inconvenient.
Like the hard drive manufacturers, everybody is trying very hard to
make things always work, but eventually luck always runs out. This is
why you can never completely rely on a single hard drive or a single
service from AWS. As with RAID, you must diversify solutions to your
critical requirements. For AWS specifically, this means being able to
deploy a new copy of the VM very quickly and very easily. Some people
believe that if you have to log into the VM to fuss with it, you have
failed to make it easy enough to recreate. Obviously all data needs to
be backed up aggressively. An aspect that is subtle yet not at all
surprising is that if you rely on a stable AWS IP number, that will
That's what DNS is for Although it is
not what DNS was designed for, DNS can help provide resiliency to this
While it was extremely unnerving to see my cloud VM disappear in a cloud of smoke, I forgive AWS for that. I understand that clouds sometimes come with storms. This problem is more severe for minor users who can not easily afford to engineer elaborate mechanisms of redundancy.
Despite letting this go as a learning experience, I’m still not a fan of AWS. The predatory billing arrangements and weak security are what still really bother me. I will start recommending AWS when they solve the following two problems.
I want to be certain that I will not be liable for any more than my credit card limit.
I want to get notified by email if malicious adversaries log in from China to consume maximum resources.
Note that Amazon retail shopping intelligently does both. It’s not like Amazon doesn’t know how to do this properly.
What Think Different Will Get You
Don’t ask why I was looking at Simon Tatham’s Linux Coroutines in C Howto, but I found this quite amusing.
Any coding standard which insists on syntactic clarity at the expense of algorithmic clarity should be rewritten. If your employer fires you for using this trick, tell them that repeatedly as the security staff drag you out of the building.
"Ah! Now we see the violence inherent in the system." In real life, supreme executive power derives from a mandate from the masses, which may require some farcical aquatic ceremony.
I just finished "Influence: The Psychology of Persuasion" by Robert Cialdini. First of all, let me share a pro tip about people who promise to help you detect bullshit—they are very often peddling bullshit. (Did you just rightly reflect on whether I was guilty of that? I trust you did.) In general taking advice from people in advertising is a very bad idea. It reminds me of people who marry movie stars and then complain when it falls apart, "It was all a lie." Duh. Movie stars and "compliance practitioners", to use the book’s preferred euphemism, are professional liars. It’s ok to be fooled, but not surprised at being fooled.
This book was recommended by a certain popular blogger who doesn’t need any more attention. This book strangely had the longest queue of anything I’ve ever reserved from the library. It was suggested that themes in this book could help explain Donald Trump’s bizarre lack of complete failure. I’m still not sure about that and this book certainly wasn’t any kind of manual for converting an internet troll into the president. Basically it was more of an insightful collection of cognitive biases that, had I seen it when the book came out in the 1980s, would have been mind-blowing. But now that such things are enumerated and quite well known it’s rather less exciting. Especially for people who do a lot of contemporary reading on these topics.
Still there were some interesting tidbits. One thing he talked about is how marketers manipulate a sense of scarcity. It turns out we’re prepared to make all kinds of bad decisions when we think the choice may be cut off soon. I’m reminded of Amazon’s "Only 3 left in stock! Order now!" My practice has always been to discount such items. My thinking is that product A is rushing me to make a decision and product B isn’t which is a dimension of B’s superiority. Obviously whenever there is time pressure to part with your money, you absolutely should never do it. Believe me, that great deal will come up again no matter what the salesperson says.
Perhaps the most interesting examples in the book for me related to something called "social proof". This is basically a mental shortcut that takes for granted that if a lot of people like you are doing something, it’s probably the right thing to do. There’s nothing wrong with that and I’m sure it has served people well to lighten cognitive loads. The catch here is that it only works with people who seem like you. I think my immigrant background serves me well here. Or perhaps it devastatingly precludes what normal people would consider success. Not having a hometown or a proper nationality, I don’t think of anyone as being like me. I’m a weirdo. Back when computer people were weird, I used to think that computer nerds were like me but I’ve definitely been cast adrift from that group too. Interestingly, when I first decided to become as good at programming as I possibly could, I explicitly used the heuristic of social proof. Since I did not have any peers around me to emulate, I deliberately chose to do things in the style of famous computer scientists and C programmers and take on faith that it would work out. Usually it did (Linux, Vim), although sometimes not so well (Tcl, Metafont), but it explains my old timey comfort with hardware efficiency and absolute control.
The idea of "social proof" seems most noticeable to me when travelling. When you get away from people who look, dress, and speak like you, it is much easier to scrutinize everything and really think through why people are doing what they are doing. Sometimes you can catch them doing utterly ridiculous things because all of their neighbors are doing it too. But just as often you will see how ridiculous some of the things are that you and your neighbors do. I have long said that it would be a far better education to send a young person on a grand world hitch-hiking tour than to a university’s annealing furnace of social conformity. I’m sticking with that.
This book was also an interesting product of its times. There are subtle anachronisms that I realize younger people wouldn’t even understand. His cute metaphor for automatic mental responses is that of playing a tape and he uses the phrase "click whirr" which must have been quite fashionable at the height of the Walkman era. He did have a fascinating example of an MCI (remember them?) marketing campaign that had people rat out their friends and if the friends signed up everyone would get a treat. I was struck by how absolutely similar this MCI "Call Circle" thing was to Facebook’s tactics. This recurrence of predation might lead one to believe that we just never learn, but I feel that contemplating how we are so easily victimized by our psychological blind spots has actually made us a tiny bit wiser. I am optimistic that we are getting better at being the thinking apes.
Mt. Soledad Sunrise
Can You Hear Me Now? Probably Not
Remember when I wrote an article called Mobile Security Is An Oxymoron? If you’re a normal first world person, you would have forgotten it while you read it because apparently that message just is too painful to hear.
Well, hear it again. This time it relates to the Wikileaks cirque du jour. Looking at Bruce Schneier’s newsletter I wondered if I could find anything about the CIA using phones as a vector for infiltration.
The first link was the Wikileaks dump itself (maybe best to avoid that one). But the very next link was this New York Times article whose 6th paragraph suggests someone in the room may have just stepped in a huge pile of elephant dung. (Not that there’s an elephant in the room! There probably isn’t! Don’t panic!)
In one revelation that may especially trouble the tech world if confirmed, WikiLeaks said that the C.I.A. and allied intelligence services have managed to compromise both Apple and Android smartphones, allowing their officers to bypass the encryption on popular services such as Signal, WhatsApp and Telegram. According to WikiLeaks, government hackers can penetrate smartphones and collect "audio and message traffic before encryption is applied."
Of course my reaction to that is disbelief. Not at the revelation that smartphones are fundamentally insecure, which should be obvious to anyone who knows anything about computer security. My disbelief comes from knowing exactly how much this would trouble the tech world if confirmed. (Hint: it was immediately confirmed on first principles in 2007 and only one wingnut seemed to find it troubling.)
No, the tech world doesn’t care. They are in more denial than anyone. Of course ordinary people don’t care about cryptographic esoterica but to know that the tech people have written this off as just a necessary price to be paid for the putative wonders of small computers is extremely troubling. The only argument here seems to be that "Well, everybody, really, billions of people, essentially everybody now has a guy in their attic listening to everything they do." (Don’t scoff.)
But should people who understand computer security accept this? Accept computers into their intimate lives that they do not control? Perhaps between Silicon Valley and the spook agencies, a critical mass of would-be defenders are working for the predators. That’s my only explanation.