My Amazon Piranha Attack

Whew! That’s a lot to consider. It turns out that in order to use AWS safely, understanding all of these things is necessary but unfortunately not sufficient. Since I now know what really happened I can assert a few truths. No mismanagement of the AWS features was required. It also turns out that none of the security weaknesses implied by the listed issues were required for this particular exploit to happen. Imagine the cleanest setup you can think of (something like a brand new iMac or maybe Linux from a boot disk with verified checksums). It turns out that such a pristine system may actually exacerbate the problem!

When I first tried to reconstruct what had happened I thought back to that night and what had transpired… I had the idea to use AWS. I thought about how to get there. Was it aws.com or amazon.com/aws? I looked it up and the first hit reminded me that it was aws.amazon.com. Now very often it is my habit to simply type Ctrl-L and just type the entire URL and hit enter. Thinking I had done that I realized that I would not have typed https://aws.amazon.com. Was this my mistake?

I then found Moxie Marlinspike’s excellent presentation about SSL stripping. This is surely the most impressive discussion of this problem. I felt less foolish when he said this.

I realized that if you connect to a site in an untrusted network (i.e. the internet) with HTTP (not HTTPS) there is no defense. It really is game over. As Mr. Marlinspike says:

Really, this is an incredible video and worth watching.

I went down this rabbit hole far enough to learn that because of the desperate nature of the problem Firefox actually has very elaborate mechanisms to handle this. If you are a bank, for example, obviously you should never allow unencrypted log ins or even connections. Never. Banks can register their domains with Mozilla’s Strict Transport Security whitelist. Once done, if a customer tries to connect to your bank with HTTP, Firefox will simply upgrade it to HTTPS. Seriously, Firefox keeps a whitelist in its damn source code! I was simultaneously horrified and deeply impressed to learn this. It gives you an idea of the seriousness of the problem.

Ok, is amazon.com on that whitelist? Turns out, no. I dug deeper to find out why. This list is apparently joined by submitting a domain name to this HSTS preload form. However, to do this one must be serious about security because rule #2 for inclusion says that you must:

Unfortunately, Amazon (even high stakes aws.amazon.com) has some reason to think this is not a good idea. Indeed, Amazon seems to be training people to make unencrypted connections to them. On *http*://smile.amazon.com/about/ I find "To shop at AmazonSmile simply go to smile.amazon.com from the web browser on your computer or mobile device." There are 6 links to "smile.amazon.com", all HTTP. I found the same problem on the AWS sign in page which said "Visit aws.amazon.com/free for full offer terms." That one wasn’t even linked.

One thing I learned during all this is that you should not assume that the protocol security will be "taken care of for you" in a good way. When visiting very critical sites, you really should type https://host.domain.tld explicitly. This is especially critical the first time your browser ever meets the site (hence why a LiveCD or new install is more problematic). Oh, and don’t mistype it or it’s game over with you being the loser.

What this research did do for me was make me wonder just exactly what I did type for the URL. I don’t know why I didn’t think of it earlier and why no one on the AWS forum (or from Amazon) thought to recommend this, but in hindsight it’s obvious. If something bad happened - check your browser history logs. (Another problem with the ephemeral live CD OS approach to security which I generally endorse.) Normally I think of my overly zealous browser history as a stinking security failure, but I have now seen the light. (This is a good time to thank my professional security analyst friend for this tip and tons of other extremely useful information. He knows who he is and probably doesn’t want his name appearing here, but thanks amigo!)

After some SQLite fiddling to decode the data, one look at the logs told the whole story. It’s hard to remember every keystroke and every reaction from a browser session that transpired many days ago. I remembered what I meant to do and it fit with how I typically do things. I remembered seeing the American Welding Society in the search results and being glad I double checked the URL. I remembered looking at the first link which looked 100% legit. I don’t remember it saying "AD" (AdBlockPlus was active). The URL was clear and told me all I needed to know: "aws.amazon.com". Although I strongly had feelings about wanting to simply type that in as I often do (no "https://" though) apparently that’s not what I did. Memory is a bad witness. Somehow I activated that link. I may have used a Vimperator key binding. The search engine I was using was DuckDuckGo and, thanks to overly helpful JavaScript, simply pressing enter a second time is enough to activate the first hit’s link. I can’t even rule out just using the mouse like a normal person or being stoned from my neighbors' second hand weed smoke. So let’s look at the record.

Here’s how events unfolded. (Breaks added for readability.)

Basically this shows that after looking at the search results for "aws" for 5 seconds, I activated the first link which took me on a quick obfuscatory ride through Yahoo and MSN (Bing) search engines. (Were results there also rigged?) Let me stop there to stress that the attackers were in control of the top hit for the search term "aws". I was using DuckDuckGo, but here’s a report of the exact same trick being played on Google. I then was bounced to topsweb.info and then landed at tebae.info. No HTTPS but, of course as we’ve now seen, it doesn’t matter at this point. I timed that it takes me about 14 seconds to type my credentials; I spent another 20 seconds looking at this fake login/registration page and I suspected nothing.

Obviously it’s easy after the fact to criticize my lack of perspicacity in scrutinizing this web page. 20 seconds is actually a long time to look at a log in page. I strongly want to believe that the true URL was not what was shown in my browser’s address bar. I tried to figure out exactly how this supposedly inviolate feature of the browser was tampered with, but I could not replicate it. This paper from some academic security researchers gives plenty of ideas (note especially 1.3.2). Other possibilities involve iframes, HTML5 history manipulation, cross frame scripting (XFS), repositioning what was visible in the address bar to put the correct address in the right place, painting a graphic over the whole thing, etc. If you figure out the details, please let me know! I would like to figure this out because I do not like to think that I let that URL go unnoticed but of course that’s a possibility too.

Speaking of verifying the complete legitimacy of URLs, the Amazon sign in URL which I truncated above is quite interesting. We can presume that the attackers stole my authentication information and then sent me on to the legitimate web site. See how long it takes you to completely verify the full correctness of the AWS log in page’s true legitimate URL which looks like this today (line breaks added).

Got that? They keep telling me that web interfaces are "easy". How about an easier one? This one is obviously fake, right?

I was at least proud of myself for catching one bogus URL that night. But no! That’s actually from a valid email from Amazon with the subject, "Your AWS Account is Ready - Get Started Now" (the link has been confirmed real by AWS support but I dared not click it). What about the sketchy domains awsstatic.com or ssl-images-amazon.com or d36cz9buwru1tt.cloudfront.net? Fake, right? No again. All are essential to the AWS website! Of course you will also need to know all about IDN homographs to play this game sufficiently well.

Got all that figured out? Good because it gets even worse. As I was researching how an attacker could establish a position on a network to execute a man-in-the-middle (MITM) attack, I learned more about BGP problems, specifically BGP IP hijacking. This involves manipulating the routers deep in the internet that control where and how traffic flows. Surely these guys aren’t going to be doing things that exotic and sophisticated, right? Wrong. Here’s a report of BGP hijacking used by bitcoin pirates. And more details. And a nice talk about it (especially at 5min and 14min). Here is a Washington Post article discussing the BGP problem in depth. Is Amazon now relying on my skill at casually spotting forged SSL certificates?

UPDATE 2018-11-30: This BGP hijacking is still an ominous problem. Here’s an excellent paper explaining the problem clearly and highlighting some specific BGP attacks spotted on the internet.

That’s all pretty onerous. It makes one really question the web as a medium for extremely high value transactions. The constant "arms race" problem is nicely described in this great article which illustrates the eternal struggle between security measures and criminals for cases like this. This also mentions multi-factor authentication which was proposed to me many times, including by Amazon. I’m going to save my misgivings about that for another day. If MFA was essential for me to safely use AWS in the most minor way possible, Amazon should not have let me proceed without it. Let’s just say that while generally wholesome, MFA has its own problems too. These problems are related to why "billing alert" settings can only reasonably be considered to apply to legitimate mistakes; if an attacker completely compromises your system, they will simply remove the billing alert. (And intercept the notification emails if they can!)

One thing my research discovered was that I was not alone with this problem. It seems pretty common. Here are some other stories about this kind of thing:

And in all of these stories it seems that Amazon is pretty generous about just eating the cost of the problem. I’ve very rarely had problems with merchandise or shipping from their retail business but they’ve always been very classy about making such problems go away. I’m actually quite an Amazon fan. It was an unnerving few weeks but I am pleased to report that yesterday, October 22, I got word from Amazon that they were waiving the charges for this spurious use. All of the customer service people involved did a very good job and were very professional and sympathetic, even when they basically couldn’t really do much for me or tell me anything substantive.

While I did fret over the details, I had plenty of opportunity to consider the big picture. Why did this happen? I believe the answer is simple economics.

The first question to ask is why only 180 machines? Why not a million? Well, there is, it seems, a limit. An absurd limit. What if I was a legitimate customer who needed a million machines? The point here is that any legitimate customer who truly needs more than $100/day is willing to talk to an Amazon account representative in person to positively assert that extraordinary situation. Heck, they’d at least be ok with a captcha or confirmation email. One could argue that allowing unconstrained and automated spending is "convenient". I now know this to be false. I believe it is actually a bit shoddy and puts innocent people at risk.

A revealing question is why is multi-factor authentication not required for Amazon retail shopping accounts? This event has really made me think about just what the difference is that makes MFA so essential with AWS and yet an obscure practice among Amazon retail customers. I believe the answer to this question is the fundamental problem here. (Actually, Amazon retail and AWS both do use and require MFA by default, but only the retail side does so in a sensible way - the other factor is called email.)

The real problem as I see it is infinite profit potential for criminals. Perhaps there is a limit but judging by the amounts other people have been hit with the cap on what someone can become liable for is truly absurd. When I walk into a grocery store I never fear that a pickpocket will surreptitiously purchase $39k worth of breakfast cereal in my name. I don’t even fear this walking into a Ferrari or yacht dealership. There are sane controls involving real humans at that scale.

Amazon, however, has made a perfect system for people who want to build a machine that automatically steals untraceable assets. The only way this can be discouraged is to cap the potential loot. Until there is a conceivable limit on what return a criminal can expect they will never stop. Any expense required by the criminal to overcome computer security measures will always be justified. Nobody will be safe.

If Amazon does not fundamentally change something, the value of hacked credentials, currently on the order of about $10, must rise. Even people with no dealings with Amazon will find AWS accounts set up in their name. Criminals will start managing your email and more aggressively hack your phone to block alerts and foil MFA. The inexorable economics of computer crime demand it. The only cure, and thankfully it’s an easy one, is to create a reasonable default spending cap for AWS that can only be expanded by a real discussion with a real human account rep. Until this is done, until there is a reason for a criminal not to do whatever it takes to automate AWS abuse, you’re not safe regardless of who you are or what you do.

UPDATE 2018-03-08

Krebs has a chilling article about homograph attacks. At this point the browser URL bar is not trustworthy at all.

UPDATE 2020-12-11

I love this story about an ex-Google cloud expert using Google Cloud to do some testing and quickly finding himself owing $72,000 — that’s with a simple coding mistake and no hostile attack or flagrant stupidity. The article highlights the same absurd problem I’m critical of by quoting from the official Google Cloud necessary reading:

"Unfortunately, a billing budget does not automatically cap Google Cloud or Google Maps Platform usage/spending, according to the docs."

Indeed. And until there is a safe way to guarantee you don’t spend orders of magnitude more than you planned, these cloud services are too toxic for me. Really the only sane way to use these kinds of services is for corporate purposes where a sales rep has put together a contract that precludes absurd bills like this.