Earlier this month I was investigating alternative possibilities for my web hosting needs. It would be hard for someone interested in cloud services to not think of Amazon Web Services. Since my own personal "cloud" service uses roughly the same underlying Linux Xen VM technology as AWS I thought it would be interesting to see how using AWS compared. On Friday night, October 2, I decided to do some tests.

I went to AWS, logged in with my retail Amazon account and set up an AWS account. I spawned a "tiny" instance of a single Amazon Linux AMI 2015.09 virtual machine (paravirtualized). I downloaded an SSH key, logged in to the VM with no trouble and checked out the distro. I did a yum update. I installed lighttpd and served a web page just to test that was possible. (Are they really giving out free, world-routable IPv4 addressess? Really? Free? Yes, they are.) Determining this, I shut the VM down and terminated it. All in all, less than two hours in total. Seemed great. AWS seemed like it would work very well for my needs. My expected cost for a year’s worth of highly efficient minimal usage was $0.00.

Monday morning, an email was waiting for me saying my Amazon account was blocked. After some predictable struggles to reset my password, I logged into the control center to find 180 instances (20 in each of 9 regions) of the largest VMs running! Following the instructions in the email, I stopped all of the VMs immediately. (You can imagine what an absurd pain in the ass it is to actually "terminate" 180 VMs with "Termination Lock" enabled.) In case you were wondering, mining bitcoins was undoubtedly how my stolen VMs were put to use.

When I looked at the billing console it said I owed $6700!

aws-bill.png

WTF?

Something horrendously uncool had happened and I needed to know exactly what that was. As someone who takes computer security very seriously, this really was disturbing. I have now spent a lot of time studying the various aspects of security relating to AWS usage. I have done a lot of research and received feedback from a lot of knowledgeable people.

Paradox warning! Note that researching computer security is essential to using computers safely. Note also that researching computer security is fundamentally insecure. While researching computer security topics you will never find a more wretched hive of scum and villainy. We must be cautious.

What follows is a checklist of things to consider when trying to isolate the cause of a situation like this. While this is structured to address a compromise that has already occurred, of course the best time to familiarize yourself with these considerations is before you have the problem.

AWS Compromise Analysis Checklist

  • Password Strength

  • E-mail Phishing

    • Did you respond to dodgy email? Do not be quick to dismiss this. Note that Amazon themselves send out dodgy email solicitations to use AWS; I get genuine unsolicited bulk emails from Amazon with spammy sounding subjects like "Instructions to Redeem Unlimited Cloud Storage".

  • Credentials Compromised "Elsewhere"

    • Do you reuse the same username and password that you use for Amazon anywhere else? That is a terrible idea. Probably worse than you thought. The other party with whom you share this information could have had a data breach.

It seems that someone obtained your personal account and/or financial information elsewhere, and used it to access your Amazon Web Services account.

— From Amazon's first email alerting me to the problem
  • Mishandling API Keys - API keys are cryptographic magic numbers that enable the bearer to control the account using the "application programming interface" instead of the normal web based interactive browser interface.

    • Did you email an API key?

    • Did you use an API key in an unencrypted HTTP connection?

    • Did you leave your API key in the software you develop? This problem seems to generally occur when people leave API keys in software they store publicly on GitHub. This is the programming equivalent to leaving your house key on top of your doormat instead of underneath it. Still, it seems to be quite common, even among otherwise competent professionals. At least Amazon is actively scanning for this now.

  • Identity and Access Management - I am reassured that IAM users/roles are not created by default and security credential settings are sane by default.

    • Did you create an overly permissive "role" or otherwise weaken security through the IAM?

    • Did you use any "security credential" features for your users?

  • AWS Account Access Exotica - AWS Security Best Practices mentions several exotic security features.

    • Did you use "Cross-Account Access" features?

    • Did you use "Identity Broker" features?

  • Environment

    • Is your OS, browser, and third party software (Java, Adobe malware, etc.) up to date?

    • Are you using a phone?

    • Did you set up your computer yourself?

    • Any employer or OEM or ISP provided software or strange modifications?

    • Any strange USB keyboard "adapters", etc.? Are you using a bluetooth keyboard? Got a webcam pointing at your keyboard?

    • Is physical security good? Is the computer completely off limits to untrusted parties?

    • Are you in an unusual network situation (university residence hall, corporate network, etc.)?

    • Are you using wifi, especially public (cafe, hotel, airport, etc.) or unknown open access points? Is someone spoofing your normal AP?

    • Any access to the machine from the outside internet directly or through port forwarding?

    • Are you using a VPN or a proxy to route your internet traffic?

    • Are you using strange browser plugins?

    • Do you have a lot of tabs open? Especially unsavory ones or a watering hole?

    • Do you allow pop-ups? Ads?

    • Did you disable the browser setting "Warn me when web sites try to redirect or reload the page"?

    • Did you "allow" any security exceptions when the browser asked?

  • Malware - While I decry the facile assumption that all computer problems are caused by some nebulous "virus", it is true that malware is a real thing.

    • Do you have risky habits like warez, shareware, pr0n, 1337 hax0r forums, or powerful enemies, etc?

    • Do you use Windows? Especially if you’ve had problems with this kind of thing before. Is your preferred anti-virus magic ritual at least giving the appearance that all is well when correctly deployed?

  • Domain Name System problems

    • Does your DNS come from a reliable provider? A spectacular example of an unreliable provider looks like this.

    • Does your legitimate seeming DNS provider have very shady practices? For example, the DNS server provided to me by Time-Warner Cable’s DHCP reliably hijacks connections to nonexistent domains and sends them to a malware delivery system. This article describes it as a "browser hijacker" and rightly points out, "If you are unfortunately redirected to hijacked websites, Trojans can be automatically dropped on your computer."

    • Anything suspicious in your /etc/hosts or Windows equivalent? (My /etc/hosts has a fake IP for the aforementioned TWC DNS exploit.)

Whew! That’s a lot to consider. It turns out that in order to use AWS safely, understanding all of these things is necessary but unfortunately not sufficient. Since I now know what really happened I can assert a few truths. No mismanagement of the AWS features was required. It also turns out that none of the security weaknesses implied by the listed issues were required for this particular exploit to happen. Imagine the cleanest setup you can think of (something like a brand new iMac or maybe Linux from a boot disk with verified checksums). It turns out that such a pristine system may actually exacerbate the problem!

When I first tried to reconstruct what had happened I thought back to that night and what had transpired… I had the idea to use AWS. I thought about how to get there. Was it aws.com or amazon.com/aws? I looked it up and the first hit reminded me that it was aws.amazon.com. Now very often it is my habit to simply type Ctrl-L and just type the entire URL and hit enter. Thinking I had done that I realized that I would not have typed https://aws.amazon.com. Was this my mistake?

I then found Moxie Marlinspike’s excellent presentation about SSL stripping. This is surely the most impressive discussion of this problem. I felt less foolish when he said this.

"I don’t think anybody types https:// into the URL bar anymore. I think it’s very rare. Just as nobody types http:// into the URL bar anymore."

— 23m30s

I realized that if you connect to a site in an untrusted network (i.e. the internet) with HTTP (not HTTPS) there is no defense. It really is game over. As Mr. Marlinspike says:

"The real fix here is encrypt everything… The fundamental problem is that when you have a secure protocol that depends on an insecure protocol you just attack the insecure protocol. And so the answer is encrypt everything."

— 57m30s

Really, this is an incredible video and worth watching.

I went down this rabbit hole far enough to learn that because of the desperate nature of the problem Firefox actually has very elaborate mechanisms to handle this. If you are a bank, for example, obviously you should never allow unencrypted log ins or even connections. Never. Banks can register their domains with Mozilla’s Strict Transport Security whitelist. Once done, if a customer tries to connect to your bank with HTTP, Firefox will simply upgrade it to HTTPS. Seriously, Firefox keeps a whitelist in its damn source code! I was simultaneously horrified and deeply impressed to learn this. It gives you an idea of the seriousness of the problem.

Ok, is amazon.com on that whitelist? Turns out, no. I dug deeper to find out why. This list is apparently joined by submitting a domain name to this HSTS preload form. However, to do this one must be serious about security because rule #2 for inclusion says that you must:

"2. Redirect all HTTP traffic to HTTPS - i.e. be HTTPS only".

Unfortunately, Amazon (even high stakes aws.amazon.com) has some reason to think this is not a good idea. Indeed, Amazon seems to be training people to make unencrypted connections to them. On *http*://smile.amazon.com/about/ I find "To shop at AmazonSmile simply go to smile.amazon.com from the web browser on your computer or mobile device." There are 6 links to "smile.amazon.com", all HTTP. I found the same problem on the AWS sign in page which said "Visit aws.amazon.com/free for full offer terms." That one wasn’t even linked.

One thing I learned during all this is that you should not assume that the protocol security will be "taken care of for you" in a good way. When visiting very critical sites, you really should type https://host.domain.tld explicitly. This is especially critical the first time your browser ever meets the site (hence why a LiveCD or new install is more problematic). Oh, and don’t mistype it or it’s game over with you being the loser.

What this research did do for me was make me wonder just exactly what I did type for the URL. I don’t know why I didn’t think of it earlier and why no one on the AWS forum (or from Amazon) thought to recommend this, but in hindsight it’s obvious. If something bad happened - check your browser history logs. (Another problem with the ephemeral live CD OS approach to security which I generally endorse.) Normally I think of my overly zealous browser history as a stinking security failure, but I have now seen the light. (This is a good time to thank my professional security analyst friend for this tip and tons of other extremely useful information. He knows who he is and probably doesn’t want his name appearing here, but thanks amigo!)

After some SQLite fiddling to decode the data, one look at the logs told the whole story. It’s hard to remember every keystroke and every reaction from a browser session that transpired many days ago. I remembered what I meant to do and it fit with how I typically do things. I remembered seeing the American Welding Society in the search results and being glad I double checked the URL. I remembered looking at the first link which looked 100% legit. I don’t remember it saying "AD" (AdBlockPlus was active). The URL was clear and told me all I needed to know: "aws.amazon.com". Although I strongly had feelings about wanting to simply type that in as I often do (no "https://" though) apparently that’s not what I did. Memory is a bad witness. Somehow I activated that link. I may have used a Vimperator key binding. The search engine I was using was DuckDuckGo and, thanks to overly helpful JavaScript, simply pressing enter a second time is enough to activate the first hit’s link. I can’t even rule out just using the mouse like a normal person or being stoned from my neighbors' second hand weed smoke. So let’s look at the record.

Here’s how events unfolded. (Breaks added for readability.)

2015-10-03 04:17:37 |https://duckduckgo.com/?q=aws&t=canonical&ia=meanings

2015-10-03 04:17:42 |http://r.search.yahoo.com/cbclk/dWU9QjRGQzVCN0UxMD\
  Y5NDdBMSZ1dD0xNDQzODQ1ODU2MTc5JnVvPTk4NzM5NzMwODcmbHQ9Mg--/RV=2/RE=144\
  3874656/RO=10/RU=http%3a%2f%2f2605029.r.msn.com%2f%3fld%3dd3Vm3sB0wiG-\
  BPN_kzuHq5KjVUCUxERXnnzpfxhcfXM-XmftHzqOthl_BXVHPb9liXZiwq_G7JtPd_QpVv\
  O6yDvTp3R9KKQsKlQrCEpmZdz9Hf4I0-rpieTj7uzBd08Jz1YS5qi0mNFIiWaPBha3YrB2\
  y6ewwgLvHGnTeKOqgr2PXywvbW%26u%3dtopsweb.info%252fatisl.php/RK=0/RS=GE\
  P6Vx.mgSTGiw_yhKoPL853hcA-

2015-10-03 04:17:42 |http://2605029.r.msn.com/?ld=d3Vm3sB0wiG-BPN_kzuHq\
  5KjVUCUxERXnnzpfxhcfXM-XmftHzqOthl_BXVHPb9liXZiwq_G7JtPd_QpVvO6yDvTp3R\
  9KKQsKlQrCEpmZdz9Hf4I0-rpieTj7uzBd08Jz1YS5qi0mNFIiWaPBha3YrB2y6ewwgLvH\
  GnTeKOqgr2PXywvbW&u=topsweb.info%2fatisl.php

2015-10-03 04:17:43 |http://topsweb.info/atisl.php

2015-10-03 04:17:43 |http://tebae.info/accounts/accounts.asp?redirect=\
  aws.amazon.com&showinfo=0&controls=2&rel=0&autoplay=0&\
  amp;fs=0&theme=light&enablejsapi=1

2015-10-03 04:18:17 |https://www.amazon.com/ap/signin?....   (* See below)

Basically this shows that after looking at the search results for "aws" for 5 seconds, I activated the first link which took me on a quick obfuscatory ride through Yahoo and MSN (Bing) search engines. (Were results there also rigged?) Let me stop there to stress that the attackers were in control of the top hit for the search term "aws". I was using DuckDuckGo, but here’s a report of the exact same trick being played on Google. I then was bounced to topsweb.info and then landed at tebae.info. No HTTPS but, of course as we’ve now seen, it doesn’t matter at this point. I timed that it takes me about 14 seconds to type my credentials; I spent another 20 seconds looking at this fake login/registration page and I suspected nothing.

Obviously it’s easy after the fact to criticize my lack of perspicacity in scrutinizing this web page. 20 seconds is actually a long time to look at a log in page. I strongly want to believe that the true URL was not what was shown in my browser’s address bar. I tried to figure out exactly how this supposedly inviolate feature of the browser was tampered with, but I could not replicate it. This paper from some academic security researchers gives plenty of ideas (note especially 1.3.2). Other possibilities involve iframes, HTML5 history manipulation, cross frame scripting (XFS), repositioning what was visible in the address bar to put the correct address in the right place, painting a graphic over the whole thing, etc. If you figure out the details, please let me know! I would like to figure this out because I do not like to think that I let that URL go unnoticed but of course that’s a possibility too.

Speaking of verifying the complete legitimacy of URLs, the Amazon sign in URL which I truncated above is quite interesting. We can presume that the attackers stole my authentication information and then sent me on to the legitimate web site. See how long it takes you to completely verify the full correctness of the AWS log in page’s true legitimate URL which looks like this today (line breaks added).

https://www.amazon.com/ap/signin?openid.assoc_handle=aws&openid.return\
_to=https%3A%2F%2Fsignin.aws.amazon.com%2Foauth%3Fresponse_type%3Dcode\
%26client_id%3Darn%253Aaws%253Aiam%253A%253A015428540659%253Auser%252F\
awssignupportal%26redirect_uri%3Dhttps%253A%252F%252Fportal.aws.amazon\
.com%252Fbilling%252Fsignup%253Fredirect_url%253Dhttps%25253A%25252F%2\
5252Faws.amazon.com%25252Fregistration-confirmation%2526state%253Dhash\
Args%252523%2526isauthcode%253Dtrue%26noAuthCookie%3Dtrue&openid.mode=\
checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&ope\
nid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_s\
elect&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fi\
dentifier_select&action=&disableCorpSignUp=&clientContext=&marketPlace\
Id=&poolName=&authCookies=&pageId=aws.ssop&siteState=registered%2Cen_U\
S&accountStatusPolicy=P1&sso=&openid.pape.preferred_auth_policies=Mult\
ifactorPhysical&openid.pape.max_auth_age=120&openid.ns.pape=http%3A%2F\
%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&server=%2Fap%2Fsignin%3F\
ie%3DUTF8&accountPoolAlias=&forceMobileApp=0&language=en_US&forceMobil\
eLayout=0

Got that? They keep telling me that web interfaces are "easy". How about an easier one? This one is obviously fake, right?

http://aws.asia.qualtrics.com/SE/?SID=SV_0joxRVdBYMHVDWs

I was at least proud of myself for catching one bogus URL that night. But no! That’s actually from a valid email from Amazon with the subject, "Your AWS Account is Ready - Get Started Now" (the link has been confirmed real by AWS support but I dared not click it). What about the sketchy domains awsstatic.com or ssl-images-amazon.com or d36cz9buwru1tt.cloudfront.net? Fake, right? No again. All are essential to the AWS website! Of course you will also need to know all about IDN homographs to play this game sufficiently well.

Got all that figured out? Good because it gets even worse. As I was researching how an attacker could establish a position on a network to execute a man-in-the-middle (MITM) attack, I learned more about BGP problems, specifically BGP IP hijacking. This involves manipulating the routers deep in the internet that control where and how traffic flows. Surely these guys aren’t going to be doing things that exotic and sophisticated, right? Wrong. Here’s a report of BGP hijacking used by bitcoin pirates. And more details. And a nice talk about it (especially at 5min and 14min). Here is a Washington Post article discussing the BGP problem in depth. Is Amazon now relying on my skill at casually spotting forged SSL certificates?

UPDATE 2018-11-30: This BGP hijacking is still an ominous problem. Here’s an excellent paper explaining the problem clearly and highlighting some specific BGP attacks spotted on the internet.

That’s all pretty onerous. It makes one really question the web as a medium for extremely high value transactions. The constant "arms race" problem is nicely described in this great article which illustrates the eternal struggle between security measures and criminals for cases like this. This also mentions multi-factor authentication which was proposed to me many times, including by Amazon. I’m going to save my misgivings about that for another day. If MFA was essential for me to safely use AWS in the most minor way possible, Amazon should not have let me proceed without it. Let’s just say that while generally wholesome, MFA has its own problems too. These problems are related to why "billing alert" settings can only reasonably be considered to apply to legitimate mistakes; if an attacker completely compromises your system, they will simply remove the billing alert. (And intercept the notification emails if they can!)

Resolution

One thing my research discovered was that I was not alone with this problem. It seems pretty common. Here are some other stories about this kind of thing:

And in all of these stories it seems that Amazon is pretty generous about just eating the cost of the problem. I’ve very rarely had problems with merchandise or shipping from their retail business but they’ve always been very classy about making such problems go away. I’m actually quite an Amazon fan. It was an unnerving few weeks but I am pleased to report that yesterday, October 22, I got word from Amazon that they were waiving the charges for this spurious use. All of the customer service people involved did a very good job and were very professional and sympathetic, even when they basically couldn’t really do much for me or tell me anything substantive.

The Real Problem

While I did fret over the details, I had plenty of opportunity to consider the big picture. Why did this happen? I believe the answer is simple economics.

The first question to ask is why only 180 machines? Why not a million? Well, there is, it seems, a limit. An absurd limit. What if I was a legitimate customer who needed a million machines? The point here is that any legitimate customer who truly needs more than $100/day is willing to talk to an Amazon account representative in person to positively assert that extraordinary situation. Heck, they’d at least be ok with a captcha or confirmation email. One could argue that allowing unconstrained and automated spending is "convenient". I now know this to be false. I believe it is actually a bit shoddy and puts innocent people at risk.

A revealing question is why is multi-factor authentication not required for Amazon retail shopping accounts? This event has really made me think about just what the difference is that makes MFA so essential with AWS and yet an obscure practice among Amazon retail customers. I believe the answer to this question is the fundamental problem here. (Actually, Amazon retail and AWS both do use and require MFA by default, but only the retail side does so in a sensible way - the other factor is called email.)

The real problem as I see it is infinite profit potential for criminals. Perhaps there is a limit but judging by the amounts other people have been hit with the cap on what someone can become liable for is truly absurd. When I walk into a grocery store I never fear that a pickpocket will surreptitiously purchase $39k worth of breakfast cereal in my name. I don’t even fear this walking into a Ferrari or yacht dealership. There are sane controls involving real humans at that scale.

Amazon, however, has made a perfect system for people who want to build a machine that automatically steals untraceable assets. The only way this can be discouraged is to cap the potential loot. Until there is a conceivable limit on what return a criminal can expect they will never stop. Any expense required by the criminal to overcome computer security measures will always be justified. Nobody will be safe.

If Amazon does not fundamentally change something, the value of hacked credentials, currently on the order of about $10, must rise. Even people with no dealings with Amazon will find AWS accounts set up in their name. Criminals will start managing your email and more aggressively hack your phone to block alerts and foil MFA. The inexorable economics of computer crime demand it. The only cure, and thankfully it’s an easy one, is to create a reasonable default spending cap for AWS that can only be expanded by a real discussion with a real human account rep. Until this is done, until there is a reason for a criminal not to do whatever it takes to automate AWS abuse, you’re not safe regardless of who you are or what you do.

Important Lessons Learned

  • LiveOS CD security approaches are good but will have more problems with HTTPS and logging.

  • Assume all search engine results are malicious. This is a subset of never trust any links ever.

  • Always explicitly type sensitive URLs when first visiting them. However, do not misspell them and do not cut and paste them (homographs).

  • Always explicitly type https:// when first using sensitive URLs.

  • aws.amazon.com is an absurdly sensitive URL for reasons beyond your control.

UPDATE 2018-03-08

Krebs has a chilling article about homograph attacks. At this point the browser URL bar is not trustworthy at all.

UPDATE 2020-12-11

I love this story about an ex-Google cloud expert using Google Cloud to do some testing and quickly finding himself owing $72,000 — that’s with a simple coding mistake and no hostile attack or flagrant stupidity. The article highlights the same absurd problem I’m critical of by quoting from the official Google Cloud necessary reading:

"Unfortunately, a billing budget does not automatically cap Google Cloud or Google Maps Platform usage/spending, according to the docs."

Indeed. And until there is a safe way to guarantee you don’t spend orders of magnitude more than you planned, these cloud services are too toxic for me. Really the only sane way to use these kinds of services is for corporate purposes where a sales rep has put together a contract that precludes absurd bills like this.