Chris X Edwards

In a strange but fascinating dog and pony show that seemed staged to take the edge off of 2019Q1’s $700 million loss Tesla gave a masterclass on how to do computer science in the style of Tesla. This video was several hours long and astonishingly technical. As someone in the business of autonomous vehicles, I found much of it fascinating. This talk featured a complete computer vision lecture by super genius Andrej Karpathy which reminded me of his computer science lectures.

Let’s remember what I said about Andrej Karpathy two years ago when he was just a lowly Stanford CS professor:

I hope that makes it clear that if Andrej Karpathy says something about my vocation, I listen carefully and take it seriously.

Elon Musk on the other hand… Well, he’s a definitely an interesting guy and a colorful character; this seems to be what made the biggest impresssion. Specifically, Musk literally made headlines by openly mocking his competitors.

Here.

Musk is clearly a smart guy and he has accomplished a lot that I approve of but he is clearly not infallible. Let me go on record right now, for example, and predict that out of the flurry of projections Musk made in the video, not a single one will be completed by the stated schedule. People who keep an eye on Musk will know that it is not especially bold to say such a thing.

It is bold for a CEO to openly deride the technology of competitors. This is especially true when you’re basically on the frontier of technology doing primary research. You can have a hunch your competitors are wrong, but it wouldn’t be research if you were really sure. It’s also weird and risky to disparage people who are clearly very, very smart. Of course I’m weird and risky and I do exactly that all the time so maybe Musk has a worthwhile point to make.

This is my blog so before digging into the technical details, I want to share some personal background that establishes how important this topic is to me. A few years before Andrej Karpathy was busy being born, I owned the first touchpad called the KoalaPad. With very excruciating patience and dexterity, one could almost trace features in a photograph and thereby bring a tiny slice of the visual world into a computer.

This was in the days before digital photography when digital images were lovingly crafted pixel by pixel by hand. A few years later in my early career as an engineer, I had a 12x12 Calcomp tablet which strangely seems to still be for sale exactly as I remember them 25 years ago.

It was common to use these to digitize blueprints. I did not do that because with blueprints you know the ground truth (that is what the document is trying to communicate) so I would routinely resynthesize blue prints as proper accurate CAD models numerically. That made it clear to me what the tablet was good for which was artwork.

As a hobby, I was very interested in fine art sculpture at the time and professionally I was writing 3d contouring software for CNC milling machines. Combining those things, I felt that carving figurative sculpture out of solid steel was an interesting new possibility. The problem, which seems backwards today, was how could one get a fine enough computer model of the subject? I gave this problem a lot of thought. I remember using my Calcomp tablet to outline the profiles of objects taken from many different angled photographs which I then could reconstruct into a 3d model. This technique worked perfectly for convex shapes such as a model sailboat keel bulb I machined. This fails, however, for concave shapes such as eye sockets recessed into a face.

With that background I hope you can appreciate the plausibility of me asking and answering the following question before the advent of consumer digital photography.

Given the information of a 2d image, is it theoretically possible to mathematically reconstruct a 3d model of the scene?

And the answer that I asserted (in 1993) was yes. Today I have been pretty much proven correct but it is important to review why that assertion then was quite unintuitive. At that time, there was absolutely no conceivable technical path to making this transformation so it was reasonable to believe that it might well be impossible. I based my answer on my knowledge of sculpture — if a human sculptor can look at a photograph and create a sculpture from it, then it can not be impossible. I knew that we just merely lacked the imagination to think of a technical mechanism.

Karpathy makes the exact same argument in the presentation when he points out that humans can drive using (predominantly) visual sensory information, therefore it must not be impossible. Today, however, there are conceivable and even practical mechanisms. This miracle of our physiology, sculpting by sight, can now be recreated algorithmically using many techniques. I have been delighted to watch the progress of this technology mature — stereophotogrammetry, mocap, structured light, structure from motion, CNN-based single image depth estimation, etc. Check out Tesla’s brilliant 3d reconstrcution from 2d video in the presentation at 2h19m42s.

My hope is that by understanding my motivation to solve this problem decades ago we can have a useful analogy for the important topic today. Ok, some talented humans can look at a photograph and then sculpt it, so what? Well, let’s ask what an alternative method would be. What if the sculptor had access to the live subject? One technique that suits less talented sculptors (like me) is to get out calipers and a tape measure and simply directly measure all of the important geometry and make sure the sculpted work matches.

The brain of an autonomous car must basically "sculpt" a little model diorama of the world around it and then use that toy to play out different possibilities so that it can confidently choose the best outcome. If the car’s toy model is very inaccurate, decisions it makes will likely be suboptimal. There are two competing theories about how that model should be created. The Tesla approach is to be like a virtuoso sculptor who can look at a scene and sculpt it perfectly. We know that sort of thing is possible, but note that even the most talented sculptors don’t ridicule direct measurement and often use it. The other approach, used by Waymo/Google and pretty much the entire field, is this direct measurement approach (in addition to vision). Some object looks 10 meters away; can we drive 10 meters before hitting it? Tesla will eyeball it and decide. Other players favor extending their measuring sticks and actually explicitly prodding and exploring the intermediate space before driving into it.

That measuring stick is lidar.

Musk did allow that some exotic applications require lidar, for example his own rocket docking adventures. Here’s a photo of me installing a lidar sensor for old-fashioned dock docking (a photo which will give nightmares to anyone who has ever accidentally dropped and lost a small piece of hardware while assembling something).

As someone who professionally studies this sensor, I have some basic knowledge about lidar. Lidar is very much like a measuring stick — you extend it out and when it hits something you check how far the stick is extended and that’s how far away it is. Or you could think of it like how a cane is used by a blind person. How does a blind person know where to direct their single cane to navigate through a complex world? I have no idea but it seems like a very sophisticated skill. Lidars avoid the need for that skill by using a shotgun approach — they send out measurement probes basically in all directions, all the time. The details are obviously more nuanced but that is a reasonable way to think of it.

Here is a visualization of the 3d lidar data, the explicit coordinates found by laser probes, of me sitting on my bike in a room outside my office.

The lidar is inside my office, its walls are at the bottom, and the interesting part of the scene is sensed through the open doorway. This image looks quite rough but remember, this data is not about how things look. It is about knowing how things are. It is about knowing with very high certainty what space is occupied by something that blocks lasers. What this means and what I think the most important point here is this: if the lidar says something is present then do not drive into it! There are some very minor exceptions like puffs of exhaust (which frankly, I’m fine with avoiding out of an abundance of caution) but generally, if the lidar says that something is in the way, you need to avoid hitting it!

And just to give lidar a bit more credit, consider this image mathematically constructed from the lidar’s 3d data points.

That is very helpful for humans who do not quite understand the world in explicit numerical coordinates. What’s remarkable about this image is that it is coded on depth changes which are very likely to be accurate.

Let’s now consider the alternative to a sensor that can reach out and explicitly check if something is in the way. You don’t hold your hands out in front of you when you walk through a doorway in case the door is closed (actually I do this every night in the dark). Tesla imagines the same confidence we have with our eyes being valid with computer analysis of camera feeds. This may be completely feasible and within spec for driving but let’s take a quick look at how computer vision can fail.

Here is a well-known general classifier I was playing with yesterday while testing a Jetson Nano (as described on the Computer Vision Hustler’s excellent website).

The best guess was a "white fox" which is pretty damn good given that "toy sheep" probably is not one of its known categories. But what struck me was the fact that "parachute" was number two. All the other low probability guesses were animals but why a parachute? Then I finally saw it in my mind. I’ve mocked it up so you don’t have to use your warped computer vision imagination.

Of course Tesla’s interesting vision claims include guessing depth values, not just object classifiers. What I’m trying to stress here is that neural networks do pretty well with the known unknowns but sometimes they produce unknown unknowns that are quite surreal. Basically computers have their own optical illusions and in life or death situations this is bad. It’s not an insurmountable problem. Neural net image processing can be really focused and leave much less room for error than human vision processing. However, it is something to be aware of and I think Tesla is.

I’m mostly giving you my personal reflections here, but if you want a good proper analysis of the issues, my always sensible resource for this technology, Brad Templeton, has a perfectly good article discussing the subtle points of lidar applied to autonomous vehicles. And to demonstrate his qualifications, here’s an article on the topic that is still quite useful from 2013.

Arguments Against Tesla’s Strategy

Arguments For Tesla’s Strategy

In summary I am completely aligned with Brad who offers this opinion. "I predict that cameras will always be present, and that their role will increase over time, but the lidars will not go away for a long time." That seems very fair. A lot will depend on just how much improvement lidars see. I can easily imagine a small $50 device that can tell you with 99% certainty high level things like the presence of a dog or motorcycle and their exact coordinates. And for that unknown 1% there would be a 99.99% chance that it is something you really, really should not hit with a car. We’ll see.

I was impressed with the Tesla presentation and strategy. It definitely seems like Musk has a fair amount of hubris, but I guess people who don’t have strong opinions are more comfortable killing each other in their status-quo cars. Tesla’s methodology is very clever and interesting but not necessarily correct. It might just work though. Time will tell.

I wish them the best of luck of course. You can’t say Musk is sitting back doing nothing. He’s definitely got a great team energized to give it their best effort. Although I disagree about the following comment made by Musk (only because consumers' demands are so disappointing) it is righteous enough for me to overlook any Tesla style faux pas. (3h23m50s)

If you can make that happen, Elon, you are my savior.

For a long time now I have been referring to a phenomenon in autonomous vehicle technology which I call The Tesla Paradox. Rather than my overly-complicated explanation, I’ll let Brad Templeton sum it up perfectly in one of his recent Forbes articles.

Consider also Cadillac’s Super Cruise, a pioneer in autonomus ADAS. Very fancy Mercedes was sure that you’d love extra fancy adaptive cruise control which they were first to sell using radar. Here’s a job ad for a "Senior Systems Engineer - Augmented Driving (Autonomy)" at McLaren. Etc.

On the other hand, Ferrari has understandably gone Amish. Perhaps a "Ferrari Paradox"? I.e. the most high performance (advanced?) sports car company refusing to acknowledge important new technology.

And now if you’ll excuse me, I need to continue my work making a fun-to-drive pleasure boat drive by itself.

A long time ago, I used to bicycle to work in cold winter weather. One day I rode out of my driveway and I marveled at how fast I was lying on the ground. Pow. No warning. Intense gravitational pull. That is what ice does to the concept of bicycles. I figured this may be a natural limitation and simply tried to avoid any possibility of encountering ice.

Now that I’m back in a relatively wintry biome, this topic again came up for me. One day I was riding home and I thought to myself, "Hmmm, this looks exactly like the kind of conditions where ic…" Pow. On the ground, I finished the thought, "…where ice is likely." That was me being hyper alert for that exact situation and being as careful as possible. It would seem hopeless to persist.

But there was one last hope — spiked tires! I ordered a set of Schwalbe (the brand) Marathon Winter Plus tires (47-559). They cost about $75 each and that turned out to be a fantastic bargain.

I mounted these tires and did the dry road riding they suggested to seat the spikes. And from there, winter came and I did not stop riding. Although my commute is sensibly short by design, I averaged about five rides per week all winter long. This gave me a chance to ride in all kinds of winter conditions.

I love these tires! You can’t ever ride on extremely treacherous surfaces without taking some care, but these tires radically reclassified what kinds of conditions were even possible to bike in. The black ice or sheet ice that will put you on the floor instantly with no warning on normal tires now can be safely negotiated. The huge difference is that if there is any slipping, with spikes there is some warning, enough feedback to make adjustments. Besides that one crash on the ice before I got these tires, I didn’t go down at all this winter. I didn’t even have any close calls and when I think about what I rode in, that is astonishing.

I found that on smooth frictionless ice surfaces like this flooded/frozen bike path behind my house the spikes were perfectly adequate. The same goes for my driveway with a layer of freezing rain which some mornings I could barely stand on to shovel the subsequent snow. The tires handled that with no problem. Amazing.

The only slipping problems were in deep snow once they got packed and that’s a pretty safe situation if your bike skills are ok. Let’s just say you have a fighting chance.

Switching back to normal tires I was reminded that these spikes were pretty loud. They make a crunchy sound like riding over gravel all the time — I came to like that sound. The handling on dry pavement is better than I expected. Sure they’re a little more squirrelly but still able to do what needs to be done with confidence. They’re heavy, but what else would you expect? They are very well made (in Deutschland) and they even have the reflective strip that should be on all bicycle tires. They also have a bottle dynamo strip to remind one of normal Europeans riding their normal bikes.

Inspecting the carbide spikes today, they look to be in good shape. None came loose or are missing. They are definitely good for another season. And so am I!

Well, here’s one that I wasn’t expecting so soon. Facebook has just cut me off!

Remember a few weeks ago when I posted a cautionary tale about how big important heavily relied upon "services" are quite likely to cut you off in a less than ideal way? That focused mostly on a particular Google example, but I now have a fresh example from Facebook. Just when I was thinking that Facebook couldn’t get any more odious, they have set a new low for me.

I’m well known to disparage this terrible social network but every couple of months I check in and have a look at family members' stuff. Recently, I was getting sick of getting FB email reminders with subjects containing bad unicode glyphs that annoy my text email client. Every day I’m cleaning up my mail queue because Facebook sent me something utterly pointless. (Of course whatever it is they think I’d be so keen to see, they can’t just put in the email which makes their motives clear.) Since the way I handle email is so alien to ordinary humans, here’s a little example of how I see my email so you can get a vague idea of what email is like for me (since 1987).

Don’t feel sorry for me. I’m the one heartbroken at how normal people are brutally abused by their predatory mail user agents. Anyway, if Facebook can send me mail with a MTA (mail transfer agent) called "ZuckMail [version 1.00]", I should be able to use Mutt as my MUA (mail user agent).

Last week I had the idea to log in and unhitch myself from a lot of people I don’t care about who post things I really don’t care about. These are people from as far back as 15 years ago from hobbies I don’t even do anymore. Or colleagues I haven’t heard a peep from in 10 years.

I tried to log in and was confronted with this crazy thing.

Clicking on "Confirm Identity Another Way" takes me to a selection of ways this process can be done. That list has one item, "Collect Confirmation Codes". Hilarious!

One of those people (the one in the kayak I built) is still in contact with me (which you can see evidence of in my inbox) and an actual friend (ironically, I don’t think he’s much of a Facebook user). The other 4 are very long lost — 2 I was going to unfriend! Two live on different continents and I haven’t talked to them in a long time. Two were former colleagues whom I also have completely lost touch with. The chance of me being able to call three of them and ask them to reset my Facebook account is nil. Why couldn’t they pick contacts they know I’m interested in because I only look at their stuff? To pick random distant "friends" is the same as locking me out. Consider also the endless random people I had never heard of in real life which Facebook suggested I connect with.

This just effectively cut me off from that Facebook account after 16 years! Certainly there are worse things that could happen, but this is annoying because the whole point of logging in was to unhook some "friends" and now I’ll have to go make a special rule to have the email address I used for dealings with Facebook go to /dev/null. Seriously, this is why everyone should buy a domain and use custom email addresses for each and every asshat company that forces you to make a log in account. And why no one should use Facebook. And why you should not exclusively trust any company like this with anything important.

And that would be that but today Facebook handed me a bullwhip to beat this dead horse. I got a series of mails reporting that someone suspicious (a Linux user! gasp!) had tried to log into my account last week (spoiler: it was me!) but since nothing untoward had occurred since then, they’ve unlocked it. Yay!

(I don’t know why they had to have "Hi Chris" twice, but if you search the entire message for the letters "php" you will find them.) Anyway, I thought, great, they fixed that problem. I tried to log in. Nope. Same deal. And although I was still locked out as before, just checking prompted this stupid email.

They told me it was unlocked and so I tried to log in. Now they’re warning me that someone tried to log in. Obviously if someone has compromised my email account, well their first message was kind of pointless. Just a mess.

For laughs I looked at the stupid spam trying to clickbait me into irresistible "medieval memes" — so fun! And here’s what I found.

I have been redacting some information to prevent any unfortunate public disclosure and it looks like I blacked out the email’s content, but I swear, that was how it was. It looks incompetent. Normal people shouldn’t have to be experts in multipart content types for email but if you’re one of the biggest tech companies in the world, then ya, you probably need to hire someone who has had a look at RFC1341.

The nonsensical problem (that I see everyday) is that you can send proper email (which is handicap accessible by the way) or dangerous HTML-ized security nightmare email or both at the same time; most companies try to send both in a multipart message but often the plain email part, while present, is blank. It would be like if there was a link on a French website for "English version" which contains nothing. (Don’t get any funny ideas, Quebec!) It’s fine to not have the other format, but don’t say you have it if you do not!

All quite petty, I know. But I have a bit more petty carping. What about this?

Here’s the HTML version of the "email" warning me of the connection attempt. This one had a plain text email just fine but you’ll see I also marked up that it had two independent copies of the message in the HTML part. To be clear, it’s forgiveable if they had the message as a proper email and also repeated it one other time in an HTML message. But twice? Why?

Why care? Well look at that first green box. That was the essential message they were trying to communicate to me. All that other cruft is wasted (and the first green box too really since there was the third copy in the plain text part). Again no big deal if you or I do some stupid email stuff. But this is Facebook. They’re sending untold gazillions of messages which must require some kind of resources.

Maybe it doesn’t matter and roughly optimizing something targeted to billions of humans isn’t an obvious and essential requirement any more. As a curmudgeonly old man (with a degree in industrial engineering) I can’t help but feel this is shoddy work. Facebook’s security arrangements are definitely shoddy work. Let’s just count our blessings that their incompetence has let too few people log in rather than too many. We all need to do what we can to get that number of Facebook logins down to zero!

Back in the Ballmer days I always used to say that the only thing that kept Microsoft, with their absolute control over all human information, from enslaving humanity was their manifest incompetence. With Facebook, that logic is also clearly valid.

UPDATE 2019-05-02

The normal person question I’m likely to get is, "Why don’t you just use your phone?" I’ve already written about this extensively. However, note Facebook’s special evil.

When I first arrived in Buffalo last summer I pointed out to the astonished locals, who inevitably asked why I would move here from San Diego, that as far as I was concerned the problem with both places was that it was too hot. They would then answer their own question with a cry of, "Just wait for winter!" You see, in San Diego these days waiting for winter does little to relieve the "too hot" problem. Here, the weather is far more pleasant. For me.

Although I’m nostalgically keeping the snow poles in my driveway another week or so (and the chance of a decent snow is still pretty good), spring has definitely begun. It is time to review Buffalo’s hallmark season, winter!

By autumn the "too hot" problem had definitely been cured and I was delighted. There was also a bit of apprehension — could it actually become not hot enough? There were ominous signs — the gleaming battlegroup of snowblowers lined up outside of Home Depot, the snow poles being planted everywhere, frost-proof plumbing, gates to close the freeways, a pretty decent amount of autumn snow. In November I saw some guys at UB testing out a snowmobile. All pretty exciting!

Perhaps one of the reasons that I like winter weather more than normal people is that when planning and preparation can easily cure an obvious problem, it’s pretty much a problem I never have. So plan and prepare I did. I bought tons of winter gear, much it of while shopping in northeast Canada. I didn’t just turn off my outside spigots to keep my pipes from freezing — I cut them out and completely removed them for the winter. I had so much fun buying weird stuff like snow poles, crampons, spiked bike tires, and snow shovels (yes, three). I worried that my little car would not be adequate but settled on getting the best snow tires money can buy. Etc.

As I braced for this famous heavy-duty winter, the weather actually turned annoyingly mild. The autumn snow melted and did not seriously make an effort to return throughout December. Just when my wife and I were beginning to think the whole winter thing was a hoax, it did finally show up. I will say that the weather here can be volatile!

Never a dull moment looking out the window! My time lapse video certainly illustrates that. Another example of that is the longer sunrises and sunsets this season. Here is a sunrise from our house.

A lot of those winter cliches are simply normal life here. For example, here is one of those little decorative trees they paint white to simulate snow contrasted with the real thing.

To get a feel for how this season went, I’ll order some photos in what seems to be the wrong order, but which are in fact chronological.

Here’s a flock (rafter?) of turkeys walking past my office window.

Oh ya, serious winter here! Late December and I’m eating lunch outside (though not every day like I did in SD). Apparently authentic Buffalo "culture" would compel me to jump on that plastic table until it breaks. At least they like good music here. I was reminded of that as the holiday ear torture gave way to music with guitars again.

Christmas is so full of fatuous songs about snow, surely there will be snow, right? Not as much as I would have thought. Finally on Christmas day there was a bit of light snow (you can see falling), but my son is standing mostly on what’s left of November’s plowed snow.

After a sluggish start, by the end of January winter finally started to get interesting! Here is Lake LaSalle looking pretty wintry. (Summer version for comparison, camera vector is -1x.) The key point I was noting with this photo is the headwind which the blowing snow makes more visible than usual.

And on the way home from work back through UB it was definitely nice and snowy, but very, very far from impossible to bike in.

I had to get home early because I needed to get my skiing in. When my skis arrived in late January (as I report here), I decided I would ski every day it was possible to do so.

I skied 9 days straight and (barring a prodigious amount of freakish spring snow) that was it for the entire season. Better than nothing to be sure, but I’d love to see a lot more.

Here’s a single turkey walking through my backyard. There is still a lot of wildlife here in the winter.

We get rabbits, squirrels, chipmunks, Canadian geese, and tons of other birds. Here four deer come to visit. I can only find three in the photo, but that’s how those deer are — very good at blending in. I often find their tracks on the driveway when I shovel it.

The only time I enjoy being too hot is if I am literally steaming. That phenomenon occurs only when you burn a ton of calories in a short amount of time and it’s freezing cold. You can’t quite see it in this photo of me cooling off, but I will remember that commute fondly.

Riding through some tough snow on this bike path was fun. The hard part can be simply the rough surface of footprints frozen solid. It’s like very rough cobbles.

Here I was practicing my compulsory figures. The real objective was to try to dislodge the slushy snow stuck in my tires before bringing my bike inside. I finally worked out that I could go inside and get a pitcher of water and pour it on the tires. That was helpful for the salt buildup too. I did accidentally make a dangerous little skating rink one morning. Oops! Sorry! I thought it was mid 30sF but really it was low 20sF. Lesson learned.

Although much of the core of winter was less snowy than I would have liked, even on the last day of March there was enough snow for everything to look nice and snowy. This is the view from my desk at home (cf. the autumn view).

What the Buffalo winter taught me is that I probably should move to Canada or Alaska if I ever get the chance. I really enjoyed the winter as I previously reported. The week of that polar vortex stuff I was really taken by surprise. Not with the weather or the "cold" but by how much Buffalonicians really freaked out. It was nice how quiet the University of Buffalo was when it was closed because of snowy weather. That made cycling through it on my way to work that much more pleasant. But really? Closed? Everything was closed. For several days!

I didn’t even think it was that severe. This was probably one of the coldest rides I did this season and it’s nowhere close to my cold biking record temperature. One day my wife and I walked through the woods to go to a nearby restaurant — we found a sign "Closed Due To Weather". Sure there was some snow. Just like everyone expected, right? Come on now, maybe I’m some kind of cold tolerant freak of nature, but when my San Diego born and raised wife walks a couple of miles through the snow to go out to eat, we sort of expected the place to not have figuratively collapsed from the weather. (I did actually enjoy cycling into the same headwind that literally blew the roof off of the UB bookstore which I pass on my way to work.) My wife actually went out walking for at least an hour a day pretty much every day this winter. We both really appreciated the break from the heat. We both loved the winter weather and scenery. Our only complaint was there was not more of it!

I do, however, understand why people do not like the winter. Winter makes the horror of car driving much more salient. In mild weather it is too easy to forget how absurdly lethal having anything to do with cars is. Snow and ice intensify all of the problems with driving to the point that people start feeling the correct level of fear and stress. And those who don’t, crash. I personally saw three cars seriously lose control and dozens of others smashed or stranded in awkward positions. But hey don’t blame the winter. Blame the cars and their human drivers!