Chris X Edwards


Autonomous Cars Are Not Happening Soon

2017-01-20 14:25

Last October I made a depressing prediction that autonomous cars were not happening soon. That five to ten year estimate you’ve been hearing about for ten years? No. Just not happening.

Assuming I wasn’t delighted to be proven wrong, I wondered how long it would take the technologists involved in this science fiction trope to catch on and start dealing with reality. Today I found a tiny glimmer of hope from Nissan. In an article titled "Nissan’s Path to Self-Driving Cars? Humans in Call Centers", an incorrect solution to the problem is proposed.

But at least they understand that there is a problem!

It’s a stunning admission, in its way: Nissan’s R&D chief believes the truly driverless car—something many carmakers and tech giants have promised to deliver within five years or fewer—is an unreachable short-term goal.

I believe that 95% of roads could comfortably be driven autonomously with 30 year old technology. 98.5% could be done with 10 year old technology. Today, 99%. But what about that 1%? While the traditional appproach struggles to get this down to .5%, the fact is .5% is too much. A doubling of effort may, in 30 years, get the amount of driving that a computer can’t handle down to .25%. But even that is too much. How long are we going to want to wait to get this down to the 0% of driving conditions humans believe are undrivable? Can it even happen? I do not think so.

Do we give up on autonomous driving? No, we give up on the stupidly hard parts. Nissan proposes to have the autonomous car call a control center and have a real human assess situations where the car finds itself stuck. The idea is that designing autonomy for very odd situations is not easier than just phoning up some real human help. This is somewhat true.

Unfortunately, this is the wrong approach and it will fail. There will be too many such situations. However, having humans keeping an eye on feedback from autonomous cars is a smart thing. It’s fine to develop the necessary telemetry and communications systems this technology implies, but it won’t really solve that last .5% problem.

What will allow computers to comprehend the most exotic driving conditions humans can safely negotiate? Isn’t it obvious? Nothing. Let me repeat that trying to do this is like designing aircraft to have flapping wings. It may seem like a smart design goal, but there is an easier way.

Eliminate the kinds of driving conditions computers can’t deal with.

Autonomous transportation today focuses exclusively on the cars. How much work is being done to make roads that work well with autonomous cars? That eliminate the problem spots? I know of almost no such work less than 20 years old.

We will have autonomous cars. It will probably start on private roads where the owner can exercise complete control of the driving environment. When will this happen? Probably around 2014. When will society start looking at both the cars and the roads as necessarily interrelated for an autonomous solution to work? That is harder to say, but that is exactly what needs to happen.

Review: Dark Money

2017-01-19 19:00

I wasn’t going to mention this book. I now try to only review books that I find interesting and that I might like to be reminded of in the future. I thought nobody wants to hear about this depressing work and I really didn’t have anything interesting to say about it. But ultimately I did find something interesting about it.

More than half of Americans are rather horrified with the impending regime change and the rest I can’t speak for because I don’t know any of them personally. The country does seem to be divided. And that’s where this book, Dark Money by Jane Mayer went from mildly interesting to quite interesting in a meta kind of way.

The subtitle of the book is "The Hidden History of the Billionaires Behind the Rise of the Radical Right". And really the book could be summarized with the sentence, "Billionaires behind the rise of the radical right." Really, that’s all you need to know summarized perfectly well in eight words. As awful as this topic is, this book actually gives a reason for optimism and hope. Sort of.

I went to Amazon to see how this book was doing and I expected it to be about a 45/55 good/bad split and just like all congressional votes, down party lines. But I was shocked and delighted to find out that if there is one thing Americans agree on it is that this is a good book! I was astounded. Take a look at this astonishing high rating.

Verified Purchases

★★★★★ 891 ★★★★ 121 ★★★ 20 ★★ 5 ★ 18
Click me to toggle.

I was compelled to write some little Javascript function to show a nice comparison between verified and unverified Amazon reviews. I got the book from the library of course, but I get the feeling that a lot of people who didn’t read the book came to write negative reviews. Even still, this is book is freakishly well regarded.

To understand this better I wanted learn more about the few people who didn’t like it. I went ahead and classified all of the verified reviews that received either one or two stars.

9 Partisan 9 WTF 4 Style 1 Valence

The first category I call "partisan" and these seem to be knee-jerk members of what the book’s subtitle called "the radical right". Go ahead and suffer reading all of these; I found it kind of entertaining and amusing. These are definitely some edge of the bell curve morons but isn’t it uplifting how few of them there are?

Amazon customer ★
It seems a book like this should be more apolitical. The story seems
half told. There must be more when considering the political system.
Disappointing ★
Very slanted view. It also uses inflammatory language. Was hoping to
read an objective book on the flow of political funds but was very
One Star ★
Prefer books that are more objective about their subject matter.
Left Wing Propoganda ★
I couldn't continue reading after page 7...if you want any truth about
"dark money", don't read this. She just points fingers at the
Republican Party and their corruption, completely ignoring all that is
the Democratic Party with their corruptions/ bribery/ agenda/ elite
power. She focuses on the Koch Family; how could you ignore the
Rothschild's, Morgan's, Rockefeller's, or even George Soros?? I'm
upset. I bought this seeking truth, but instead it's nothing more than
left wing propaganda disguising itself as a "whistleblower" book.
Aweful ★
Propaganda... I am not a big fan of the Koch Brothers, but this gal is
over the top. She wants to guide the reader and influence, versus
inform. Radical, Extremist. Abuser, over and over... She takes things
out of historical perspective and inflicts her own interpretation of
the facts.
Too biased. ★
Ugh. Too one-sided. I prefer a discussion rather than a rant.
Pretty annoying. ★ ★
Seems to be a promotional piece for the democratic party. I fully
admit that a lot of the guys she lists are questionable characters at
best, but her relentless (and embarrassing) promotion of the left gets
annoying...very quickly. I was looking for something less emotional
and much more about the facts.
Extremely one-sided. The Democrats/Liberals have their private funders, … ★ ★
Extremely one-sided. The Democrats/Liberals have their private
funders, too. Mayer convienently overlooks this truth, and targets
only the wealthy who support conservative causes. In the few mentions
of Soros, and others, who finance the Clintons, Mayer excuses their
surreptitious monied plots as being "good for the country." HAHHAA
Very poorly written book ★ ★
Extremely wordy, everything could have been written as effectively
with half the word. Very biased, you'd think the Republicans were the
only ones playing the game, one mention of George Soros, no mention of
Service Employees International Union.

Just for the record, so you know why I call these people morons, let me show you how partisan it really was by quoting from the book a bit.

Page 321 of Dark Money

The audience cheered. The problem, though, was that no matter how keenly Obama wanted to address economic inequality, he was going to have to turn to his party’s own billionaires and multimillionaires for help. Soon, in fact, Obama would set a record for the number of fund-raisers attended by an incumbent president.

Page 323 of Dark Money

In an interview on CNN, [Bill Clinton] said, "I don’t think we ought to get into the position where we say this is bad work—this is good work." From 2006 until 2009, Chelsea Clinton, the daughter of the former president, worked as an associate at Avenue Capital Group, a $14 billion private equity and hedge fund firm. Marc Lasry, co-founder of Avenue Capital, was a major Clinton supporter as well as a $1 million investor in a fund managed by the Clinton’s son-in-law, Marc Mezvinsky. The Clinton administration had been rife with Wall Street tycoons. Now, as the Obama administration was teeing up Romney’s rapacious business record as his key disqualification, [Bill] Clinton summarily announced that Romney’s "sterling business career crosses the qualification threshold."

In response, the Obama campaign tailored its message more carefully. For the most part, rather than hammering Romney’s wealth directly, it relied on sly symbolism to address the touchy issue of class. "There was too much blowback, so we used cues," says Margolis. "We showed him standing next to Trump’s private jet."

The next species of negative reviews I call "WTF". Just have a look and you’ll see what I mean. Remember, these are all one or two star reviews, and to be clear, out of five where five is good and one is bad. One is bad; five is good. Got it? Of course you do. Check it out. These are also quite entertaining.

One Star ★
enjoyed it very much
Audio by mistake. ★
I am sorry, but I can not review as I accidentally hit audio and I
wanted the book. I would like to return unopened in exchange for the
One Star ★
excellent book and speed of delivery was surprising...
Eye opening. ★
Very educational. Always knew there was big $$ but this book certainly
gave me some eye opening facts. Interesting because it is what is
happening now and I recognize names and organizational names.
bad reader ★
teh reader sounds so bored you will fall asleep
One Star ★
Excellent. A must read.
Binder broke ★
Opened the book and the binder broke!! I am sure the book will be
great but it's disappointing to spend this much money on a hardcover
book and it's so cheaply made.
One Star ★
Bought the book or even really read. ★
I have ordered the book and will write a review after I read it. I was
most interested in the lack of verified purchases in the 1 reviews. I
always read the negative reviews first but was particularly interested
in the reviews again by folks who obviously have not read the book and
all say the same thing. Is there a group of people called upon to
write these things or that get their "thoughts" from the same place?

All righty then. Moving on.

The next category of poor reviews for this book related to style. To a certain extent I sympathize with some of this sentiment.

One Star ★
One Star ★
So boring!!!
One Star ★
Two Stars ★ ★
Written like a textbook.

I must admit that this book is not by a skilled politician or poet aiming to stir emotions. The author was trying very, very hard to achieve a pretty serious level of dispassionate credible scholarship. Forty-five pages of fine print end notes is not how you write slanted inflammatory propaganda.

And finally we come to what I call a "valence" reaction. Interestingly this review perfectly captures my personal feelings about the book.

The minutia of evil ★ ★
This book is too depressing. Jane mayer is thorough in the extreme.
But who wants to read the minutia of evil people - who you already new
were evil

I’m glad I didn’t share this thought on Amazon and discourage people from reading and respecting this book with a two star vote. Really this is one of those modern problems with unsubtle social media. It reminds me of the 169,259 people who "liked" the US Holocaust Museum on Facebook.

Frankly I hated this book. It was really sickening. But in a five star first class kind of way. I can somewhat handle contemplating nuclear weapons, but thinking about how many Americans needed to be fooled or complicit to get to the presidential and congressional situation we have today, I just can’t really think about it too much. But despite that situation, my Amazon review audit of this high quality book is a cause for optimism even if the book’s material is not. The country has been harried into a terrible situation by the bluster of political influence bought with truly stupendous amounts of money. It is clear to me, however, that a huge and satisfying percent of Americans aren’t delighted that billionaires are spending hundreds of millions of dollars to bend and erode government to enrich themselves even more. After the recent election, that was a pleasant surprise.

We may have democracy, or we may have wealth concentrated in the hands of a few, but we can’t have both.

— Louis Brandeis

Drive Fail FTW

2017-01-16 08:45

Nothing fills an experienced computer user with dread like a failing hard drive. I’m experienced enough to have the word when firmly stuck in my head, not if. Even if you take great care to have this entropic force of nature under control (back ups, back ups, back ups), the actual decline, as with the health of old humans, can be rather unnerving. Crazy non-deterministic stuff happens and computer people hate that above all else.

The main tool for restoring a sense of control and comprehension in this situation is Self-Monitoring Analysis and Reporting Technology, SMART, which is a part of most modern (2006+) hard drives. Unfortunately, SMART feedback is almost incomprehensible. The drive can report between one dozen and three dozen or so attributes about drive health. Unfortunately, although there are common themes, each vendor has their own particular system. For example, is a "Start_Stop_Count" of 816 good or bad? Who knows? It’s probably irrelevant.

There are some attributes that would seem to be obvious, like the ones with the word "error" in them, but even that can be kind of tricky. Sometimes you get attributes like "life left" and "seek time performance" which you’d assume should ideally be a high value. Simply looking for big numbers isn’t sufficient to find errors. And how big do those numbers have to be before you start worrying anyway?

I just discovered a very interesting hard drive failure. I know it is failing because one of the file system journals on the drive gets corrupted without ever being mounted and used. In other words, I fix it, reboot, and it’s broken again. What makes this interesting and even quite lucky is that it is installed in a computer with an identical second drive that is working fine.

This allows me to compare SMART reports of the two drives and get a very clear understanding of the difference between an old drive that is failing and an old drive that is still good. Here is a look at the two reports using vimdiff (installed by default on all sane computers).

Failing Drive | Healthy Drive


What I find most interesting about this is that the "VALUE", "WORST", and "THRESH" metrics are almost useless. Besides being comically incomprehensible, the good drive is indistinguishable from the bad using these metrics on "Raw_Read_Error_Rate" and "Offline_Uncorrectable". For the one problem metric where these are different, "Multi_Zone_Error_Rate", they make no sense at all (to me). Having these side by side has done more for helping me understand how to read this information than all other resources I’ve come across.

Of course I have no idea about other models of drives, especially solid state drives, but I imagine there’d be some common themes. The main heuristic I’m getting from this is that if you have attributes which contain the word "error" with RAW_VALUES that are not zero, pay attention to that. Everything else you can mostly ignore. For example, note that both drives reported this.

SMART overall-health self-assessment test result: PASSED

Obviously FAILED would be worse, but don’t assume your drive is good because it says "PASSED".

This side by side comparison also suggests a strategy that I had not thought of before but which, in hindsight, is obvious. When you install a new hard drive, immediately get a SMART report and put it somewhere safe. If the drive starts flaking out at a later time, it will give you something to compare with. More details about hard drive diagnostics can be found in my hard drive notes.

Review: Lovecraft Country

2017-01-15 22:33

In honor of Martin Luther King Day I’ll share my thoughts about Matt Ruff’s latest book, Lovecraft Country.

But first, because it’s my blog and I can, I’ll talk about Martin Luther King Day and, uh, stuff like that. I have to say that I very much like MLK Day, and not just because I get a day off. I like it conceptually. If there is any federal holiday that I relate to personally, it’s MLK Day. You may be wondering how that can be possible since I don’t appear to fit the correct demographic; I am not black. However, if you go to the Martin Luther King Day Wikipedia article today and search for the word "black" you’ll notice that it is only there in the phrase "Black History Month". The first sentence of the article to describe the holiday spells out its purpose clearly, "King was the chief spokesman for nonviolent activism in the Civil Rights Movement…"

Wars are poor chisels for carving out peaceful tomorrows.

— Martin Luther King Jr.

Unlike other federal holidays which sicken and shame me with their perverse glorification of violence and war, MLK Day is refreshingly subversive on the topic.

Now, it should be incandescently clear that no one who has any concern for the integrity and life of America today can ignore the present war. If America’s soul becomes totally poisoned, part of the autopsy must read Vietnam. It can never be saved so long as it destroys the deepest hopes of men the world over. So it is that those of us who are yet determined that America will be are led down the path of protest and dissent, working for the health of our land.

— Martin Luther King Jr.

As Dr. King was well aware, there’s more than one way to be a minority in the US. In January 1991 on the eve of the Gulf War, 79% of Americans polled supported the Gulf War. In March 2003, 76% of Americans supported the Iraq War. As a pacifist Dr. King was part of yet another minority ostracized by mainstream American society. This I share with Dr. King and in a very real way, I derive inspiration and courage from him.

However, in any of the meager publicity generated by this holiday, the main theme of Dr. King’s life would seem to an outsider to merely have been that he was black. I don’t have any insight into what that’s like personally. I can say that I am a strong believer in the fundamental pillars of Liberalism, liberty and justice for all. It doesn’t take a statistician to see that something is severely broken in how America treats its black community.

Why for example are latinos three times more likely than whites to be incarcerated in Scranton, PA? Blacks there are thirty times more likely than whites to be in jail ( source). The only explanations for this are the following. Either black people are worse people than whites or they’re being unfairly targeted by laws and/or policing. Since we know that disastrous policies established by known white racists were implemented specifically to repress minorities, I think it’s safe to assume that black people are just as much people as anyone else and no worse fundamentally. Here’s recent news about Nixon’s motivations for prosecuting the War on Drugs. Note that this policy was disingenuously designed to specifically marginalize two groups, blacks and anti-war protesters (yes, like me!). Luckily I’ve never been interested in any kind of drugs and have never used them but because of civil rights issues and the horrible stink of drug prohibition’s unconstitutional racist hypocrisy, I’ve always been a strong and vocal proponent of complete legalization (or a return to alcohol Prohibition if you like). What drug prohibition has done, disproportionally, to the black community is a shameful disgrace to the principles the US was founded on ("…that all men are created equal…").

I mention the idiotic War On Drugs because it is so idiotic. I’ve never met any intelligent person who thought it was a great idea and that is because we now know with certainty that it was never designed to be a good idea with respect to substance abuse. It is the obvious place to start in moving American civil rights yet forward. Of course the day to day details beyond that are brusting with incomprehensible complexity. Yet in general I support the "Black Lives Matter" movement. I completely agree with Tyler Cowen who believes that invoking Black Lives Matter "reflects the fact that many white people have been unaware of the extra burdens that many innocent black people must carry due to their treatment at the hands of the police. The slogan is a way of informing others of this reality."

And that brings us back to the original topic, how does a brilliant author like Matt Ruff convey the idea that black lives matter? With an ingeniously crafted novel like Lovecraft Country.

One of the strange problems with fiction is that if you know where to look, it seldom achieves its goal of being more interesting than real life. For example, one of my favorite authors, Neal Stehpenson, and a bunch of his friends were barely able to make the Mongol Empire more interesting than the Mongols of reality. I feel that Matt Ruff is fighting a similar battle here. In Lovecraft Country, Ruff uses this to brilliantly create one of the most terrifying and creepy books I’ve ever read. The effect could only be this strong by an essentially authentic depiction of the ordinary lives of 1950s black people.

The book is layered with irony. The title comes from H.P. Lovecraft who was an overt racist. The novel’s characters are a literate black family which enjoys science fiction and fantasy books and is conflicted by popular racist authors like Lovecraft. I personally have not read tons of H.P. Lovecraft, but enough to get the gist of it and I can see that Lovecraft Country is a kind of cool parody of Lovecraft’s writings. Instead of the swarthy natives doing uncivilized voodoo things that shock the proper and refined white explorer types, the roles are completely reversed. The poor but educated and intelligent black family stumbles upon a cabal of voodoo (essentially) practicing rich white men. Just that alone is priceless.

By simply reversing typical racial roles it becomes quite clear how absurdly racist a lot of literature is. True to a Lovecraft style story there is magic and eerie weirdness and scary monster things. But the irony is that none of that is scary in the context of this book. A tentacled beach monster devours some people and, yawn. What would have been climatically chilling for Lovecraft’s readers barely rates remembering in this book.

Yet this memorable book is one of the most eerie and disturbing I’ve ever read. The effect is brilliantly created by a deadpan style that simply shows what ordinary life was like, especially things white people take for granted. It’s difficult to describe how compelling and dramatic that can be and fortunately I don’t have to. Matt Ruff has a PDF of the first chapter on his web site which I highly recommend reading. This doesn’t get into the real story but it shows how for black people at the time, fictional science fiction monsters couldn’t be any worse than ordinary people being their ordinary selves. I highly recommend reading just that chapter if nothing else.

I’m a huge fan of Matt Ruff generally. His calm and measured style belie the subversive ideas of his books. He has an unparalleled ability to see things from a very different perspective and to convey that vision clearly with apparent nonchalance. Although he can be quite hilarious, his humor is usually quite dry and subtle and in this book I think he took special care to avoid flippancy. I’ve read his entire oeuvre and have loved every work. Some of his books I felt had weak endgames but Lovecraft Country was fine in that respect. I highly recommend everyone read the first chapter which stands alone as a thought-provoking bit of context for all Americans. And if you think you might be entertained reading more of the same, do read the whole thing.

Twitter Fail

2017-01-13 20:59

Twitter is weird. I have to say, I do not understand Twitter. I was doing some research with it and I discovered something slightly surprising. Check out this ugly command line. (Or safely ignore it.)

for N in {1..1000}; \
do ( \
T=$(sort -R $WORDS | grep -v "'"| head -n1)$(( $RANDOM % 10 ));\
set -o pipefail; \
echo -n ${T}: ; \
if wget -qO-${T} \
  | sed -n -e "/Account suspended/s/^.*$/suspended/p" \
           -e "/>Joined/s/^.*$/active/p"; \
then true; \
else echo "none"; fi ;) \
| tee -a result ; done

It produces a random word plucked from my Linux spelling dictionary (WORDS=/usr/share/dict/words). Then that word is appended with a random digit (0-9). The result, something like "plateau4" or "spryest9", is used as a Twitter account which I then try to access. I check if the account is "active", "suspended", or "none". None means that the random string turns out to not really even be an account. Of course this implementation goes slow because of the shuffling of the dictionary every time but I left that alone to not give Twitter a reason to block me. I did this 1000 times.

Now is a good time to use my Unix pie chart trick with something like this.

cut -d: -f2 result | sort | uniq -c | pie

Which produces the following.

477 active 103 none 420 suspended

Are there really so many suspended accounts? I’m sure someone went through long ago with a script much like mine and made a bunch of accounts. (Don’t they have captchas or something?) But if Twitter could suspend them, presumably for having some sign that they were created by a bot, couldn’t they then release them back into the namespace pool? Seems strange to me. Although I was working on a different piece of research which I may write more about later, my original hypothesis, that "Twitter is weird", has not been falsified.


For older posts and RSS feed see the blog archives.
Chris X Edwards © 1999-2017