Chris X Edwards

Go AI? Ha! I can thwart the best chess AI every time. And humiliate it further by effortlessly picking up the pieces off the floor.
2017-02-21 07:50
Fantastic rainbow on the top of Mt. Soledad this morning. (This note to myself will be no worse than a bad photo of it.)
2017-02-20 08:40
Still unnerved by utf-8 in Vim or Mutt ® © §. Ѕeeing "ω=ω0+α*t" in Python3 is like watching a baby's first steps near a busy road.
2017-02-18 07:38
Pyplot docs:"...a collection of command style functions that make matplotlib work like MATLAB." Ah. That's why it's not like proper modules.
2017-02-17 13:41
After 11 years of non-stop Asciidoc, I have mixed feelings about GitHub making Markdown a thing. If I switch, it will be to my own system.
2017-02-16 19:50
Etc.
--------------------------

Redundant Array Of Inexpensive Discombobulations

2017-02-09 14:06

I think I’m finally fully recovered. Whew. Last Saturday afternoon I got an email from the power company saying, oh hey, we’re turning the power off tonight, sorry about that. Such a thing is always a possibility but still never welcome. Since I’m pretty serious about avoiding interruptions of computer service, it is something I try have a plan for. But on this occasion I had a little problem.

First let’s introduce my main personal server, raven.

raven

Raven lives on my desk and is on 24/7 where it does all of my important things. Email, cvsroot, most programming, blog stuff, help notes, and general writing all happen on this server. It doesn’t have a monitor or a keyboard. Instead, I log into it over SSH from wherever I happen to be. It is my own personal cloud.

One of the important features of this machine is a RAID1 disk pair which means that everything that is ever written to disk is actually written to two disks simultaneously. In the event a disk fails, the other carries on with no loss of service. I’ve been using this setup since 2005 and have written about it before.

This machine also runs a Xen VM hypervisor and the real host of interest is a Xen VM. This means that if something causes problems with the VM, it’s easy to restore a backup image of it and try again. And when I say "something" I mean Gentoo update. Oh ya, it’s all Gentoo, of course, because I’m insane. Anyway, many months ago, I updated the hypervisor and I watched with some apprehension as GRUB got switched out for GRUB2. Normally, that kind of thing in the computer world is progress, but GRUB2 is a huge pain in the ass and plagued with challenges. So much so, that it is to me, basically unusable. Thanks to this updated bootloader, I wrote the following note to myself.

There was some emerge shenanigans recently and GRUB seems to have been changed from legacy to GRUB2. I don’t know why GRUB2 wasn’t used to begin with but it looks like I attempted it and then gave up. Maybe I’ll find out why.

I created a new /boot/grub/grub.cfg (none of that makefile bullshit) and I ran grub-install which I think is pointing to GRUB2 these days. Hard to tell really.

I anticipate there will be reboot problems. Raven is up and fine now, but if a reboot happens, it may be very hard to restart because the hypervisor won’t want to boot. This must be tested.

All screens on raven will have to be shut down. After the attempted reboot fails, a monitor and perhaps a keyboard will have to be arranged for xedxen and troubleshooting can begin. Chances are good that sysresc will be needed.

And guess what? After 10 hours of no power I discovered everything in that note came true. Since you don’t care about this nearly as much as I do, you’re asking, what’s the point here? Who cares?

This situation got me thinking about my server design and I came up with some modifications.

  • There is no escape from GRUB2. It simply must be used. This is problematic because, ironically, doing complicated things with it is much harder. The reason for that is that they worked very hard to make it much easier to do complicated things but they completely failed. Now instead of a well-documented configuration file that can be configured, there are distro specific automagical configuration systems that "compile" into the configuration file which, if done this way, is not human readable. My solution - use GRUB2 in the simplest way possible.

  • I want RAID1 and GRUB2 maybe can do this, maybe not. Don’t worry about it. Don’t let GRUB2 even try. Make GRUB2 boot the simplest thing possible and let that system figure out anything more complicated.

  • I feel the same way about initramfs. From my notes on the topic.

I like to have my kernel do what is needed because it is set up exactly the way it should be. And in 2005, this worked for something like mounting a simple Linux system that just happened to be protected by a kernel managed software RAID1 double drive setup. But as time wore on, the metadata for the RAID volumes changed and although the new bootloader understood it, the kernel, weirdly, did not. No one seemed to care that these formerly robust systems were now broken. This was because *every*body used an initramfs so that the kernel wouldn’t have to do its job. I fought this and tried to do it the Right Way, but eventually I just ran into too many brick walls.

But with the simplest boot setup possible, one does not need an initramfs. And this is good. I’m also weird in that I do not load modules. Any of them. If I need kernel functionality, I compile it in. Normal off-the-rack clothes are not better than bespoke tailored ones just becuase the former are popular.

  • Magnifying the complexity considerably is the Xen hypervisor. Since I felt I needed this setup to be a VM, that’s what needed to happen. But it turns out that this is fine. In fact it’s a perfectly reasonable answer to all of the above problems.

The solution to all the problems is to take the preboot complexity of GRUB and initramfs and the right kernel modules and a RAID1 setup and exchange it for a Xen hypervisor (which I was already committed to anyway). The idea is that instead of running all these very complex things (GRUB2, initramfs) to provide for what will be needed on the complicated running server, just boot a hypervisor in the simplest way possible. Then assemble your RAID1 disk set and select the right kernel with the right stuff and fire up the VM.

The only change from my previous system is that the hypervisor is not also on the RAID1 system. (And now finally the point!) Here’s the thing, it doesn’t matter! If I do everything with my VM, and nothing with my hypervisor, why would I ever need RAID1 with that? In fact, it’s better to not have that be mirrored in real time.

On my hardware (you can kind of see it), I have 2 small micro SSDs and 2 normal SSDs. The former were the RAID set for the hypervisor and the latter were for my real server. But I realized, why not just set up one of the micro SSDs to boot the hypervisor just fine and then copy that setup to the other one. It wouldn’t be live, but if I messed up the live hypervisor (as I did with the GRUB clobbering update), I could actually just boot up the reserve one. Since no value gets added to the hypervisor drive once its all up and running, there’s no reason for RAID1. And that cures all of the problems.

Now once the hypervisor is happily running with absolutely no complicating elements, I can put together the important RAID1. Then I just use that raid set for the important server. Here’s my Xen configuration showing the multi-disk device md127 being mapped to the VM’s /dev/sda1.

disk = ['phy:/dev/md127,sda1,w'] # <-- raven

Let’s not ask why it’s 127 and not 0. I’m not happy about it, but I’ve learned to just accept that madness.

Review: Echopraxia

2017-02-04 15:51

I’m not normally a huge fan of hard science fiction. Conceptually it is fine, but realistic spaceship stuff can be pretty sterile. The problem with spaceships is that they’re just not nearly as interesting as they were in the 1960s and 1970s. Back then it was clear from the exponential growth in space travel that by 2017 we’d all regularly travel to space for holidays. As a kid I personally had absolutely no doubt that my profession would be astronaut. But the funny thing about exponential progress in an area of technology is that it doesn’t necessarily go on like that forever (ahem AI enthusiasts).

Far worse than spaceships for me are stories with zombies. Zombies are a stupid trope that has been quite overused in the last decade or so. And if you tell me a story has vampires, I flat out won’t even look at it. Vampires are just too camp. I can only tolerate vampire mockery. If you told me a book has spaceships, vampires, and zombies, well, no, that’s just of no interest to me.

And yet I really liked Echopraxia by Peter Watts which indeed had all of the above. The key to the book’s success with me, I think, was that it was modern. It’s harder to maintain enthusiasm for books by smart people like Asimov saying 1960s stuff that was no doubt fascinating then, but less compelling today. The author claims in the end notes (n.b. there are end notes) that he really tried to reach for something different and I think he fully succeeded.

I’m still working on my fiction grading criteria, but I’ll go through what I’ve got so far.

  • Style - A lot of panache. Quite complex but not soporific. Clean grammar. Solid readable prose for someone with a PhD.

  • Organization - Good. No 80 page chapters. Creative and functional divisions.

  • Character names - Ok. Mostly not irritating or incomprehensible.

  • Believable characters - Good. No problems. Even with advanced, modified, and alien intellects.

  • Natural dialog - Worked. Seemed believable, even for advanced intelligence characters which can be very tricky.

  • Plot complexity - Excellent. Uses some hard philosophy to make some really tricky plot devices. Cerebral but superb.

  • Plot resolution - Excellent. The bar is low here. Like a Tolkien book, this is an out and back story. That’s good enough, but it’s more subtle than that too. Some of it I don’t think I really even understood through no fault of the author.

  • Erudition - World class. Amazing. A huge depth and breadth.

  • Gems - Fair. Watts' entire style is so energized that punctuating it with special flair may not be practical.

  • Immersion/world building - Very good. It was all a little flashy, but there was a lot of it. It is hard science fiction because there was attention to detail and plausibility.

  • Believable tech - Good. It was pretty outlandish, but all vaguely plausible. I’m sure Watts would say specifically plausible, hence the serious end notes.

  • Imagination/creativity - Superb. Bringing the aforementioned vampires and zombies out of the stupor of their tropes was masterful. If anything, I would almost say it was too imaginative.

I first checked out this author by reading some of the work he has publicly available on his web site. I was impressed enough to read more. He sounds like a cool guy too. "He spent ten years getting a bunch of degrees in the ecophysiology of marine mammals and another ten trying make a living on those qualifications without becoming a whore for special-interest groups." Bravo.

Autonomous Car Plan Without Supernatural AI

2017-02-02 00:16

I just got back from visiting JPL today. That long car trip got me thinking about the best general strategy for pursuing the goal of autonomous vehicles.

There are two competing approaches. One approach is the refinement of overly ambitious AI until the car can drive on all roads and situations just like a human. I’ve already mentioned that I strongly believe that this way is not happening any time soon.

The other approach, my approach, is to figure out what we could have been doing ten years ago, and just get on with it. Today! To better explain the general concept I’m trying to outline, I’ve thought of a specific example of my strategy. What follows is a strategic outline that could, from a technical and practical standpoint, begin to be executed immediately.

Let’s start with the company that has done the most to advance the state of the art of autonomous cars. No, not them; I’m talking about Tesla. Tesla is in a unique position to really make a big difference. They’re not a perfect fit for my strategy, but for illustration purposes they’ll do.

Tesla already makes cars. They don’t exactly make the right kind of cars, but in some important ways they do. The fact that their cars are computer controlled drive-by-wire is important. The fact that they have cameras and enough sense to be able to do their lane keeping mode is helpful. But, as we know, the best hands free driving in a Tesla still demands that the driver pay scrupulous attention. This is the thing that I think is critical to eliminate as quickly as possible where possible.

So we have a Tesla that darn near drives itself, but not really. What progress can be made? Here’s the strategy that Tesla should pursue.

  • Find popular long travel routes. A classic is Los Angeles to Las Vegas. Or Los Angeles to San Francisco.

  • Prioritize these routes by how monotonous they are to drive. Also prioritize routes that see no snow and little rain. The stretch of I-8 between Yuma and Gila Bend is a perfect example.

  • Start collecting Tesla Autopilot data on these routes. Note any quirks or problems that ever come up. Add site-specific software if need be. The goal at this stage should be to show that a significant number of cars made this trip with zero human input. Maybe more than would be expected to do so without a human mishap.

  • Build some special lots (think truck stops) right off the exits at each end of the long stretch. For example, one just east of Yuma and one just west of Gila Bend. For all intents and purposes, there is nothing in between but hours of very plain driving. These lots would be run by Tesla and they would have several functions. The primary function would be that a Tesla in this lot should know how to get on the freeway and then drive to the next lot’s exit, get off the freeway there, and park in that lot. That’s it!

  • Rent out Teslas. In this scenario, I should be able to rent a Tesla in San Diego or LA and drive to Yuma the normal unsafe irritating way. Maybe I could use attended autopilot where feasible, but that’s not too important. When I got to Yuma, I would pull off to the special lot, verify my plan and see if the road conditions are ok for such a thing, and turn over complete unsupervised control to the car for the next couple of hours. When I get to the Gila Bend lot, I wake up and resume driving.

  • When I get to my destination, say Tucson, I’ll have a car to do any flexible driving I want, but for this round trip, I’ll knock hours off of it where my attention and energy are required.

  • One tricky paradox with this is that Teslas, being electric, are not ideal cars for crossing the desert, an environment which can barely supply fuel to gasoline powered cars. The answer is that these special lots should also do battery swaps. This is especially true for the rented cars.

For some reason this vision of how autonomous cars could have been working for our safety and sanity a decade ago is repugnant to fancy AI specialists. They seem to feel that 1% of ideal is a glass 99% empty and would prefer just keeping that glass 100% empty until they can make it 100% full. Which should happen about the time that computers will make transportation obsolete for other, hard to imagine reasons.

I believe that by establishing a foothold of that 1%, it will inspire 2%. And then 4%. Soon almost any monotonous long drive should have an option to swap out your battery and have the car take you down the boring road. A lot more people would be buying Teslas. If Tesla was smart and used open standards for the batteries and certifying the road safe for simple autonomous use, then other companies could make compatible cars and people would demand the system be expanded with an exponential network effect. I believe that very soon we would be at the 95% we are at today with research cars trying to handle any road like a human. The difference will be that there will be a lot more real use of real autonomous vehicles. That final 5% will be easier to cure not just from that experience, but because it’s the final 5%; the will to just get rid of it and replace it with an easy environment will be much greater.

What I’ve described is just an example of working with the AI that is perfectly good today and not perennially wishing and hoping for some science fiction AI to get here in 5 to 10 years. I think the Otto truck transportation concept is closer to mine than many others. Perhaps with my kind of system, problems like rain and snow can be avoided by simply not allowing cars to be fully autonomous in such conditions. Sure, that’s not the best situation imaginable but it might be the best situation that is actually useful today. It would certainly highlight where more work truly should be done. Today the work seems focused on the impractical quirky situations that vex even human drivers, situations computers are probably never going to properly deal with without a lot of special help.

Not only am I proposing that this kind of plan can happen. I’ll go farther and make another prediction: that it will happen. In rough terms the way I’m describing the advent of autonomous cars will happen and current well-known efforts like Google’s will fail. Autonomous cars will first begin to be useful by the public only in some limited and incremental form and only on certified roads.

The question I have is really, when? Should we start today? Or should we wait for another 50 years of AI research to try to bring us closer to the human-level intelligence needed to drive like a human? A goal that may be unattainable, especially if the last 50 years of AI research is anything to go by.

Beyond Twitter

2017-01-29 18:13

Here’s an article titled I’ve left Twitter. It is unusable for anyone but trolls, robots and dictators. I haven’t left Twitter because I never really went there. I’ve had an account since 2008 (apparently) but I never really used it. There are some old Tweets there because I did hook it up to my own messaging system that I developed until the (proprietary) delivery service failed. But I’ve never logged into Twitter and actively used it as intended. Every once in a while I take a look to see if I’m missing anything, and I never get the feeling that I am. Here are some of the reasons I feel that way.

One of the promises of Twitter is that it can, in theory, provide very immediate information about what’s going on in very specific places. But I’ve never had that work out. For example, my balcony has a nice view of a large urban area and I once saw a massive fire raging below which produced smoke that blackened the sky over much of the city. I wondered if anyone on Twitter nearer to it could provide more information about what it was. Nope. No one I could find even mentioned it. Maybe it was just my Twitter ineptitude, but I was surprised and definitely hoping for better.

Ok, one should not expect highly local timely news. What about other "news" you can get? Problem one is that spurious news is a big topic today. Could Twitter be part of the fake news revolution? Seems so. I liked how this article entitled Facebook, Twitter and Google bosses to be grilled by Parliament over spread of fake news had the compulsory but nonetheless ironic "Follow us: Facebook/Twitter" at the bottom.

The "news" on Twitter is not local, timely, helpful, or even true. What’s else can be found? For me one of the most irritating things about Twitter is the juxtaposed spaghetti graph of bewildering "conversations" with "replies" and stuff. Imagine if you were reading this text and I took a moment to say—Oh and Bob, let’s definitely do that thing we talked about!--You’d be thinking WTF? But then if I just kept doing it—Your arguments are completely stupid, Mr. Squeaky—you’d really want—If you don’t believe me, Tom, check out Russian dashcam videos!--to stop reading. It’s very much like when someone interrupts an in-person conversation to rudely take a cell phone call. I’ve been known to just walk away from that situation.

Let’s assume you can filter that stuff out (though probably only for each conversation). I would be looking forward to a delightful stream of small messages micro-blogged for micro-attention spans. But that’s not what Twitter is, is it? The 140 character message concept is a fiction. Why do I say that? For a while now Twitter has been mailing me six tweets every day. At first I looked at them (in Mutt, my text email client) but soon realized that they would never be interesting. I went ahead and collected a few weeks of these and analyzed their content.

75 Includes A Link 5 Includes Two Links 3 Re-Tweet 1 Self-Contained Message

Twitter probably is taking special care to send me only tweets that are meaningless without a marked visit to Twitter (which I scrupulously avoid) but that’s not terribly encouraging. Here’s an idea of what typical tweets with links look like. (I’m including the mystery links for completeness, but I sure don’t recommend clicking on them!)

Character Development https://t.co/EJ0xMUnqxk

Trumpism, summed up in 3 tweets: https://t.co/i1GYT0DO13

And where do the links really take you? Perhaps terriblemalware.ru. Who really knows?

Although such links don’t help without further investigation, retweets, can be even more bewildering. This one, for example, doesn’t seem to include a specific link but I assume that it is implied by the very concept of retweet.

RT @SecureThisNow: @_lennart great talk about logging on steroids with full customization at @BsidesSD . #muchlearning #bsidesSD

Finally, here is the one bog standard tweet which had a canonical format.

The ten albums that defined my teenage years were probably all s3ms.

I have no idea what that means but at least it’s the right format.

The fact is that Twitter is just as much about referring people to other things as it is about publishing any content and it is not at all rare that this other material is more than 140 bytes. Sometimes it is weird prose stories, quite long, broken into awkward 140 character packets. Really, Twitter is a baroque URL shortener.

This seems to be a fundamental collapse of the internet. It seems that to a certain extent web search engines have been replaced by celebrity tweets for guiding people to information. Perhaps this is excusable, sort of like Digg or StumbleUpon but with more celebrity influence, but I think it’s really messing with some key principles that made the internet compelling in the first place. For example, it’s definitely shameful how many "140 character tweets" are nothing more than links to non-searchable images containing vast amounts of text. That’s just the worst of all worlds.

Of course if you love celebrity news, and I know that’s by definition really popular, Twitter is pretty great. I just noticed, for example, that Elon Musk does a fairly good job of making his own news so reading his tweets would be like being plugged into what would have been press releases in the old days. But most people aren’t all that newsworthy.

So what of it? Well, I have mixed feelings about it all. If people want to have a way to say short simple things and if those short simple things are essentially links to noteworthy resources, fine. But Twitter, Inc. kind of creeps me out. As I mentioned I had developed my own system, a Bash script, for generating and archiving my short messages and I liked it very much. I used a service called ping.fm to replicate these messages on Facebook and Twitter. Here’s an important fact, however—that company died before I did. Big deal, you might say, but the bigger important point is that if you’re believing there is no chance that Twitter itself can die, you’re not looking at a long enough time frame. Just look at the pitiful state of Yahoo to know that big companies do not equal a safe haven for your data. This is especially true when the true product is you. For example, here is cool hombre JWZ reminding us of this from the comments of a Perl script he wrote for archiving Twitter messages.

# Unfortunately, Twitter seems not to let you retrieve anything more than
# about a year old. So for those old messages, you're just screwed.
# That's what you get for trusting your data to someone else!

Recently I came across my old status message program and I enjoyed re-reading the safely archived content. Since my grand criteria for whether I publish something is whether I think I’ll enjoy revisiting it, I decided I should revive that tradition. Since archiving is important to me, I won’t even get into the mess that JWZ ran into. Some people have their Twitter feeds show up on their blogs (with a ton of obnoxious and slow redirects and pernicious JavaScript). That is the wrong way to do it. The correct way, the proper internet way, is to invert that model. Host your messages in a decentralized way on your own indexable, static web site, and if you feel like dealing with Twitter (or even Facebook), send it to them as an afterthought. If they die like ping.fm, oh well.

If you visit my main web site today, http://xed.ch, you’ll see that I’ve just added a new section of recent status messages. I also did some other changes like finally moving my whole blog system from "dev" to "live" while leaving a dev version to play with. Hopefully it all still works!

I haven’t hooked the status messages up to Twitter yet, but I may if the API key circus isn’t too onerous. But first, I want it to work where it really matters, on my web site. I rewrote the entire system; basically I edited my Makefile and my main page template, and created a one line sed program. It’s not really that hard! I’m going to see how it goes. I’m also going to think about best practices for posting links now that URL shorteners are clearly evil (including goo.gl and especially the non-optional t.co and facebook hijackers).

Update: Here is another good article questioning Twitter written on the same day as mine.

Update 2: My status system is now configured to publish to Twitter too using Python. The important result is that if you like Twitter, you can follow me at: http://twitter.com/chrisxed

America Gets The Point

2017-01-26 13:46

If you’re an actor or interested in writing fiction, it’s a good idea to read books like What Every Body Is Saying by former FBI agent, Joe Navarro. This book is about how to read body language. The FBI is interested in this for the same reason they’re interested in polygraphs. Like polygraphs, this stuff doesn’t always work reliably. Nonetheless, I still find non-verbal communication fascinating. If you’re writing fiction, its lack of reliability doesn’t matter. If a certain hand gesture supposedly indicates a certain mood, when you’re trying to convey that certain mood you can throw in the plausible gesture as a bonus. It doesn’t matter if it’s really accurate; what matters is if it rings true to you and, if so, maybe your readers will feel the same. Anyway. Enough of my fiction writing tips.

Palm In, Serious

unclesam.jpg

While reading this book I came across this passage which I took special note of.

In many countries throughout the world, finger pointing is viewed as one of the most offensive gestures a person can display. Studies show that people don’t like it when someone points a finger at them. In schools as well as prison yards, finger pointing is often the precursor to many fights. When talking with their children, parents should be careful to avoid pointing at them while saying things like "I know you did it." The finger pointing is so distasteful that it may actually divert the child’s attention from what is being said as they process the hostile message of the gesture.

Lumping schools in with prisons always makes me smile, but there was something deeper about this. After giving it some thought I wondered what situation I had been in where someone pointed at me. It is pretty rare. But then it came to me — Trump.

Palm Down, Serious

trump.jpg

I had this image in my head and wondered if it were valid. So I did a very rough experiment. I did a Google Image search involving a person and the word "pointing" ( like this) and for each of the first N images in order, I classified the types of pointing the person was doing. The quantity N varied based on the quality of the results. I kept recording until I had a decent sample or until the images stopped being sensible (a lot of pointless "other" images).

Palm In, Smiling

obama.jpg

For some of my controls like "fred rogers", "beyonce", and "bon jovi", etc. it was clear that there really wasn’t much pointing to be seen at all. But when I turned to politicians it became quite interesting. I’m again using my recent enthusiasm for pie charts and a bit of JavaScript (so turn that on). You can click the names and change the chart.

"dalai lama pointing"

Palm In, Smiling 4 Palm In, Serious 2 Palm Down, Smiling 3 Palm Down, Serious 2 Point Up/Down/Away/Self 10 Other 40
Click search term to select.
dalai lama biden hillary pence obama steve jobs trump

Trump’s narrative never seems to require actual facts. This is unnerving for some of us, but that leaves the conclusion that there is more going on than standard models of reality. It would seem for this reason that Trump is an especially interesting subject for analysis of his non-verbal communication. My conjecture is that being the scion of a wealthy family, he has naturally internalized subconscious mannerisms of a high status individual. He exudes the idea that he is our boss, liege lord, or mafia don. He’s simply the guy who is more important than us. Actual facts supporting that claim aren’t terribly important. He can get away with vulgar boorish behavior because it is expected of people who are less exposed to consequences. For many, apparently, it serves to remind them that Trump is their natural superior and leader. From my perspective, not so much. Now enjoy this video of upper class twits kicking beggars that has an unusually happy ending.

twit.jpg

--------------------------

For older posts and RSS feed see the blog archives.
Chris X Edwards © 1999-2017