Chris X Edwards


Mobile Security Is An Oxymoron

2016-11-30 08:27

Ok, so I’m a very weird guy, I can see that. Mobile phones can be very handy, I can see that too. Combining those things it turns out that mobile phones are still pretty useless to me personally. That fact, I concede, is strange. I participate in less than a dozen actual telephone conversations a year even when including the hardwired telephone sitting in front of me on my desk at work. I can not remember a telephone call that would not have been better served in an email. Many of my telephone calls involve waiting on hold for dozens of minutes. Today, the fundamental advantage of telephones isn’t the frequency-clipped disembodied voice you can hear, it is that the rudeness and imposition of interrupting someone are usually overlooked. But not by me.

Ok, so unless you just like hearing the sound of other people’s voices nattering away while you’re in the grocery store, and I understand there are many such people, the whole 19th century telephony thing is not extremely compelling. But what about all the other great stuff? Throwing enraged birds into pigs? Everyone loves that, right? Nobody knows how to use a paper map any more so it’s not like it’s even optional, right? Everyone wants to know what their friends' lunch looked like and gleefully stalks Pintergramerbook to find out, right? I could go on and on, but, folks, I’m sorry, I’m as aware of the benefits of technology as anyone and I do not find smartphones compelling. Sorry.

I can confidently surmise that I had my Sharp Zaurus SL 6000 before you had a smart phone. That brilliant pocket computer highlights exactly why I find modern small computers so uninteresting. On the Zaurus I could easily open a terminal. It ran a proper Linux kernel and I had full root permissions. I could run a Python interpreter in the normal Linux way. I could install any Linux software, Vim for example, and write my own. I could tape it to my Roomba and steer with it over wifi SSH. I felt like I was in control. But Android revoked almost all of that control. Don’t talk to me about the nightmare of warranty-voiding freakish rooting hacks. The necessity of such tricks is exactly the problem.


Now we come to the elephant in the room. What happens to security when you use a networked system which is controllable by clever hacks and back door tricks, but impossible to control by authorized users using ordinary methods? It sucks. When I look at almost any security measure taken by the computing systems that I feel are safe enough to use, they are almost all invalid with modern phone operating systems. For example, why do package maintainers provide MD5 hashes of packages? So you can verify things came from a trusted source. With Android, you can’t verify anything of the kind, and you have no idea who is a trusted source. Or take a simple thing like user accounts. These are designed to restrict privileges so that if some software is acting in bad faith, it can be contained. Android has a disgustingly perverse privilege model that just mocks proper security. The only thing that user accounts seem to restrict with Android is the device’s legitimate owner.

Blah, blah, blah. Ok, ok. There are zillions of Windows users out there who obviously don’t care about terrible security and bizarre conflict-of-interest turf wars in their computers. Fine, fine. What is blowing my mind in the smartphone era are the Linux people. The acquiescence of the people who should know better is what really freaks me out.

The first iPhone I ever saw was being proudly shown off by a sysnet (systems and networking) professor, a guy who studies computer security at the highest level for a living. I remember the thought I had at the time which remains the same to this day, "Hmmm…and you’re ok with that?"

Distrust and caution are the parents of security.

— Benjamin Franklin

Security is hard. In the Linux world at least 50% of knowing roughly what you’re doing involves various security measures. With the advent of smartphones it seems like everyone took the opportunity to make a clean break with the truly onerous task of secure computing. By relinquishing control, even technical people seemed relieved to relinquish responsibility too. Another quick example — I asked the head computer security analyst at my university (largest employer in the 8th largest city in the USA) what he thought about phone security. His answer was, essentially, it’s bad. Very bad. Ridiculous bad. Sure. Whatever. So what does he do about it? Well, nothing special really. He mostly just assumes it’s insecure and behaves accordingly (lucky for him he’s a professional at that). And yet, he believes that his smartphone was responsible for his Amazon account being hacked (and if anyone should know that, it is he). How did that then change his behavior? Amazingly, not much! This is typical! People who know better stop caring for some reason. I don’t understand this. I haven’t been able to stop worrying and love this bomb. I have never been interested in using a computer as a computer that I can’t control with the full force of computer science. I don’t care what magic services it provides. If I don’t have at least theoretical control and it knows who I am, it creeps me out and I want nothing to do with it.

Obviously I have a lot to say about modern telephones, but at the same time, I don’t. I wish I could point to the smoking gun and say, ah, here is why you shouldn’t use this. But I am not an expert in these systems which are designed to prevent me from properly understanding them. I just know what I appreciate and trust about the Linux systems I do use, and I can’t see any similarity to the way smartphones are controlled. For the same reasons I (and people like Richard Stallman) boycotted Microsoft operating systems for almost 20 years now, I can’t accept Android. I feel vindicated that my 1999 assertion that the Linux kernel could be made usable by normal people was true, but at this point Android is worse than Windows. (Do I even need to point out that IOS is worse than Android?)

Ok, I don’t like telephones. I don’t like proprietary operating systems that exclude you from control, destroy your privacy, and prey on you at every opportunity. With all that baggage, in comes a new topic that is of particular importance to fancy computer people taking care with security: multi-factor authentication. Abbreviated as MFA or 2FA, this basically is about using your phone to add another layer of security to unrelated services. I think you can guess by now how I feel about that. I’m not impressed. I’m horrified.

But again, I’m not the head of some nation state’s hacking agency. Although I take a serious and diligent interest in security concepts, I don’t make a career of studying the dark corners of proprietary software. When I step out of the light of the non-proprietary, free software world, I am overwhelmed and terrified.

This post is just a starting point, a way to dump my rough misgivings along with some links to why someone might feel this way. Make of it what you will.

Phone Insecurity

MFA With Phones Is Bad

  • Can we agree that whoever has complete control over your device can completely subvert MFA? Please?

  • Given the history of large companies' security (Sony, LinkedIn, Home Depot, etc), if you give your phone number out to companies the odds are high that it will end up in the hands of criminals.

  • What happens if your phone is lost, stolen, out of batteries, left at home, etc. — MFA using that phone is worse than useless. MFA always suffers from the problem of what to do if the various components fail.

  • Why is a mobile phone considered an additional factor for multi-factor authentication, but email is not? This is absurd. For some of us anyway.

  • Is it multi-factor when someone uses their phone to access a service and then receives the MFA token on the same phone?

  • In my AWS attack they had me before I ever would have had a chance to register MFA. In theory, they’d have had my phone as well, compromising all other MFA schemes.

  • NIST is No Longer Recommending Two-Factor Authentication Using SMS. NIST Recommends SMS Two-Factor Authentication Deprecation.

  • Krebs talking about a phone based MFA scam.

  • Social Security security apparantly requires an insecure telephoneAnd rescinded!

  • This bizarre article argues for phone 2FA but do carefully check out the utterly insane contradictory section "Faking Two-Factor Authentication".

  • Dedicated MFA authentication devices seem like a reasonable idea. Using another computer, one which is likely to be insecure, does not.

Cell Network Attacks

SIM Card Attacks

Client OS/App Vulnerabilities/Malware

My Absurd Telephone

  • Providers and vendors sell tweaked versions of Android that may not properly implement security updates. My phone’s Android version is from 2011 though the phone says it’s up to date. What could go wrong?

  • "Settings → About Phone → System updates → Update Android" → "Your system is currently up to date."

  • "Settings → About Phone → Android Version" → "2.2.2"

  • Futex attack vulnerable on Android versions 4.0 through 4.3. So am I immune? Ha.

  • Linux is currently on kernel version 4.9. Here’s my "up-to-date" phone’s Linux kernel.

$ cat /proc/version
Linux version (jinyoung.chon@Sprint14) (gcc version 4.4.1
(Sourcery G++ Lite 2009q3-67) ) #1 Mon Mar 7 11:55:10 KST 2011
  • The last kernel was packaged on 23-Feb-2010 15:43. This means that when the vendor compiled this Linux kernel for their Android system, it was already a year old.

Why Your Zip Files Irk Me

2016-11-29 18:31

In the world of normal people, when a collection of files needs to be downloaded from the internet as a single entity the odds are good that these files will be packaged into a zip file. This is a shame.

The reason is that it lets people think that this is a good way to bundle files and it is not. It may be an adequate way and it may work fine. Linux and Mac people certainly have an unzip command at their disposal but the problem is more subtle than a simple ability to unpack it. The reason I know that zip is a bad idea is because I know of a better way. Of course I’m talking about the Unix way.

The problem of zip is that it packages up files and it compresses them (let’s ignore bizarre options like -n that may be able to suppress this default behavior). Why this is bad can be seen with two examples.

Imagine I had a collection of mp3s that I wanted to make available for download as a set. For these files zip will pointlessly "waste its time trying to compress them", to quote from the zip man page. The problem is that mp3 files are already compressed so that compressing them again when I package them makes little to no sense.

Here’s another example. Let’s say I had an enormous file containing an ASCII text SQL dump. I want to compress this because that will be very effective but there is no need for any kind of archive container. Yet if I zip the file, I will have to work with it as if it were a collection of one. Why should there be any accounting about possible other files when I know I just want to compress a single file? Have you ever purchased a single bag of snack food at a store and the cashier rings you up and asks if you’d like a bag? If anything like that ever happens to you, I hope you think the same thought I always do, "No thanks, it’s already got a bag." How many freaking bags do you need?

The Unix way tries to break down the fundamental operations into separate steps so that they can both be used if needed and not if not. Not only that, but this architecture allows one to change the parts if they are not suitable. The normal Unix way to bundle a collection of files into a single file is tar (my notes). Note that tar by default does not compress things. It can and often does, but the important functionality is aggregating files into a single manageable file. If you don’t like tar for some reason, there are other options. An even more ancient archiving system is cpio (Mac people also have it ready to go by default). This may seem like pointless ancient history, but the Linux distributions (e.g. Red Hat) still seem to think that cpio archives have advantages for making initial ram disks for OS booting. To really drive home the point that there really is more than one way to skin this cat, Unix has yet another very common archiving tool called ar (Macs also have this by default, see man ar). This is mostly used to package dynamic library object files, but nothing prevents you from rounding up your mp3s into an ar archive if you want. If your files are all the same exact size, which isn’t especially uncommon with many data sources, you can use a simple Unix cat to pack them and use the Unix split command to unpack.

The huge point here, is that the Unix way conceptually separates the archiving from the compression (even if they occur simultaneously). Unix has even more ways to compress things. The classic way is gzip (ready to go on Macs). This is not merely the GNU implementation of Zip, it is gzip, quite a different beast. First it does no multi-file aggregation. That is, quite properly, outside of its scope. With gzip you can specify exactly how you want the files compressed (--fast which is -1 or maybe --best which is -9, default is -6). But it’s a lot easier to figure out how to get what you want since there’s no archiving cruft to figure out.

But that’s the tip of the iceberg for compression. The real power comes from being able to pick and choose which compression program you want to use. There is an ancient compression system called, plainly, compress. I don’t have it on my Debian Linux system by default, but Macs do seem to naturally have it. The reason that it’s not so common today is that gzip can uncompress files reduced with compress but today there are just more effective compression algorithms. Chiefly is bzip2 (Mac, installed). This works very similarly to gzip with extra aggressive compression. The Linux kernel maintainers seem to prefer using the compression program xz (Mac ready). Another one is 7z. This one seems to appeal to Windows people. I find it slightly annoying because it can, like zip, archive files (but badly - doesn’t preserve file order), but Linux and Mac today have perfectly good 7z utilities by default and it has better compression than zip.

Not content with archiving and compressing zip also can encrypt the contents. Hopefully by now you’re understanding the problem. It’s better to apply a separate utility that specializes in encryption rather than take what zip threw together as an afterthought. Options for encryption include ccrypt, pgp/gpg, and mcrypt. My latest favorite way uses the universally available OpenSSL suite. All of these are documented in my crypto notes.

I’m writing this carefully because I often have a difficult time properly convincing people that the Unix way is correct. To further make the point, let me share an illustrative example of someone using the Unix way, but doing it wrong. Using zip would have been even more wrong, but take a look at this archive.

$ tar -tvzf chembl.tar.gz
0 2016-11-14 06:23 chembl/
9044737177 2016-11-14 04:09 chembl/chembl.sql
1001 2016-11-14 06:23 chembl/INSTALL

It’s ok. It’s a normal gzip compressed tar archive. It’s ok but it could be better. The problem is that there are two files, one tiny and one huge. Both are compressed with the archive. If I want to read the tiny INSTALL file, I will have to unpack the entire archive including the 9GB file.

The correct way would have been to have it set up like this.

$ tar -tvf chembl.tar
0 2016-11-14 06:23 chembl/
2030884202 2016-11-14 04:09 chembl/chembl.sql.gz
1001 2016-11-14 06:23 chembl/INSTALL

Packaged like this, the tiny file doesn’t even get compressed. Why should it be? Now when I extract the tar archive I’ll wind up with a 2GB file and the INSTALL file which I can then read. The files extracted from the archive will be about the same size as the archive. This kind of fine control can be very helpful when you’re pushing the limits of your hardware or piping large file trees between various processes and tunnelling them to various hosts. Obviously if you’re creating an archive for linear storage as on a back up tape, tar is ideal; its name comes from "tape archive".

I also feel that zip has aesthetic flaws. Sure, normal people with their little normal files just open a magical file manager window of the zip archive and they muddle through simple things fine. On such non-explicit systems I sometimes get confused about whether it really is unpacked or it just could be unpacked or what the heck is really going on. In fact even in Unix command line mode this is annoying. You can see what a zip archive contains with unzip -l and the special zipinfo command but I don’t think that zip proper, the main command, actually has a flag to just dump the archive’s contents (it’s not -l which is inconsistently --to-crlf).

Obviously you need to know how to handle zip files. They aren’t going away. Indeed, those master turd polishers known as Java programmers have something called jar (Java archive) as their main software distribution format and it is actually exactly a zip file with some proscribed contents. But please, if you’re going to package and/or compress and/or encrypt some files, please consider doing it properly.

Your Headphones Are Listening

2016-11-28 08:05

Although he was raised a good Linux boy, my son likes to use a variety of insecure operating systems to keep in touch with his bad-influence friends. Sometimes he leaves creepy chat programs running with a live mic while he is AFK. I’ll hear some faint high-pitched noises and realize it’s a gang of squeakers coming through his headphones and everything I say has, unbeknownst to me, been broadcast to their little party. Come on, I know normal people seem to welcome the Stasi, but having your private conversations piped to random adolescents? Am I the only one this bothers?

Being the curmudgeonly old man that I am, I have set up his computer so that I can now easily yank out the microphone plug on his headset. Even if my son isn’t around, I always do this if I’m going to power up that machine, even if I’m logging into my own account with a secure OS. My main desk computers very deliberately have no microphones. When my son leaves his iPhone lying around at night, I treat it just like James Bond treats a KBG planted surveillance listening device he finds. I have been known to wrap this noisome iPhone in conductive foil and then in a pillow.

Call me silly, but some privacy is important to me.

Today’s topic is this interesting question. If I notice a pair of headphones plugged into the microphone jack, would I worry about that? Obviously something is physically misconfigured, but would I flag a privacy threat based on that configuration? I think I would because I well know that microphones and speakers (including headphones) have very little structural difference and can be repurposed to do each other’s job. But that’s a weird hypothetical situation, right?

Turns out no. It turns out that certain Realtek audio chips have a feature (?) where the jacks can be reconfigured in software. This means that what you think of as the output jack can be reprogrammed as the input jack. Now, given the right malware, the situation could easily be as described above with the headphones plugged into the microphone jack. Of course, nothing would seem or even be misconfigured physically.

Can people listen in on ambient conversations in your room through an ordinary pair of headphones with a compromised computer containing this Realtek chipset? The answer, of course, is yes.

The computer I’m sitting at now has a pair of earbud headphones plugged in and lying face up on the desk. I hardly ever use them. I have the Intel 8 series chipset but I can’t really tell from the datasheet if retasking is supported. However, I think it’s best to assume so. Unlike the datasheet, this Intel HD Audio description of enhanced features is in plain English and says this.

Intel HD Audio also provides improvements that support better jack retasking. The computer can sense when a device is plugged into an audio jack, determine what kind of device it is, and change the port function if the device has been plugged into the wrong port. For example, if a microphone is plugged into a speaker jack, the computer will recognize the error and will be able to change the jack to function as a microphone jack. This is an important step in getting audio to a point where it "just works" — users won’t need to worry about getting the right device plugged into the right audio jack.

I actually have thought of cutting off the main body of a 3.5mm stereo plug and using it to short out the mic jack on my laptop with the hope of disabling the internal mic. But I can see that strategy has even more problems than I originally thought.

If you have accepted the built-in microphones in smartphones and laptops always listening to everything you say in the privacy of your living room or car, this news will probably not increase the stink of surveillance in your life by a noticeable amount. If however you have taken great pains to not let that stink permeate your life, this feature will smell pretty bad.

Review: Player Piano

2016-11-26 15:27

I recently highlighted Kurt Vonnegut as an oasis of sanity and intellectual achievement in the metaphorical desert of Indiana. I just finished reading Vonnegut’s notable first novel, Player Piano, and I can’t help but note some notes. Vonnegut is a genius. While his later novels are even more sublime, Player Piano is damn clever and profoundly creative in many dimensions. The major plot twist is absurd and silly yet so clever and wicked that I’m left with the feeling of having just watched a master magician saw a woman in half with a chainsaw, no box, and then the show ends. Wait! What? What happened? How did that turn out?

Unlike many, many otherwise quality novels I read that flail trying to construct proper endings, this book ends perfectly. The unsettling lack of finality is perhaps the major theme of the work. This theme is given context in an overtly idealogical treatment and the whole time I was reading I kept thinking, wait, Vonnegut is not one of "those people" is he? But no, he turns out not to be. It is rather extraordinary when an author can so eloquently present all sides to a very complex situation and walk away at the end without having clearly invested any particular inclination with biased virtue or reproach.

What makes this book almost essential reading for 2016 is its shocking prescience. In a widely read recent article, The New York Times wrote about a 1998 book which seems to predict some important elements of 2016’s election. I have not read that book, but I would like to propose that Player Piano, published in 1952 (!), was no less predictive and important.

The main setting of the book is a future America (post 1952) where machines and automation have replaced most of the labor force. The only people who work are the engineering and the managerial elites. These elites are paid quite well while ordinary people are compelled to take government entitlements and live unfocused, unrewarding lives. While Vonnegut specifically focuses on basic machinery upgraded with basic automatic controls, i.e. the technology of his time, the profound societal change is still relevant for many other such industrial revolutions. For example, I have personally worked with machinists to help them learn how to transition from manual machine tools to computer controlled (CNC) machine tools. I have seen brilliant machinists walk away from that career late in life because they couldn’t adapt to computers radically changing their profession. That example is fairly similar to the exact kind of displacement found in the book, but just imagine all the common jobs of a couple of decades ago which are declining or gone from the modern economy like travel agent, taxi driver, newspaper people, etc. Also when reading I mentally substituted "some stupid app" for "automatic machine" and found myself nodding knowingly. Whenever Vonnegut mentions the engineers and managers that run the world, I could mentally substitute "Silicon Valley/Wall St." and it all made perfect sense.

Why is focus returning to this topic today? My theory is that in the early 2000s, military misadventures and construction related to the housing bubble masked structural problems in the labor market by providing more opportunity for low education males than perhaps society really should have. When these factors faded, the problem of working class America’s oversupply became more acute. These people were energized by promises to either make educational self-improvement more attractive or bring low tech jobs back to the USA. There are good reasons the low tech jobs should not be and are not coming back but reasoning that out with the disaffected is not easy. If you were uneducated and proudly anti-intellectual, why would you believe fancy educated people like this and this and this and this and this when they contradict your beliefs and hopes?

It’s not just uneducated anti-intellectual workers that are struggling to cope with a changing situation. The nature of employment broadly seems to be ever more grim and dissatisfying.

This is actually what the novel really sheds insight into. Vonnegut isn’t anti-progress exactly, but he correctly observes that a society whose people have no real purpose that they can believe in is not going to be very stable. The lack of a purpose, something to do with their time which they can be proud of, leads to an affront on people’s self-worth and dignity. They will be vulnerable to self-serving demagogues who promise to restore feelings of national and personal pride.

How do we find our way forward as a civilization if all of the occupations and traditions of that civilization require a thorough upheaval every century? We’ve barely adjusted to the fact that only a tiny fraction of the 19th century (or third world) agricultural labor force is required to produce record yields today. Just when our society’s labor allocation sensibilities have about got the hang of the internal combustion engine, huge changes loom yet again. We’ve mostly pulled through the revolution of transistors and semi-conductor electronics, but computers and robotics are still sowing structural confusion in our macroeconomic lives even as they gift us with near magical efficiency and economy. Transistors and computers have relentlessly created entirely new ways to think about publishing, medicine, politics, art, music, transportation, engineering, communication, social interaction… everything really.

If there is a grand conclusion to be inferred from the book, the 2016 election validates it: people left behind by changing technology must be looked after humanely. I’ll save my thoughts for how that might be possible for another time and conclude with some interesting passages from the book.


"These are dangerous times—more dangerous than you’d suspect from the surface. But it’s also the Golden Age…"

Thirty-six years to go?

"Is he starving?"
"Of course not. Nobody starves."
"And he’s got a place to live and warm clothes. He has what he’d have if he were running a stupid machine, swearing at it, making mistakes, striking every year, fighting with the foreman, coming in with hangovers."
"You’re right, you’re right." He held up his hands. "Of course you’re right. It’s just a hell of a time to be alive, is all — just this goddamn messy business of people having to get used to new ideas. And people just don’t, that’s all. I wish this were a hundred years from now, with everybody used to the change."

2016 Election

Now you people have engineered them out of their part in the economy, in the market place, and they’re finding out — most of them — that what’s left is just about zero… What do you expect? … For generations they’ve been built up to worship competition and the market, productivity and economic usefulness, and the envy of their fellow men — and boom! it’s all yanked out from under them. They can’t participate, can’t be useful any more. Their whole culture is shot to hell.
… Things, gentlemen, are ripe for a phony Messiah, and when he comes, it’s sure to be a bloody business.
… Sooner or later someone’s going to catch the imagination of these people with some new magic. At the bottom of it will be a promise of regaining the feeling of participation, the feeling of being needed on earth — hell, dignity.
… Things are certainly set up for a class war based on conveniently established lines of demarcation.

Donald Trump

"…he had gone directly from a three-hour television program to the White House."

Working class hero Donald Trump

The break had done anything but teach him humility. He took it as evidence that his money and name could beat the system any time and, paraphrased, he’d said as much. … Paul supposed, gloomily, that beaters of systems had always been admired by the conventional.

Modern fitness trends

"…Paul drove up to his office building. He went up the steps two at a time — his only exercise…"


…a postwar development of three thousand dream houses for three thousand families with presumably identical dreams.

The war on hope

If we plot man hours worked against the number of vacuum tubes in use, the man hours worked drop as the tubes increase. And dope addiction, alcoholism, and suicide went up proportionately.

Technology overload

Paul wondered at what thorough believers in mechanization most Americans were, even when their lives had been badly damaged by mechanization. The conductor’s plaint, like the lament of so many, wasn’t that it was unjust to take jobs from men and give them to machines, but that the machines didn’t do nearly as many human things as good designers could have made them do.

Tech workers

The machines are to practically everybody what the white men were to the Indians. People are finding that, because of the way the machines are changing the world, more and more of their old values don’t apply any more. People have no choice but to become second-rate machines themselves, or wards of the machines.

Online education

"A lot different from my day," said Halyard. "By gosh, we had to get up every morning bright and early, climb the hill in all kinds of weather, and sit there and listen to some of the dullest lectures you ever heard of. And, of course, some poor fish would have to get up in front of us and talk every day of the week, and chances are he wasn’t much of a speaker, and anyway no showman."
"Yes, the professional actors and the television circuits are a big improvement, sir," said Buck.
"And the exams!" said Halyard. "Pretty cute, you know, punching out the answers, and then finding out right off if you passed or flunked. Boy, believe me, we used to have to write our arms off, and then we’d have to wait weeks for a prof to grade the exams. And plenty of times they made bad mistakes on the grades."

College degree inflation

"Call yourself a doctor, too, do you?" said Mr. Haycox.
… "[You’re a] real-estate salesman," said Mr. Haycox. He looked back and forth between Paul and Doctor Pond, waiting for them to say something worth his attention. When they’d failed to rally after twenty seconds, he turned to go. "I’m doctor of cowshit, pigshit, and chickenshit," he said. "When you doctors figure out what you want, you’ll find me out in the barn shoveling my thesis."

College sports

Roseberry frowned. "Well—there’s some pretty stiff rulings about that. You can’t play college football, and go to school. They tried that once, and you know what a silly mess that was."

Military and infrastructure

Those who couldn’t compete economically with machines had their choice, if they had no source of income, of the Army or the Reconstruction and Reclamation Corps.


"Shah says, if these not slaves, how do you get them to do what they do?"
"Patriotism," said General of the Armies Bromley sternly.

No-fly list

"That’s all," said the police sergeant. He dropped the card into a slot, and the card went racing through a system of switches and sidings, until it came to rest against a thick pile of similar cards.
"What does that mean?" said Paul.
The sergeant looked at the pile without interest. "Potential saboteurs."
Wait a minute—what’s going on here? Who says I am?"
"No reflection on you," said the sergeant patiently.
"Nobody’s said you are. It’s all automatic. The machines do it."
"What right have they got to say that about me?"
"Oh, they know, they know," said the sergeant. "They’ve been around. They do that with anybody who’s got more’n four years of college and no job." He studied Paul through narrowed lids. "And you’d be surprised, Doc, how right they are."

Surveillance state drone

Looking up, they saw a robot helicopter in the sky, its belly and blades reddened by the fires below. "People of Ilium, lay down your arms," said its loudspeaker.

DRM Digitally Managed Right

2016-11-20 15:40

One thing about me that surprises people is that I read books, the kind printed on paper made with murdered trees. Why don’t I have a Kindle or some other magical book gadget? I’m not sure about the ecological tradeoffs if you get dead tree books from the library like I do; if 100 people make good use of a paper book, that could be a better deal than 100 people all charging lithium ion batteries to read in other ways. Could be. Hard to say. Another problem I have with reading books on digital devices is that I like to read outside in the daytime. In the overly bright SoCal sun, an actively lit display is simply not actively lit enough. The problem e ink was designed to solve is still a problem I have.

Of course I read books indoors too, maybe twice as often, sometimes in the dark. I know how to configure computers so that the text on them is, for me, optimally readable. But anytime I have looked into reading a book on some kind of computer, I’m always balked by what, for me, is a very suboptimal reading experience. This usually takes the form of some very irritating web browser shenanigans or worse that basically drive me crazy. Add to this the turf wars of each content provider trying stalk your reading habits and monopolize your dollars, and my lack of dollars; the whole proposition just doesn’t seem worth it.

Recently I was checking the public library’s website for a real book I had reserved and there was an "ebook" option with a button that said "Read Now". As a bit of a lark I clicked that and after allowing only one mysterious domain to run Javascript in my browser I was looking at the front of the book. That was impressively easy and free of charge. I started reading it and got about 100 pages into it before I had my meltdown. It was ok, but ok is often not good enough for me. I wanted to keep reading, but my way. I wondered, can I turn this thing into a proper text file? Of course there was all kinds of tricksy browser smoke and mirrors to keep you from even selecting passages (super annoying as I was quoting heavily from the text).

About 90 minutes after deciding to see if it could be done, I had a proper text file containing the book’s text. I’m not saying, run amok and do illegal and terrible things with properly copyrighted stuff, but sometimes responsible people can have legitimate uses for this kind of thing. For example I’m definitely going to do some kind of frequency analysis on the text (as I did here). Another possibly legitimate application relates to the book scanner I built. I never got the cameras configured, but I may revisit that project now.

I’m sure there are serious improvements I could make to my strategy but basically here is how I did it.

Auto-indexing - You need a way to automatically advance to the next page in the reader. In the case of the setup I was using, a space while the browser was focused would advance to the next page. What I needed was a programmable way to make my computer think I pressed the space bar when, in fact, I did not. With Linux you have complete authority and control over what your computer does and because simulating a space key press event is not something computer science prohibits, Linux does not either. The way I chose to do this involves the uinput Python module that can interface with the uinput kernel module. Getting both of those things available is the hardest part of this whole operation.

Here’s the tiny Python program I use to synthesize a space key press event.

'''Don't forget! You must load the kernel module with: sudo modprobe uinput'''
import uinput
import time
ukeys= list()
device = uinput.Device(ukeys)
time.sleep(1) # Needed or things don't get set up properly.

When this program is run, a 1 second delay happens and then the kernel is told that the user input system has just pressed space (even though it hasn’t). The 1 second delay seems necessary ensure the input system is ready for events from this module. There probably is a way to shorten or eliminate this but I was being conservative. Another approach would be to look into expect.

Iterate each page - You need to repeat the steps for each page. I used a Bash script with something like this.

for N in {01..547}; do
    : ... Do each page's stuff here.

Capture the image - If you can see it, you can probably dump a bitmap of it somewhere. I use ImageMagick's import command which is so clever it’s painful. First use xwininfo to find out the window ID of the browser where the material is. Then do something like this.

import -window "0x3200097" +frame ${OUT}/uncropped.png

This produces a PNG file called uncropped.png containing the contents of the specified window.

Next - Run the aforementioned Python script at this point. The idea is that while further processing is transpiring, the (laggy?) browser can be settling down. Note that this needs to be run as root. Also since I didn’t want to put uinput among my real Python modules, the PYTHONPATH needs to be specified.

sudo \
PYTHONPATH=/home/xed/X/P/prog/py/uinput/lib/python2.7/site-packages \

Crop - Often the specified window will contain all kinds of superfluous junk that should be cut out. It may be possible to combine this step with the import command, but I didn’t check this too closely. Here’s the command that I used to isolate just the targeted text part of the image.

convert -crop 700x740+50+167 ${OUT}/uncropped.png ${PNG}

The geometry option argument means that I want a 700 pixel wide by 740 pixel tall image taken from the parent image starting at 50 pixels over from the left edge and 167 pixels from the top.

OCR - Once you have the image of the text capture in a file you need to convert it to text. This is where the computer needs to actually "read" dots on the screen into semantic text. I used a program called tesseract which I discovered is astonishingly good. Debian/Ubuntu users can just use apt to install tesseract-ocr. Really just take a look at how hard this is to use.

tesseract ${PNG} ${OUT}/pp.${N}

That’s it. This will produce a text file containing the text found in the image. The second argument will get a .txt appened to it.

Concatenate - Just focus the browser and run the script. You should end up with a collection of text files. The point of this exercise can be highlighted by showing how easy it is to concatenate them all into a single file.

cat *.txt > entire_pp.txt

Ligatures - The OCR likes to faithfully transcribe ligatures. This can be fixed with some sed like this.

sed -i 's/fl/fl/g' entire_pp.txt

Quotes - The same goes for fancy (open and closing quotes) and unnatural apostrophes. Same fix.

M-dashes - A tricker problem is that the OCR transcribed many of the long dashes into the letter i. This strange quirk makes a lot of ugly misspelled words. It can probably be fixed by actually learning how the tesseract program really works, but I already know how Unix works, so…

$ function fixi { W=( $(echo $1 | sed 's/i/ i /g') ) ; for p in $(seq \
0 ${#W[@]}); do for P in $(seq 0 ${#W[@]}); do if [ "$P" == "$p" ];   \
then echo -n " "; else echo -n ${W[$P]}; fi; done; echo; done | while \
read N; do if [ ! "$(echo $N | spell)" ] ; then                       \
echo \"s/$1/${N}/g\" | sed 's/ \([^ ]*\)$/--\1/'; fi; done }

$ spell <entire_pp.txt | grep i | while read N; do fixi $N; done  \
  | grep -v sed- | tee dashcorrections

I know, it’s a horrible hacky mess but it does show a proof of concept of what is possible. This produces a list sort of like this.


There are some false positives, but overall it works well.


You can use this with Vim or sed or whatever and fix things up. Running sed 510 times on a 564k file (547 book pages) took only 98 seconds on an extremely weak machine.

I’m told that converting the images to bitonal before sending them to tesseract will improve quality quite a bit, but my results were good enough to not bother.

So there you go, it is not just possible to convert an ebook into a text file, it’s not even that terribly difficult. Certainly manually converting a page or two that you’d like to not have to retype to cite is quite easy and worth considering.

Update - 2016-11-24

Dr. S points out another potentially easier way to automate the browser. "I usually use selenium for those things but it doesn’t always work." He also reassures me that I’m not the only one who’s ever been driven mad by bad (web) interfaces that require unnatural clicking.


For older posts and RSS feed see the blog archives.
Chris X Edwards © 1999-2016