USB Drive Notes

Let’s say you’ve overwritten all of the device with dd if=/dev/zero ... how can you get back a factory condition such that an SD card is ready to be used by a normal device or person?

Turns out fdisk works fine just like it did in the old days.

There seems to be two approaches. One is mkfs.vfat which is built into the kernel and seems pretty universally available. Some people suggest mkdosfs which is just a symlink to mkfs.vfat.

Seems to work at least for SD cards; maybe flash drives are different?

f3 - A Linux utility that tests flash cards by filling them with files (f3write) that it then can read and check (f3read).

Note that this can take a long time. Make sure you get a pretty quick SD card to USB converter. Less than 5MB/s is bad. 45MB/s is decent.

flakyflash is a program that probes at cards writing and reading and looking for inconsistencies.

These notes are ok, but at least the author is trying to do the same thing I am trying to do which is unravel the mystical processes that take place (or fail to) inside of utility programs that try to automagically do this job.

An interesting reference is the man page for makebootfat which says this.

It is an open debate how or even if one should partition a USB drive. Ultimately I’d like to put the OS on a 10GB partition or so and have another partition with a sane (ext.*) file system to use with data. I have accomplished this with sysresc so I know it’s possible.

Ideally I could partition with no VFAT or FAT32 but I can’t figure out how at this time.

From the same reference, here’s my interpretation of the process.

When the partitions and the MBR are prepared but nothing else you might get an error (like my laptop) which says this.

This actually represents a level of success since that first line is the MBR successfully being hit by the BIOS and then looking for the actual boot loader (which is not present).

Syslinux has another derivative called extlinux which boots directly to ext2 file systems. All of these operate outside of normal distro support so they run pretty independently. This means if it’s not easily available as a package, just install it from source. Here is extlinux’s web page.

And here is a link to a typical download location. In some cases I had to install from source. It’s a well behaved tar file that compiles with a simple make. On CentOS7 I needed to install packages nasm and libuuid-devel.

It’s RPM is syslinux-extlinux. For Gentoo it’s sys-boot/syslinux.

Make sure the drive isn’t automounted.

First use fdisk to make a simple partition. You might be able to use parted or a fancy new way, but there can be trouble with that. This does present problems for very large drives. This should look something like this for an 8GB drive.

Update the kernel about any partition changes.

Then make the file system. There is some chatter that journals (ext3 and ext4) do not help with USB drives. I tend to believe that. KISS.

Mount it.

It might be smart here to boot the installation disk along with the USB drive (or install Gentoo with chroot) and do the installation. When it is time to partition the drives, do it manually. Don’t format since you’ve already done that. Select the USB drive to mount as root. If there are any other drives on the system check to see that its swap is not automatically included in the fstab. No big deal if it is though.

Find an MBR. Check in /usr/share/syslinux/mbr*bin. If using syslinux from source, check ${PARENT}/syslinux-6.03/bios/mbr. Or if you love adventure, replace bios with efi64 or efi32. Put an MBR in place.

Run extlinux. Yes, the man page confirms that it is designed to be pointed at a mounted directory’s mount point.

Edit the /mnt/usb/extlinux.conf or copy another syslinux type loader’s config file into that file. That file needs to be there.

Hmm. It seems it’s harder than it should be to find an example of this. Here’s one.

I’ve commented out lines that are there for reference if you’re trying to get a menu to work but are probably not needed for a simple boot loader.

Add a kernel and initrd as described in your extlinux.conf file. The kernels and paths were correct for the Debian I tested. This actually worked for me. With this I was able to boot an installed Debian system.

The Debian boot disk was not so successful. I was able to boot a Debian install CD’s kernel from an ext partition on a USB drive. The problem is that then the installer was confused about where to look for the install payload. Since there was no cdrom per se it had to look on the drives. Unfortunately that kernel seemed only capable of looking at FAT partitions so it couldn’t mount and find the full iso sitting right on the USB key. So close! This won’t work for installers, but unetbootin works fine for them, but it does offer hope of how to get USB drives to boot nicely with an arbitrary system installed.

I really appreciate this tool because most bootable usb drives I’ve ever made were made with it. At the same time, I hate it’s mystical opaqueness. What is it really doing? How can I do those steps (perhaps in a customized way)? The big problem with Unetbootin is that it is designed for FAT file systems which are a bit of a joke for serious purposes.

Anyway, here’s the normal procedure for doing things like a normal person.

Create a file of precalculated pesudorandomness ready to go from shared memory.

Dump that on to the raw device. Obviously this will hose all contents so be careful. The fdatasync "conversion" makes sure that the data is properly on the device (not just promised in caches somewhere) before quitting with the final timing.

Now that the data is there, see if it can be read back. Ideally the device should be unplugged and reinserted before this step to prevent Linux from knowing what it just wrote there and making some stupifyingly clever optimizations, which it often does.

Test what optimal MD5 calculating looks like. This can be subtracted from the read test since it’s the part of that test that’s not really doing any reading.

So 48.426 minus 3.453 gives 44.973 seconds for 1GB or 22.2 MB/s pure read action.

What about along the whole drive? Is the writing speed even?

This writes 64kB at each place along the whole drive every 25MB.

How to that’s how to test writing. To test reading is similar.

Another completely different test uses the mysterious hdparm.

Is the device dead? Here’s what that looks like.

Note the RO flag is not "rw" as it should be. This device has freaked out and will only allow reads because it is trying to let you rescue your data but it can’t handle any more writing.

Here’s a good paper on why one would want this and some ideas about how to go about getting it.

https://sites.google.com/site/linuxpendrive/rorootfs

Some more practical information about this kind of architecture.

http://www.funtoo.org/Funtoo_Filesystem_Guide,_Part_3

One inspiration is the Ubuntu guest accounts. This solves the problem of programs that need a writeable space (which, as with all writeable targets, must be in a tmpfs) but which already contains some things (which would clearly not already exist in tmpfs). This is a fundamental problem with using something like a window manager that expects a certain directory structure to be present (~/.config/mate/gtk or something like that) but must also be on a tmpfs.

Here’s a decent description of how Ubuntu guest accounts work.

https://help.ubuntu.com/community/CustomizeGuestSession