tinyapps.org / blog

Randomize lines with custom delimiter #

Arthur de Jong's rl (randomize lines) "reads lines from an input file or stdin, randomizes the lines and outputs a specified number of lines. It does this with only a single pass over the input while trying to use as little memory as possible." It came in handy when I wanted to randomize a collection of quotes which looked like this:

Nothing, to my way of thinking, is a better proof of a well ordered mind than a man's ability to stop just where he is and pass some time in his own company.



There is an old story about a famous rabbi living in Europe who was visited one day by a man who had traveled by ship from New York to see him. The man came to the great rabbi's dwelling, a large house on a street in a European city, and was directed to the rabbi's room, which was in the attic. He entered to find the master living in a room with a bed, a chair, and a few books. The man expected much more. After greetings, he asked, "Rabbi, where are your things?" The rabbi asked in return, "Well, where are yours?" His visitor replied, "But, Rabbi, I'm only passing though," and the master answered, "So am I, so am I."

Jack Kornfield


Simplicity of life, even the barest, is not a misery, but the very foundation of refinement.

William Morris

Here's how I randomly reordered them with rl, specifying "%" as the delimiter instead of the default newline:

$ rl -d% quotes.txt > randomized_quotes.txt

The latest version is 0.2.7. Install via ./configure && make && make install, or OS X users can grab this binary I compiled under 10.6.

(tags: unix, randomize, lines, text, reorder, rearrange)

/nix | Apr 29, 2010

Transfer large files directly between computers on the Net #

There are a number of free, third-party solutions for transferring files directly from computer A to computer B across the Internet: However, these services have several issues: Here is a quick and easy alternative for Mac users which involves no signups or third parties:
  1. On server: System Preferences > Sharing > check "Remote Login"
  2. On server router: Forward port 22 to server
  3. On client: Use a SFTP client (either the included CLI sftp or a GUI-based app like Transmit, CyberDuck, YummyFTP, etc) to connect to the external IP address of the server - instant file sharing!
Opening ports and enabling services can expose your network to additional risk. See also: Hardening SSH and Mounting Remote Filesystem in OS X Finder via SSHFS.

/mac | Apr 28, 2010

Shrink PDF file sizes with ShrinkIt #

ShrinkIt 1.1 [181K] Strips needless metadata from PDFs to reduce file size.  Screenshot

/mac | Apr 27, 2010

The disk cannot be partitioned because some files cannot be moved. #

Boot Camp offers some really bad advice when it can't partition the disk:
The disk cannot be partitioned because some files cannot be moved.
Back up the disk and use Disk Utility to format it as a single Mac OS Extended (Journaled) volume. Restore your information to the disk and try using Boot Camp Assistant again.
A much more helpful message would be:
You've got some big file(s) I can't move. Find 'em (find ./ -size +100M) and get 'em off the disk, or just use iDefrag instead.
No, actually this is what the message should read:
You need to move big_file.iso to another disk before Boot Camp setup can continue.

/mac | Apr 27, 2010

Hard drive enclosure with write protect switch #

It is getting harder and harder to find USB flash drives with write protect switches. Thankfully, ACARD Technology makes the ACP-2127, a 2.5" USB to SATA hard disk enclosure with hardware write protect switch. Paired with a fast HDD or SSD, this could be the perfect replacement for a technician's aging flash drive. Mars Tech has the best price ($20), and awesome phone-based customer service to boot.

/misc | Apr 26, 2010

Newly-added hex editor #

Green Award HxD [789k] + Hex editor / disk editor / RAM editor with support for huge file sizes ("if it fits on a disk/drive you can open it"). Features include: fast find and replace, checksum calculation, file shredding, concatenation or splitting of files, statistics, and more.  Screenshot

/windows | Apr 24, 2010

Oh, you didn't need that boot loader, did you? #

It's been years since the outcry over TurboTax writing to the boot track forced them to drop DRM altogether. Inexplicably, AutoCAD continues to use this dangerous copy protection scheme (Macrovision FLEXnet or SafeCast or whatever it's called), writing to sector 32 of the boot track. As ExtremeTech pointed out long ago:
Unfortunately, these "reserved" sectors of the hard drive aren't necessarily a safe place for data. And they're an especially dicey place to keep licensing information ... Data compression utilities, "multiboot" utilities, password protection and encryption software, and sector translation software (which allows older computer systems to accept today's huge hard drives) may also reside in this area.
According to reports, Macrovision DRM has exposed customers to malware attacks, caused TrueCrypt volumes to become unbootable, and destroyed non-Windows partitions. To top it all off, SafeCast is apparently easily circumvented by the "bad guys" anyway.

Autodesk, a humble request: please refrain from making potentially dangerous, low-level changes to your customers' hard drives. It really ruined an otherwise perfectly good week for a friend of mine.

/windows | Apr 24, 2010

Open source hex editor for OS X #

Hex Fiend 2.0 [935k] {S} Hex editor with support for huge (over 100GB) files.  Screenshot

/mac | Apr 22, 2010

Full-screen, portable text editor #

WriteMonkey is a full-screen, portable plain text editor with: customizable colors, fonts, and sounds, statistics, bookmarks, text replacement shortcuts, autosave, history, Markdown/Textile shortcuts and export, and much more. Requires .NET Framework. (via Download Squad)

/windows | Apr 22, 2010

Excellent freeware firewall for Vista and Windows 7 #

Windows 7 Firewall Control 3.5 weighs in under 1.3MB, comes in both 32 and 64 bit versions, and has a portable version to boot! Runs fast and clean, with very little impact on system performance. Offers ingress (inbound) and egress (outbound) filtering. Brings back memories of ZoneAlarm's early, pre-bloat years. Shareware version also available.

/windows | Apr 21, 2010

Changing / spoofing your MAC address in Windows and OS X #

/misc | Apr 21, 2010

4DOS > 4NT > TCC LE #

Take Command Console LE (TCC LE) is a freely available version of the command processor formerly known as 4NT (and before that, 4DOS). It's like cmd.exe on steroids, with command history, built-in file viewer, core scripting language, hundreds of new command options, and much more.

/windows | Apr 21, 2010

OS X Address Book contacts not syncing with iPhone #

mont3's instructions from 2008 helped us resolve a local (USB) Address Book sync issue. I have fixed a number of typographical and procedural errors (including the potentially tragic instructions to "Go to FINDER, LIBRARY, APPLICATION SUPPORT and delete everything in there") and updated the process for Snow Leopard:
  1. Backup. Backup. Backup.
  2. Export all contacts from the OS X Address Book as a vCard (attempting to use an Address Book Archive instead did not resolve the sync issue). Note that groups are lost during this process.
  3. Delete all contacts and groups from the OS X Address Book and then quit Address Book.
  4. Delete ~/Library/Caches/com.apple.Address Book
  5. Delete ~/Library/Preferences/com.apple.AddressBook.*
  6. Delete all files and folders inside of ~/Library/Application Support/AddressBook. Note that images are lost during this process. See mont3's original post for information on saving them first.
  7. Open /Applications/iSync.app and click iSync > Preferences > Reset Sync History... Reset Sync History.
  8. Import the vCard containing all of your contacts into Address Book.
  9. While holding the Command and Option keys (to prevent auto sync), plug in your iPhone. If "iTunes is running in safe mode" message appears, click "Continue" while continuing to hold the Command and Option keys.
  10. Click the iPhone icon under "Devices" and then click the "Info" tab.
  11. Make sure "Sync Address Book contacts" is set to sync "All contacts". For good measure you may want to select "Contacts" under "Replace information on this iPod:" if you don't mind having the Mac overwrite the iPhone on the next sync only.
  12. Click "Sync".

/mac | Apr 21, 2010

Migrate from Windows Mail to Apple Mail #

If you're moving from Vista's Windows Mail to OS X's Mail, look no further than Jürgen Lüthje's eml2mbx to convert your mail folders / eml files to mbox format. Or go the other way with his mbx2eml. Both apps are rather reminiscent of Ulrich Krebs' DbxConv, which converts Outlook Express dbx files to mbox or eml.

UPDATE: See eml to mbx converters

/windows | Apr 21, 2010

Extract strings from raw disk device or image #

After filling a hard drive with zeroes (followed by a quick format), I wanted to ensure that no sensitive data remained. My first thought was to use a disk editor/viewer like iBored or Disk Investigator, but scrolling through millions of blocks gets dull pretty quickly. So I tried piping dd to strings - bingo:
$ sudo dd if=/dev/rdisk2 bs=512 | strings -a
BSD  4.4
pEFI        FAT32   
Non-system disk
Press any key to reboot
Initially, I used sudo kill -s SIGINFO dd_pid to check on dd's progress, but then remembered dcfldd, an enhanced version of dd with a much better progress indicator (among many other improvements):
$ sudo dcfldd if=/dev/rdisk2 bs=512 | strings -a
6144 blocks (3Mb) written.EFI        (
124928 blocks (61Mb) written.
UPDATE: While looking for a hex editor that would search for non-matching patterns, I stumbled upon an even simpler solution: use hexdump or od (syntax is the same for both):
$ hexdump /dev/rdisk2
0000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Duplicate lines are truncated (as indicated by the asterisk) unless the -v option is specified:
$ od -v /dev/rdisk2
0000000 0000 0000 0000 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000

/mac | Apr 14, 2010

Computer simulators and assembly language learning tools #

6502 is an integrated macro assembler, simulator and debugger for 650x microprocessor family. Lets you write, test and run 6502/65c02 programs

emu8086Assembler Visual Builder is a web-based Java applet that allows you to create an assembly language program and to simulate running the program. The simulator shows you the machine cycle - the fetch, decode, execute steps - as each assembly instruction is executed.

CPU Sim is an interactive Java-based CPU simulator for use in introductory computer organization classes. It allows users to design simple computer CPU's at the microcode level and to run machine-language or assembly-language programs on those CPU's through simulation. It can be used to simulate a variety of architectures, including accumulator-based, RISC-like, or stack-based (such as the JVM) architectures.

EASy68K is a 68000 Structured Assembly Language IDE. EASy68K allows you to edit, assemble and run 68000 programs on a Windows PC or Wine.

emu8086 is an 8086 microprocessor emulator with integrated assembler and tutorials for beginners. The emulator runs programs like the real microprocessor in step-by-step mode. It shows registers, memory, stack, variables and flags. All memory values can be investigated and edited by a double click. The instructions can be executed back and forward. emu8086 can create a tiny operating system and write its binary code to a bootable floppy disk. The software package includes several external virtual devices: robot, stepper motor, led display, and traffic lights intersection. Additional devices can be created.

Lia is a simple computer simulator and assembly editor written in pure Java. It enables you to write/edit/save your "assembly" files and then execute the code using the built-in simulator. A list representing the memory contents is available at all times so the programmer can see what is really happening. Full source code included.

TOM Computer SimulatorLogicSim allows you to design and simulate digital logic circuits with logic gates like AND, OR, FlipFlop, etc.

Logisim is an educational tool for designing and simulating digital logic circuits. Circuit layouts can be used as "subcircuits" of other circuits, allowing for hierarchical circuit design. (via David Gould)

MARS (MIPS Assembler and Runtime Simulator) is a lightweight interactive development environment (IDE) for programming in MIPS assembly language, intended for educational-level use with Patterson and Hennessy's Computer Organization and Design.

SPIM is a self-contained simulator that will run MIPS32 assembly language programs. It reads and executes assembly language programs written for this processor. SPIM also provides a simple debugger and minimal set of operating system services. SPIM does not execute binary (compiled) programs.

TOM Computer Simulator is designed for computer architecture and programming courses. TOM stands for Thoroughly Obedient Moron, and consists of a number of components each designed to teach the fundamentals of computing via simple interactions.

/misc | Apr 14, 2010

Memory lane #

Read Creative Computing, Compute!, and more at The Classic Computer Magazine Archive.

/misc | Apr 12, 2010

Grep is greedy; make it less so #

While editing an HTML file in TextWrangler, I needed to find and replace all occurrences of:
<span class="a">text</span>
This regex:
\<span class=\"a\"\>.*\<\/span\>
returned results like:
<span class="a">text</span><span class="b">more text</span>
Using a non-greedy quantifier (the question mark) returned the desired results:
\<span class=\"a\"\>.*?\<\/span\>

UPDATE 1: Wikipedia has more on lazy quantification: "... modern regular expression tools allow a quantifier to be specified as lazy (also known as non-greedy, reluctant, minimal, or ungreedy) by putting a question mark after the quantifier (e.g., <.*?>) ..."

UPDATE 2: To match multiline patterns in TextWrangler's grep, start the expression with (?s). According to the BBEdit-TextWrangler Regular Expression Cheat-Sheet m (multiline) "allows the grep engine to match at ^ and $ after and before at \r or \n" and s (magic dot) "allows . to match \r and \n".

/mac | Apr 11, 2010

Batch download YouTube videos from the command line #

UPDATE: See also Download videos from YouTube, Metacafe, Daily Motion, Vimeo, and more.

youtube-dl 2010.04.04: Written in Python; public domain license; excellent documentation; runs under *nix, OS X, Windows; works with latest YouTube changes. First attempt to batch download in bash returned an error:

$ youtube-dl.sh -o %(title)s.%(ext)s --batch-file urls.txt --best-quality --no-overwrites
-bash: syntax error near unexpected token `('
Just had to enclose the output template in single quotation marks:
$ youtube-dl.sh -o '%(title)s.%(ext)s' --batch-file urls.txt --best-quality --no-overwrites
The URLs for urls.txt were collected from a YouTube playlist webpage like so:
$ grep -o '/watch?v=.\{11\}' source.htm | sed 's/^/http:\/\/youtube.com/' | uniq > urls.txt
with sed prepending http://youtube.com to grep's output, and uniq culling the duplicates.

UPDATE 1: To specify the desired quality format, use -f #, where # is one of the format values shown in the first row below:

fmt value 5 6 34 35 18 22 37 38 83 82 85 84 43 44 45 46 100 101 46 102 13 17
Default container FLV MP4 WebM 3GP
Video Encoding Sorenson H.263 MPEG-4 AVC (H.264) VP8 MPEG-4 Visual
Profile Main Baseline High 3D 3D
Resolution progressive 224p 270p 360p 480p 360p 720p 1080p 2304p 240p 360p 520p 720p 360p 480p 720p 1080p 360p 480p 540p 720p
Max width (pixels) 400 480 640 854 640 1280 1920 4096 854 640 1920 1280 640 854 1280 1920 640 854 1920 1280 176
Max height (pixels) 240 270 360 480 360 720 1080 3072 240 360 520 720 360 480 720 1080 360 480 540 720 144
Bitrate (Mbit/s) 0.25 0.8 0.5 0.8–1 0.5 2–2.9 3–4.3 3.5–5 0.5 2-2.9 0.5 1 2 0.5 2
Audio Encoding MP3 AAC Vorbis AAC
Channels 1–2 2 (stereo) 1
Sampling rate (Hz) 22050 44100 22050
Bitrate (kbit/s) 64 128 96 152 96 152 128 192 128 192

(Above table is from Wikipedia's YouTube entry)

UPDATE 2: Video Backup Fusion offers an easy way to save all upload or playlist URLs to a text file (the download function seems to have broken as of May 2014). Source code and binaries available for Windows, Linux, and Haiku. You may want to uncheck "Show video thumbnails" in Settings and change the "Max. videos in list" value to 5000 (the maximum it will accept). Highlight all results, right click, select "Copy url(s)", and paste into your favorite text editor.

UPDATE 3: It's been almost 5 years since this post, but youtube-dl is still going strong, having just been updated yesterday (2015-03-28). Here are two more neat ideas from a HN thread:

/nix | Apr 11, 2010

Unpack / extract .MSI file contents #

MSI Unpacker extracts files from .MSI installer packages. I used it (along with Universal Extractor) to unpack the Debugging Tools for Windows (required for using Crash Reporter in ERD Commander 2005) as follows:
  1. Downloaded dbg_x86_6.7.05.1.exe
  2. Dropped dbg_x86_6.7.05.1.exe onto UniExtract.exe, which extracted dbg_x86.msi
  3. Dropped dbg_x86.msi onto MSIUnPack.vbs, which extracted the entire Debugging Tools for Windows directory
(Note: While UniExtract does have limited MSI support, it was unable to unpack dbg_x86.msi.)

/windows | Apr 07, 2010

Best organic Japanese tea source on the Net #

Just received our first order of matcha from Yuuki-Cha.com - very impressed with their fast service and exceptional tea. In addition to matcha, they carry sencha, gyokuro, genmaicha, etc, all of which are certified organic by JAS; some also carry USDA organic certification. If you're into Japanese tea, check them out.

/misc | Apr 01, 2010

Subscribe or visit the archives