Backing up a large Gmail / G Suite account #

is theoretically possible via Google Takeout, but there are issues:

timeouts during large downloads: "Failed - Needs authorization", "Failed - Network error", etc
all email dumped into a single MBOX file by default^* (albeit with labels saved in the non-standard "X-Gmail-Labels" header, which can be queried in Thunderbird, for example) or optionally into multiple MBOX files based on labels (resulting in large numbers of duplicate messages)

Got Your Back (GYB) does not suffer from these issues; messages are downloaded as individual EML files, sorted into folders by year, month, and day:

$ gyb --email name@example.com --action estimate

Select the actions you wish GYB to be able to perform for name@example.com

[ ]  0)  Gmail Backup And Restore - read/write mailbox access
[ ]  1)  Gmail Backup Only - read-only mailbox access
[ ]  2)  Gmail Restore Only - write-only mailbox access and label management
[*]  3)  Gmail Full Access - read/write mailbox access and message purge
[ ]  4)  No Gmail Access

[ ]  5)  Groups Restore - write to G Suite Groups Archive
[*]  6)  Storage Quota - Drive app config scope used for --action quota

      7)  Continue
7

Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth...

If your browser is on a different machine then press CTRL+C and
create a file called nobrowser.txt in the same folder as GYB.

Authentication successful.
Got 67100 Message IDs                                                           
GYB needs to examine 67100 messages
GYB already has a backup of 0 messages
GYB needs to estimate 67100 messages
Estimated size 26.47gb 67100/67100 messages 

$ gyb --email name@example.com --action backup

Using backup folder GYB-GMail-Backup-name@example.com
Got 67100 Message IDs                                                           
GYB needs to examine 67100 messages
GYB already has a backup of 0 messages
GYB needs to backup 67100 messages
backed up 67100 of 67100 messages                                               

GYB needs to refresh 0 messages

If desired, import the collection into Thunderbird via ImportExportTools.

If you need to copy the massive collection of EML files, tar it without compression to save a LOT of time (pv is of course optional, but very handy when tarring tens of thousands of files):

$ tar -c ./GYB-GMail-Backup-user\@example.com/ | pv -s $(gdu -sb ./GYB-GMail-Backup-user\@example.com/ | awk '{print $1}') > user\@example.com.tar
4.03GiB 0:00:23 [ 177MiB/s] [====>                             ] 15% ETA 0:02:08

^* Attempting to split a ~26GB MBOX file by label via mbox_split.py or WB Optimum's script on which it is based did not avail.

UPDATE: Royce Williams kindly informs us of outtake, which "sync[s] Gmail to maildir...quickly", "uses the Gmail API to efficiently sync only deltas", and "can also be limited to a specific label". Crafted in Go by Google software engineer, Daniel Margolis.

Evolution is swell for browsing and searching Maildir directories; add via Edit > Preferences > Mail Accounts > Add > Next > enter required info and deselect "Look up mail server details based on the entered e-mail address" > Next > for "Server Type:" select "Maildir-format mail directories" > select the "Mail Directory:" location, etc.
maildir2mbox converts maildir to mbox, handling nested folders and filesystem encoding problems.

/misc | Sep 25, 2018

Subscribe or visit the archives.