archive formats (zip, tar, gz, 7z, rar, iso, cab)

programmingops culturesecurity

Container formats that bundle files together, compress them, or both - and the distinction between those two jobs is the thing most people never learn.

Archiving and compressing are different jobs. tar bundles files and compresses nothing; gzip compresses a stream and knows nothing about files. ZIP does both, which is why it can extract one file instantly and a .tar.gz cannot.

The first thing to understand is that archiving and compressing are two different jobs, and the Unix world keeps them separate while the Windows world does not. tar is a tape archiver: it concatenates many files into one, preserving names, permissions, ownership and directory structure, and it compresses nothing at all. gzip compresses a single stream and knows nothing about files, names or directories. Put them together and you get .tar.gz, the two-step that trips up everyone coming from the other tradition: tar makes the many into one, gzip makes the one smaller. This is why you cannot extract a single file from a .tar.gz without decompressing everything before it, and why a .zip can pull one file out of a thousand instantly. ZIP does both jobs in one format, compressing each file separately and keeping an index at the end, which costs a little compression efficiency and buys random access. Neither design is wrong; they were solving different problems, and the difference still shows up every time somebody waits four minutes to extract one configuration file from a large tarball. The rest of the family divides along the same lines. 7z is a container with strong compression, usually LZMA, open-source tooling and a genuine efficiency advantage on large similar files, because it can compress across file boundaries in what is called a solid archive - the same trick that makes it slow at extracting one item. RAR is proprietary: the decompressor is documented and free to implement, the compressor is not, so only the vendor's software can create one, which is unusual in a world of open formats and is the reason RAR persists in some communities and is absent from others. ISO is not compression at all. It is a sector-by-sector image of an optical disc, a filesystem in a file, which is why mounting one shows a directory tree rather than needing extraction, and why an ISO of a 700 MB disc is 700 MB whether the disc was full or nearly empty. CAB is Microsoft's cabinet format, used inside installers and driver packages, and most people only ever meet it indirectly. On the utilities: 7-Zip is free, open source and written substantially by one person, Igor Pavlov, and it reads nearly everything while writing 7z and ZIP well; WinRAR is commercial, is the only thing that creates RAR archives, and has its own entry here for reasons that have nothing to do with technology; WinZip was the commercial standard of the 1990s and is the reason a generation says zip as a verb; PowerArchiver, PeaZip, Bandizip and others occupy the same space with different trade-offs between price, format coverage and how much they try to sell you. The formats above are the survivors, and the era before them is worth recovering, because one of its requirements has vanished so completely that the software built around it went too. In the DOS and bulletin-board years the competition was fierce and the names are half-forgotten: ARJ, written by Robert Jung, whose name it carries; LHA and its LZH format, from Haruyasu Yoshizaki in Japan, which was free to use and consequently ended up embedded in a great many other products; ZOO; and PKZIP defending the ground it had taken from ARC. Compression ratios were argued over the way frame rates are now, and the differences were real, because a file that took nine minutes to download at 2400 baud instead of eleven was a meaningful improvement. But the feature that decided which archiver you actually used was multi-volume archiving: splitting one archive across several floppy disks, each of which had to be individually reliable, in a period when disks failed constantly. ARJ was very good at this, which is why so many people who used it remember it fondly and cannot quite say what it did better - it did the thing that made a 20 MB file movable at all when nothing you owned could hold 20 MB. That requirement disappeared entirely, first to writable optical discs and then to networks, and the formats optimised for it disappeared with it. They were not defeated on compression or on licensing. The problem they existed to solve stopped being a problem, which is a more complete kind of ending than losing. Two security notes belong in any honest description. Archives are an attack surface: a zip bomb is a small file that expands to something enormous, designed to exhaust whatever is scanning it, and path traversal in archives - extracting a file whose stored name contains ../ and lands outside the target directory - has produced a long line of vulnerabilities across many extraction libraries. Encrypted archives are also the oldest way to move a payload past a scanner that cannot see inside them. If you are writing code that extracts archives supplied by anyone else, treat the filenames inside as hostile input, because they are.

Also known as: zip, tar, tar.gz, tarball, gzip, 7z, rar, iso, cab, compressed archive

← All glossary entries