tar (computing)

From Wikipedia, the free encyclopedia - View original article

tar
Filename extension.tar
Internet media typeapplication/x-tar
Uniform Type Identifierpublic.tar-archive
org.gnu.gnu-tar-archive
Magic numberu s t a r \0 0 0 at byte 257 ("ustar" followed by a null byte followed by two digits '0', 8 bytes in total)
Type of formatFile archiver
Open format?Yes
 
  (Redirected from Tar (file format))
Jump to: navigation, search
tar
Filename extension.tar
Internet media typeapplication/x-tar
Uniform Type Identifierpublic.tar-archive
org.gnu.gnu-tar-archive
Magic numberu s t a r \0 0 0 at byte 257 ("ustar" followed by a null byte followed by two digits '0', 8 bytes in total)
Type of formatFile archiver
Open format?Yes

In computing, tar (derived from tape archive) is both a file format (in the form of a type of archive bitstream) and the name of a program used to handle such files. The format was created in the early days of Unix and standardized by POSIX.1-1988[1] and later POSIX.1-2001.[2]

Initially developed to write data to sequential I/O devices for tape backup purposes, tar is now commonly used to collect many files into one larger file for distribution or archiving, while preserving file system information such as user and group permissions, dates, and directory structures.[3]

Rationale[edit]

Many historic tape drives read and write variable-length data blocks, leaving significant wasted space on the tape between blocks (for the tape to physically start and stop moving). Some tape drives (and raw disks) only support fixed-length data blocks. Also, when writing to any medium such as a filesystem or network, it takes less time to write one large block than many small blocks. Therefore, the tar command writes data in blocks of many 512 byte records. The user can specify a blocking factor, which is the number of records per block; the default is 20, producing 10 kilobyte blocks (which was large when UNIX was invented, but now seems rather small).[4]

Format details[edit]

A tar archive consists of a series of file objects. Each file object includes any file data, and is preceded by a 512-byte header record. The file data is written unaltered except that its length is rounded up to a multiple of 512 bytes and the extra space is filled with zero bytes. The end of an archive is marked by at least two consecutive zero-filled records. (The origin of tar's record size appears to be the 512-byte disk sectors used in the Version 7 Unix file system.) The final block of an archive is padded out to full length with zero bytes.

File header[edit]

The file header record contains metadata about a file. To ensure portability across different architectures with different byte orderings, the information in the header record is encoded in ASCII. Thus if all the files in an archive are text files, and have ASCII names, then the archive is essentially an ASCII text file (containing many NUL characters).

Subsequent extensions have diluted this original attribute (which was never particularly valuable, due to the presence of NULs, which are often poorly handled by text-manipulation programs).

The fields defined by the original Unix tar format are listed in the table below. The link indicator/file type table includes some modern extensions. When a field is unused it is filled with NUL bytes. The header is padded with NUL bytes to make it fill a 512 byte record.

Pre-POSIX.1-1988 tar header:

Field OffsetField SizeField
0100File name
1008File mode
1088Owner's numeric user ID
1168Group's numeric user ID
12412File size in bytes (octal basis)
13612Last modification time in numeric Unix time format (octal)
1488Checksum for header record
1561Link indicator (file type)
157100Name of linked file

The pre-POSIX.1-1988 Link indicator field can have the following values:

Link indicator field
ValueMeaning
'0' or (ASCII NUL)Normal file
'1'Hard link
'2'Symbolic link

Some pre-POSIX.1-1988 tar implementations indicated a directory by having a trailing slash (/) in the name.

Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final NUL or space character should be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files. To overcome this limitation, star in 2001 introduced a base-256 coding that is indicated by setting the high-order bit of the leftmost byte of a numeric field. GNU-tar and BSD-tar followed this idea. Additionally, versions of tar from before the first POSIX standard from 1988 pad the values with spaces instead of zeroes.

The checksum is calculated by taking the sum of the unsigned byte values of the header record with the eight checksum bytes taken to be ascii spaces (decimal value 32). It is stored as a six digit octal number with leading zeroes followed by a NUL and then a space. Various implementations do not adhere to this format. For better compatibility, ignore leading and trailing whitespace, and get the first six digits. In addition, some historic tar implementations treated bytes as signed. Implementations typically calculate the checksum both ways, and treat it as good if either the signed or unsigned sum matches the included checksum.

Unix filesystems support multiple links (names) for the same file. If several such files appear in a tar archive, only the first one is archived as a normal file; the rest are archived as hard links, with the "name of linked file" field set to the first one's name. On extraction, such hard links should be recreated in the file system.

UStar format[edit]

Most modern tar programs read and write archives in the UStar (Uniform Standard Tape ARchive) format, introduced by the POSIX IEEE P1003.1 standard from 1988. It introduced additional header fields. Older tar programs will ignore the extra information, while newer programs will test for the presence of the "ustar" string to determine if the new format is in use. The UStar format allows for longer file names and stores extra information about each file.

Field OffsetField SizeField
0156(as in old format)
1561Type flag
157100(as in old format)
2576UStar indicator "ustar"
2632UStar version "00"
26532Owner user name
29732Owner group name
3298Device major number
3378Device minor number
345155Filename prefix

The Type flag field can have the following values:

Type flag field
ValueMeaning
'0' or (ASCII NUL)Normal file
'1'Hard link
'2'Symbolic link
'3'Character special
'4'Block special
'5'Directory
'6'FIFO
'7'Contiguous file
'g'global extended header with meta data (POSIX.1-2001)
'x'extended header with meta data for the next file in the archive (POSIX.1-2001)
'A'–'Z'Vendor specific extensions (POSIX.1-1988)
All other valuesreserved for future standardization

POSIX.1-1988 vendor specific extensions using link flag values 'A'..'Z' partially have a different meaning with different vendors and thus are seen outdated and replaced by the POSIX.1-2001 extensions that include a vendor tag.

Uses[edit]

Tarpipe[edit]

A tarpipe is the process of creating a tar archive on stdout and piping the archive byte stream to another tar process, working in another directory, and unpacking the archive received on stdin. This copies the entire source directory tree including all special files, such as symlinks or character or block device files. For example:

 tar -c "$srcdir" | tar -C "$destdir" -xv    (or) tar c "$srcdir" | tar xvC "$destdir" 

Software distribution[edit]

The tar format continues to be used extensively in Unix, Linux and OS X. Linux versions use features prominently in various software distributions, with most software source code made available in .tar.gz (compressed tar archives). The tar format is a basis of some types of containment and distribution.

Problems and limitations[edit]

The original tar format was created in the early days of UNIX, and despite current widespread use, many of its design features are considered dated.[5]

Many old tar implementations (such as GNU tar) do not record extended attributes (xattrs) or ACLs. In 2001, star introduced support for ACLs and extended attributes. Later, major Linux distributions created their own patched versions of GNU tar that fully support ACL.[citation needed]

In 1997, Sun proposed a method for adding extensions to the tar format. This method was later accepted for the POSIX.1-2001 standard. This format is known as extended tar-format or pax-format. The new tar format allows users to add any type of vendor-tagged vendor-specific enhancements. The following enhancement tags are defined by the POSIX standard:

In 2001, Star became the first tar to support the new format. In 2004, GNU tar supported the new format.

Other formats have been created to address the shortcomings of tar. These formats include DAR (Disk Archiver) and rdiff-backup(see Duplicity branch of the Savannah software site). However, these formats are not part of any official standard.

Operating system support[edit]

Unix-like operating systems usually include tools to support tar files, as well as utilities commonly used to compress them, such as gzip and bzip2. There are multiple third party tools available for Microsoft Windows to read and write these formats.

Tarbomb[edit]

A tarbomb is derogatory hacker slang used to refer to a tar file that does not follow the usual conventions, e.g., it contains many files that extract into the working directory. Such a tar file can create problems by overwriting files of the same name in the working directory, or mixing one project's files into another. It is at best an inconvenience to the user, who is obliged to identify and delete a number of files scattered throughout the directory's contents. Such behavior is considered bad etiquette on the part of the archive's creator.

A related problem is the use of absolute paths or parent directory references when creating tar files. Files extracted from such archives will often be created in unusual locations outside the working directory and, like a tarbomb, have the potential to overwrite existing files. However, modern versions of GNU tar by default refuse to create or extract absolute paths and parent-directory references (unless it is explicitly allowed with -P or --absolute-names flag). The bsdtar program (which is available on a number of operating systems and is the default "tar" on Mac OS X v10.6), also by default refuses to follow parent-directory references or symlinks.[6]

A command line user can avoid both of these problems by first examining a tar file with one of the following commands:

tar -tf archive.tar
tar tf archive.tar
tar --list --file=archive.tar

These commands do not extract any files, but display the names of all files in the archive. If any are problematic, the user can create a new empty directory and extract the archive into it—or avoid the tar file entirely. Most graphical tools can display the contents of the archive before extracting them.

Random access[edit]

Another weakness of the tar format compared to other archive formats (like DAR (Disk Archiver)) is that there is no centralized location for the information about the contents of the file (a "table of contents" of sorts). So to list the names of the files that are in the archive, one must read through the entire archive and look for places where files start. Also, to extract one small file from the archive, instead of being able to look up the offset in a table and go directly to that location, like other archive formats, with tar, one has to read through the entire archive, looking for the place where the desired file starts. For large tar archives, this causes a big performance penalty, making tar archives unsuitable for situations that often require random access of individual files.

The possible reason for not using a centralized location of information is that tar was originally meant for tapes, which are bad at random access anyway: if the Table Of Contents (TOC) were at the start of the archive, creating it would mean to first calculate all the positions of all files, which needs doubled work, a big cache, or rewinding the tape after writing everything to write the TOC. On the other hand, if the TOC were at the end-of-file (as is the case with ZIP files, for example), reading the TOC would require that the tape be wound to the end, also taking up time and degrading the tape by excessive wear and tear. Compression further complicates matters; as calculating compressed positions for a TOC at the start would need compression of everything before writing the TOC, a TOC with uncompressed positions is not really useful (since one has to decompress everything anyway to get the right positions) and decompressing a TOC at the end of the file might require decompressing the whole file anyway, too.

But today there are a number of add-on utilities which implement tar file indexing, thus enabling random access, both for raw tar files and for tar files compressed with gzip (which is amenable to indexing). Such an index can be kept in a separate file, appended or prepended to the archive file.

Duplicated files in archive[edit]

Another issue with tar format is that it allows several (possibly different) files in archive to have identical path and filename. When extracting such archive, usually the latter version of a file overwrites the former.

This can be even used to create non-explicit (unobvious) tarbomb, which technically contain no files with absolute paths or referring parent directories, but still causes overwriting files outside current directory (for example, archive may contain two files with the same path and filename, first of which is symlink to some location outside current directory, and second of which is a regular file; then extracting such archive on some tar implementations may cause writing to the location pointed by symlink).

Key implementations[edit]

There have been historically many implementations of tar, and many general file archivers have at least partial support for tar (often using one of the implementations below). Most tar implementations can also read and create cpio and pax (the latter actually is a tar-format with POSIX-2001-extensions).

Additionally, most pax implementations can read and create many types of tar files.

Naming of compressed tar files[edit]

tar archive files usually have the extension .tar, as in somefile.tar. The pun, tarball,[8] is used to refer to a tar file.

A tar archive file contains uncompressed byte streams of the files which it contains. To achieve archive compression, a variety of compression programs are available, such as gzip, bzip2, xz, lzip, lzma, or compress, which compress the entire tar archive. Typically, the compressed form of the archive receives a filename by appending the format-specific compressor suffix to the archive file name. For example, a tar archive archive.tar, is named archive.tar.gz, when it is compressed by gzip.

Popular tar programs like the BSD and GNU versions of tar support the command line options Z (compress), z (gzip), and j (bzip2) to automatically compress or decompress the archive file upon creation or unpacking. GNU tar from version 1.20 onwards also supports --lzma (LZMA). 1.21 also supports lzop via --lzop, 1.22 adds support for xz via --xz or -J, and 1.23 adds support for lzip via --lzip.

MS-DOS's 8.3 filename limitations, resulted in additional conventions for naming compressed tar archives. (This practice has declined with FAT offering long filenames.)

Tar archiving is often used together with a compression method, such as gzip, to create a compressed archive. As shown, the combination of the files in the archive are compressed as one unit.
Equivalent File Suffixes
ShortLong
.tgz.tar.gz
.tbz, .tbz2 & .tb2.tar.bz2
.taz.tar.Z
.tlz.tar.lz & .tar.lzma
.txz.tar.xz

See also[edit]

References[edit]

  1. ^ "IEEE Std 1003.1-1988, IEEE Standard for Information Technology - Portable Operating System Interface (POSIX)"
  2. ^ "IEEE Std 1003.1-2001, IEEE Standard for Information Technology - Portable Operating System Interface (POSIX)"
  3. ^ "About the TAR format'" GitHub. Retrieved 28 November 2013.
  4. ^ "Blocking'" ftp.gnu.org. Retrieved 28 November 2013.
  5. ^ Proposed format to replace tar, by the Duplicity utility's developers.
  6. ^ Man page for "bsdtar", as provided by Apple.
  7. ^ Preston, W. Curtis (December 15, 1999). "Free Backup Utilities". Unix Backup and Recovery. Sebastopol, CA: O'Reilly Media. pp. 142–143. ISBN 1-56592-642-0. Retrieved May 28, 2011. 
  8. ^ Jargon File, tarball

External links[edit]