Cyclic redundancy check

From Wikipedia, the free encyclopedia - View original article

 
Jump to: navigation, search

A cyclic redundancy check (CRC) is an error-detecting code commonly used in digital networks and storage devices to detect accidental changes to raw data. Blocks of data entering these systems get a short check value attached, based on the remainder of a polynomial division of their contents; on retrieval the calculation is repeated, and corrective action can be taken against presumed data corruption if the check values do not match.

CRCs are so called because the check (data verification) value is a redundancy (it expands the message without adding information) and the algorithm is based on cyclic codes. CRCs are popular because they are simple to implement in binary hardware, easy to analyze mathematically, and particularly good at detecting common errors caused by noise in transmission channels. Because the check value has a fixed length, the function that generates it is occasionally used as a hash function.

The CRC was invented by W. Wesley Peterson in 1961; the 32-bit CRC function of Ethernet and many other standards is the work of several researchers and was published in 1975.

Introduction[edit]

CRCs are based on the theory of cyclic error-correcting codes. The use of systematic cyclic codes, which encode messages by adding a fixed-length check value, for the purpose of error detection in communication networks, was first proposed by W. Wesley Peterson in 1961.[1] Cyclic codes are not only simple to implement but have the benefit of being particularly well suited for the detection of burst errors, contiguous sequences of erroneous data symbols in messages. This is important because burst errors are common transmission errors in many communication channels, including magnetic and optical storage devices. Typically an n-bit CRC applied to a data block of arbitrary length will detect any single error burst not longer than n bits and will detect a fraction 1 − 2n of all longer error bursts.

Specification of a CRC code requires definition of a so-called generator polynomial. This polynomial becomes the divisor in a polynomial long division, which takes the message as the dividend and in which the quotient is discarded and the remainder becomes the result. The important caveat is that the polynomial coefficients are calculated according to the arithmetic of a finite field, so the addition operation can always be performed bitwise-parallel (there is no carry between digits). The length of the remainder is always less than the length of the generator polynomial, which therefore determines how long the result can be.

In practice, all commonly used CRCs employ the Galois field of two elements, GF(2). The two elements are usually called 0 and 1, comfortably matching computer architecture.

A CRC is called an n-bit CRC when its check value is n bits. For a given n, multiple CRCs are possible, each with a different polynomial. Such a polynomial has highest degree n, which means it has n + 1 terms. In other words, the polynomial has a length of n + 1; its encoding requires n + 1 bits. Note that most polynomial specifications either drop the MSB or LSB bit, since they are always 1. The CRC and associated polynomial typically have a name of the form CRC-n-XXX as in the table below.

The simplest error-detection system, the parity bit, is in fact a trivial 1-bit CRC: it uses the generator polynomial x + 1 (two terms), and has the name CRC-1.

Application[edit]

A CRC-enabled device calculates a short, fixed-length binary sequence, known as the check value or CRC, for each block of data to be sent or stored and appends it to the data, forming a codeword. When a codeword is received or read, the device either compares its check value with one freshly calculated from the data block, or equivalently, performs a CRC on the whole codeword and compares the resulting check value with an expected residue constant. If the check values do not match, then the block contains a data error. The device may take corrective action, such as rereading the block or requesting that it be sent again. Otherwise, the data is assumed to be error-free (though, with some small probability, it may contain undetected errors; this is the fundamental nature of error-checking).[2]

Data integrity[edit]

CRCs are specifically designed to protect against common types of errors on communication channels, where they can provide quick and reasonable assurance of the integrity of messages delivered. However, they are not suitable for protecting against intentional alteration of data.

Firstly, as there is no authentication, an attacker can edit a message and recompute the CRC without the substitution being detected. When stored alongside the data, CRCs and cryptographic hash functions by themselves do not protect against intentional modification of data. Any application that requires protection against such attacks must use cryptographic authentication mechanisms, such as message authentication codes or digital signatures (which are commonly based on cryptographic hash functions).

Secondly, unlike cryptographic hash functions, CRC is an easily reversible function, which makes it unsuitable for use in digital signatures.[3]

Thirdly, CRC is a linear function with a property that \operatorname{crc}(x \oplus y \oplus z) = \operatorname{crc}(x) \oplus \operatorname{crc}(y) \oplus \operatorname{crc}(z); as a result, even if the CRC is encrypted with a stream cipher that uses XOR as its combining operation (or mode of block cipher which effectively turns it into a stream cipher, such as OFB or CFB), both the message and the associated CRC can be manipulated without knowledge of the encryption key; this was one of the well-known design flaws of the Wired Equivalent Privacy (WEP) protocol.[4]

Computation[edit]

To compute an n-bit binary CRC, line the bits representing the input in a row, and position the (n + 1)-bit pattern representing the CRC's divisor (called a "polynomial") underneath the left-hand end of the row.

In this example, we shall encode 14 bits of message with a 3-bit CRC, with a polynomial x³+x+1. The polynomial is written in binary as the coefficients; a 3rd order polynomial has 4 coefficients (1x³+0x²+1x+1). In this case, the coefficients are 1,0, 1 and 1. The result of the calculation is 3 bits long.

Start with the message to be encoded:

 11010011101100 

This is first padded with zeroes corresponding to the bit length n of the CRC. Here is the first calculation for computing a 3-bit CRC:

 11010011101100 000 <--- input right padded by 3 bits 1011               <--- divisor (4 bits) = x³+x+1 ------------------ 01100011101100 000 <--- result 

The algorithm acts on the bits directly above the divisor in each step. The result for that iteration is the bitwise XOR of the polynomial divisor with the bits above it. The bits not above the divisor are simply copied directly below for that step. The divisor is then shifted one bit to the right, and the process is repeated until the divisor reaches the right-hand end of the input row. Here is the entire calculation:

 11010011101100 000 <--- input right padded by 3 bits 1011               <--- divisor 01100011101100 000 <--- result (note the first four bits are the XOR with the divisor beneath, the rest of the bits are unchanged)  1011              <--- divisor ... 00111011101100 000   1011 00010111101100 000    1011 00000001101100 000 <--- note that the divisor moves over to align with the next 1 in the dividend (since quotient for that step was zero)        1011             (in other words, it doesn't necessarily move one bit per iteration) 00000000110100 000         1011 00000000011000 000          1011 00000000001110 000           1011 00000000000101 000             101 1 ----------------- 00000000000000 100 <--- remainder (3 bits).  Division algorithm stops here as quotient is equal to zero. 

Since the leftmost divisor bit zeroed every input bit it touched, when this process ends the only bits in the input row that can be nonzero are the n bits at the right-hand end of the row. These n bits are the remainder of the division step, and will also be the value of the CRC function (unless the chosen CRC specification calls for some postprocessing).

The validity of a received message can easily be verified by performing the above calculation again, this time with the check value added instead of zeroes. The remainder should equal zero if there are no detectable errors.

 11010011101100 100 <--- input with check value 1011               <--- divisor 01100011101100 100 <--- result  1011              <--- divisor ... 00111011101100 100  ......    00000000001110 100           1011 00000000000101 100             101 1 ------------------                  0 <--- remainder 

Mathematics[edit]

Mathematical analysis of this division-like process reveals how to select a divisor that guarantees good error-detection properties. In this analysis, the digits of the bit strings are taken as the coefficients of a polynomial in some variable x—coefficients that are elements of the finite field GF(2), instead of more familiar numbers. The set of binary polynomials is a mathematical ring.

Designing polynomials[edit]

The selection of generator polynomial is the most important part of implementing the CRC algorithm. The polynomial must be chosen to maximize the error-detecting capabilities while minimizing overall collision probabilities.

The most important attribute of the polynomial is its length (largest degree(exponent) +1 of any one term in the polynomial), because of its direct influence on the length of the computed check value.

The most commonly used polynomial lengths are:

A CRC is called an n-bit CRC when its check value is n-bits. For a given n, multiple CRC's are possible, each with a different polynomial. Such a polynomial has highest degree n, and hence n + 1 terms (the polynomial has a length of n + 1). The remainder has length n. The CRC has a name of the form CRC-n-XXX.

The design of the CRC polynomial depends on the maximum total length of the block to be protected (data + CRC bits), the desired error protection features, and the type of resources for implementing the CRC, as well as the desired performance. A common misconception is that the "best" CRC polynomials are derived from either irreducible polynomials or irreducible polynomials times the factor 1 + x, which adds to the code the ability to detect all errors affecting an odd number of bits.[5] In reality, all the factors described above should enter into the selection of the polynomial and may lead to a reducible polynomial. However, choosing a reducible polynomial will result in a certain proportion of missed errors, due to the quotient ring having zero divisors.

The advantage of choosing a primitive polynomial as the generator for a CRC code is that the resulting code has maximal total block length in the sense that all 1-bit errors within that block length have different remainders (also called syndromes) and therefore, since the remainder is a linear function of the block, the code can detect all 2-bit errors within that block length. If r is the degree of the primitive generator polynomial, then the maximal total block length is 2 ^ {r} - 1 , and the associated code is able to detect any single-bit or double-bit errors.[6] We can improve this situation. If we use the generator polynomial g(x) = p(x)(1 + x), where p(x) is a primitive polynomial of degree r - 1, then the maximal total block length is 2^{r - 1} - 1, and the code is able to detect single, double, triple and any odd number of errors.

A polynomial g(x) that admits other factorizations may be chosen then so as to balance the maximal total blocklength with a desired error detection power. The BCH codes are a powerful class of such polynomials. They subsume the two examples above. Regardless of the reducibility properties of a generator polynomial of degree r, if it includes the "+1" term, the code will be able to detect error patterns that are confined to a window of r contiguous bits. These patterns are called "error bursts".

Specification[edit]

The concept of the CRC as an error-detecting code gets complicated when an implementer or standards committee uses it to design a practical system. Here are some of the complications:

These complications mean that there are three common ways to express a polynomial as an integer: the first two, which are mirror images in binary, are the constants found in code; the third is the number found in Koopman's papers. In each case, one term is omitted. So the polynomial x^4 + x + 1 may be transcribed as:

In the table below they are shown as:

Examples of CRC Representations
NameNormalReversedReversed reciprocal
CRC-40x30xC0x9

Standards and common use[edit]

Numerous varieties of cyclic redundancy checks have been incorporated into technical standards. By no means does one algorithm, or one of each degree, suit every purpose; Koopman and Chakravarty recommend selecting a polynomial according to the application requirements and the expected distribution of message lengths.[7] The number of distinct CRCs in use has confused developers, a situation which authors have sought to address.[5] There are three polynomials reported for CRC-12,[7] sixteen conflicting definitions of CRC-16, and six of CRC-32.[8]

The polynomials commonly applied are not the most efficient ones possible. Between 1993 and 2004, Koopman, Castagnoli and others surveyed the space of polynomials up to 16 bits,[7] and of 24 and 32 bits,[9][10] finding examples that have much better performance (in terms of Hamming distance for a given message size) than the polynomials of earlier protocols, and publishing the best of these with the aim of improving the error detection capacity of future standards.[10] In particular, iSCSI and SCTP have adopted one of the findings of this research, the CRC-32C (Castagnoli) polynomial.

The design of the 32-bit polynomial most commonly used by standards bodies, CRC-32-IEEE, was the result of a joint effort for the Rome Laboratory and the Air Force Electronic Systems Division by Joseph Hammond, James Brown and Shyan-Shiang Liu of the Georgia Institute of Technology and Kenneth Brayer of the MITRE Corporation. The earliest known appearances of the 32-bit polynomial were in their 1975 publications: Technical Report 2956 by Brayer for MITRE, published in January and released for public dissemination through DTIC in August,[11] and Hammond, Brown and Liu's report for the Rome Laboratory, published in May.[12] Both reports contained contributions from the other team. During December 1975, Brayer and Hammond presented their work in a paper at the IEEE National Telecommunications Conference: the IEEE CRC-32 polynomial is the generating polynomial of a Hamming code and was selected for its error detection performance.[13] Even so, the Castagnoli CRC-32C polynomial used in iSCSI or SCTP matches its performance on messages from 58 bits to 131 kbits, and outperforms it in several size ranges including the two most common sizes of Internet packet.[10] The ITU-T G.hn standard also uses CRC-32C to detect errors in the payload (although it uses CRC-16-CCITT for PHY headers).

The table below lists only the polynomials of the various algorithms in use. Variations of a particular protocol can impose pre-inversion, post-inversion and reversed bit ordering as described above. For example, the CRC32 used in both Gzip and Bzip2 use the same polynomial, but Bzip2 employs reversed bit ordering, while Gzip does not.

CRCs in proprietary protocols might use a non-trivial initial value and final XOR for obfuscation but this does not add cryptographic strength to the algorithm. An unknown error-detecting code can be characterized as a CRC, and as such fully reverse engineered, from its output codewords.[14]

See Polynomial representations of cyclic redundancy checks for the algebraic representations of the polynomials for the CRCs below.

NameUsesPolynomial representations
NormalReversedReversed reciprocal
CRC-1most hardware; also known as parity bit0x10x10x1
CRC-4-ITUG.7040x30xC0x9
CRC-5-EPCGen 2 RFID[15]0x090x120x14
CRC-5-ITUG.7040x150x150x1A
CRC-5-USBUSB token packets0x050x140x12
CRC-6-CDMA2000-Amobile networks[16]0x270x390x33
CRC-6-CDMA2000-Bmobile networks[16]0x070x380x23
CRC-6-ITUG.7040x030x300x21
CRC-7telecom systems, G.707, G.832, MMC, SD0x090x480x44
CRC-7-MVBTrain Communication Network, IEC 60870-5[17]0x650x530x72
CRC-80xD50xAB0xEA[7]
CRC-8-CCITTI.432.1; ATM HEC, ISDN HEC and cell delineation0x070xE00x83
CRC-8-Dallas/Maxim1-Wire bus0x310x8C0x98
CRC-8-SAE J1850AES30x1D0xB80x8E
CRC-8-WCDMAmobile networks[16][18]0x9B0xD90xCD[7]
CRC-10ATM; I.6100x2330x3310x319
CRC-10-CDMA2000mobile networks[16]0x3D90x26F0x3EC
CRC-11FlexRay[19]0x3850x50E0x5C2
CRC-12telecom systems[20][21]0x80F0xF010xC07[7]
CRC-12-CDMA2000mobile networks[16]0xF130xC8F0xF89
CRC-13-BBCTime signal, Radio teleswitch[22]0x1CF50x15E70x1E7A
CRC-15-CAN0x45990x4CD10x62CC
CRC-15-MPT1327[23]0x68150x540B0x740A
Chakravartyoptimal for payloads ≤64 bits[17]0x2F150xA8F40x978A
CRC-16-ARINCACARS applications[24]0xA02B0xD4050xD015
CRC-16-CCITTX.25, V.41, HDLC FCS, XMODEM, Bluetooth, PACTOR, SD, many others; known as CRC-CCITT0x10210x84080x8810[7]
CRC-16-CDMA2000mobile networks[16]0xC8670xE6130xE433
CRC-16-DECTcordless telephones[25]0x05890x91A00x82C4
CRC-16-T10-DIFSCSI DIF0x8BB7[26]0xEDD10xC5DB
CRC-16-DNPDNP, IEC 870, M-Bus0x3D650xA6BC0x9EB2
CRC-16-IBMBisync, Modbus, USB, ANSI X3.28, SIA DC-07, many others; also known as CRC-16 and CRC-16-ANSI0x80050xA0010xC002
FletcherUsed in Adler-32 A & B ChecksumsNot a CRC; see Fletcher's checksum
CRC-17-CANCAN FD[27]0x1685B0x1B42D0x1B42D
CRC-21-CANCAN FD[27]0x1028990x1322810x18144C
CRC-24FlexRay[19]0x5D6DCB0xD3B6BA0xAEB6E5
CRC-24-Radix-64OpenPGP, RTCM104v30x864CFB0xDF32610xC3267D
CRC-30CDMA0x2030B9C70x38E743010x30185CE3
Adler-32ZlibNot a CRC; see Adler-32
CRC-32HDLC, ANSI X3.66, ITU-T V.42, Ethernet, Serial ATA, MPEG-2, PKZIP, Gzip, Bzip2, PNG,[28] many others0x04C11DB70xEDB883200x82608EDB[10]
CRC-32C (Castagnoli)iSCSI, SCTP, G.hn payload, SSE4.2, Btrfs, ext40x1EDC6F410x82F63B780x8F6E37A0[10]
CRC-32K (Koopman)0x741B8CD70xEB31D82E0xBA0DC66B[10]
CRC-32Qaviation; AIXM[29]0x814141AB0xD58282810xC0A0A0D5
CRC-40-GSMGSM control channel[30][31]0x00048200090x90004120000x8002410004
CRC-64-ECMAECMA-182, XZ Utils0x42F0E1EBA9EA36930xC96C5795D7870F420xA17870F5D4F51B49
CRC-64-ISOHDLC, Swiss-Prot/TrEMBL; considered weak for hashing[32]0x000000000000001B0xD8000000000000000x800000000000000D

Implementations[edit]

See also[edit]

References[edit]

  1. ^ Peterson, W. W. and Brown, D. T. (January 1961). "Cyclic Codes for Error Detection". Proceedings of the IRE 49 (1): 228–235. doi:10.1109/JRPROC.1961.287814. 
  2. ^ Ritter, Terry (February 1986). "The Great CRC Mystery". Dr. Dobb's Journal 11 (2): 26–34, 76–83. Retrieved 21 May 2009. 
  3. ^ Stigge, Martin; Plötz, Henryk; Müller, Wolf; Redlich, Jens-Peter (May 2006). Reversing CRC – Theory and Practice. Berlin: Humboldt University Berlin. p. 17. Retrieved 4 February 2011. "The presented methods offer a very easy and efficient way to modify your data so that it will compute to a CRC you want or at least know in advance." 
  4. ^ Cam-Winget, Nancy; Housley, Russ; Wagner, David; Walker, Jesse (May 2003). "Security Flaws in 802.11 Data Link Protocols". Communications of the ACM 46 (5): 35–39. doi:10.1145/769800.769823. 
  5. ^ a b c Williams, Ross N. (24 September 1996). "A Painless Guide to CRC Error Detection Algorithms V3.00". Retrieved 5 June 2010. 
  6. ^ Press, WH; Teukolsky, SA; Vetterling, WT; Flannery, BP (2007). "Section 22.4 Cyclic Redundancy and Other Checksums". Numerical Recipes: The Art of Scientific Computing (3rd ed.). New York: Cambridge University Press. ISBN 978-0-521-88068-8. 
  7. ^ a b c d e f g Koopman, Philip; Chakravarty, Tridib (June 2004). "Cyclic Redundancy Code (CRC) Polynomial Selection For Embedded Networks". The International Conference on Dependable Systems and Networks: 145–154. doi:10.1109/DSN.2004.1311885. ISBN 0-7695-2052-9. Retrieved 14 January 2011. 
  8. ^ Cook, Greg (6 July 2012). "Catalogue of parametrised CRC algorithms". Retrieved 7 July 2012. 
  9. ^ Castagnoli, G.; Bräuer, S.; Herrmann, M. (June 1993). "Optimization of Cyclic Redundancy-Check Codes with 24 and 32 Parity Bits". IEEE Transactions on Communications 41 (6): 883. doi:10.1109/26.231911. 
  10. ^ a b c d e f Koopman, Philip (July 2002). "32-Bit Cyclic Redundancy Codes for Internet Applications". The International Conference on Dependable Systems and Networks: 459–468. doi:10.1109/DSN.2002.1028931. ISBN 0-7695-1597-5. Retrieved 14 January 2011. 
  11. ^ Brayer, Kenneth (August 1975). Evaluation of 32 Degree Polynomials in Error Detection on the SATIN IV Autovon Error Patterns. National Technical Information Service. p. 74. Retrieved 3 February 2011. 
  12. ^ Hammond, Joseph L., Jr.; Brown, James E.; Liu, Shyan-Shiang (1975). "Development of a Transmission Error Model and an Error Control Model". Unknown (National Technical Information Service, published May 1975) 76: 74. Bibcode:1975STIN...7615344H. Retrieved 7 July 2012. 
  13. ^ Brayer, Kenneth; Hammond, Joseph L., Jr. (December 1975). "Evaluation of error detection polynomial performance on the AUTOVON channel". Conference Record. IEEE National Telecommunications Conference, New Orleans, La 1. New York: Institute of Electrical and Electronics Engineers. pp. 8–21 to 8–25. Bibcode:1975ntc.....1....8B. 
  14. ^ Ewing, Gregory C. (March 2010). "Reverse-Engineering a CRC Algorithm". Christchurch: University of Canterbury. Retrieved 26 July 2011. 
  15. ^ Class-1 Generation-2 UHF RFID Protocol. 1.2.0. EPCglobal. 23 October 2008. p. 35. Retrieved 4 July 2012.  (Table 6.12)
  16. ^ a b c d e f Physical layer standard for cdma2000 spread spectrum systems. Revision D version 2.0. 3rd Generation Partnership Project 2. October 2005. pp. 2–89–2–92. Retrieved 14 October 2013. 
  17. ^ a b Chakravarty, Tridib (December 2001). Performance of Cyclic Redundancy Codes for Embedded Networks (Thesis). Philip Koopman, advisor. Pittsburgh: Carnegie Mellon University. pp. 5,18. Retrieved 8 July 2013. 
  18. ^ Richardson, Andrew (17 March 2005). WCDMA Handbook. Cambridge, UK: Cambridge University Press. p. 223. ISBN 0-521-82815-5. 
  19. ^ a b FlexRay Protocol Specification. 3.0.1. Flexray Consortium. October 2010. p. 114.  (4.2.8 Header CRC (11 bits))
  20. ^ Perez, A.; Wismer & Becker (1983). "Byte-Wise CRC Calculations". IEEE Micro 3 (3): 40–50. doi:10.1109/MM.1983.291120. 
  21. ^ Ramabadran, T.V.; Gaitonde, S.S. (1988). "A tutorial on CRC computations". IEEE Micro 8 (4): 62–75. doi:10.1109/40.7773. 
  22. ^ Ely, S.R.; Wright, D.T. (March 1982). L.F. Radio-Data: specification of BBC experimental transmissions 1982. Research Department, Engineering Division, The British Broadcasting Corporation. p. 9. Retrieved 11 October 2013. 
  23. ^ A signalling standard for trunked private land mobile radio systems (MPT 1327) (3rd ed.). Ofcom. June 1997. p. 3-3. Retrieved 16 July 2012.  (3.2.3 Encoding and error checking)
  24. ^ Rehmann, Albert; Mestre, José D. (February 1995). Air Ground Data Link VHF Airline Communications and Reporting System (ACARS) Preliminary Test Report. Federal Aviation Authority Technical Center. p. 5. Retrieved 7 July 2012. 
  25. ^ ETSI EN 300 175-3. V2.2.1. Sophia Antipolis, France: European Telecommunications Standards Institute. November 2008. 
  26. ^ Thaler, Pat (28 August 2003). 16-bit CRC polynomial selection. INCITS T10. Retrieved 11 August 2009. 
  27. ^ a b CAN with Flexible Data-Rate Specification. 1.0. Robert Bosch GmbH. April 17, 2012. p. 13.  (3.2.1 DATA FRAME)
  28. ^ Boutell, Thomas; Randers-Pehrson, Glenn; et al. (14 July 1998). "PNG (Portable Network Graphics) Specification, Version 1.2". Libpng.org. Retrieved 3 February 2011. 
  29. ^ AIXM Primer. 4.5. European Organisation for the Safety of Air Navigation. 20 March 2006. Retrieved 4 July 2012. 
  30. ^ Gammel, Berndt M. (31 October 2005). Matpack documentation: Crypto - Codes. Matpack.de. Retrieved 21 April 2013.  (Note: MpCRC.html is included with the Matpack compressed software source code, under /html/LibDoc/Crypto)
  31. ^ Geremia, Patrick (April 1999). Cyclic redundancy check computation: an implementation using the TMS320C54x (SPRA530). Texas Instruments. p. 5. Retrieved 4 July 2012. 
  32. ^ Jones, David T. An Improved 64-bit Cyclic Redundancy Check for Protein Sequences. University College London. Retrieved 15 December 2009. 

External links[edit]