Magic numbers in files

Examples

Some examples:

* Compiled Java class files (bytecode) start with hex CAFEBABE. When compressed with Pack200 the bytes are changed to CAFED00D.
* GIF image files have the ASCII code for "GIF89a" (47 49 46 38 39 61) or "GIF87a" (47 49 46 38 37 61)
* JPEG image files begin with FF D8 and end with FF D9. JPEG/JFIF files contain the ASCII code for "JFIF" (4A 46 49 46) as a null terminated string. JPEG/Exif files contain the ASCII code for "Exif" (45 78 69 66) also as a null terminated string, followed by more metadata about the file.
* PNG image files begin with an 8-byte signature which identifies the file as a PNG file and allows detection of common file transfer problems: \211 P N G \r \n \032 \n (89 50 4E 47 0D 0A 1A 0A). That signature contains various newline characters to permit detecting unwarranted automated newline conversions, such as transferring the file using FTP with the ASCII transfer mode instead of the binary mode.
* Standard MIDI music files have the ASCII code for "MThd" (4D 54 68 64) followed by more metadata.
* Unix script files usually start with a shebang, "#!" (23 21) followed by the path to an interpreter.
* PostScript files and programs start with "%!" (25 21).
* PDF files start with "%PDF" (25 50 44 46).
* Old MS-DOS .exe files and the newer Microsoft Windows PE (Portable Executable) .exe files start with the ASCII string "MZ" (4D 5A), the initials of the designer of the file format, Mark Zbikowski. The definition allows "ZM" (5A 4D) as well but this is quite uncommon.
* The Berkeley Fast File System superblock format is identified as either 19 54 01 19 or 01 19 54 depending on version; both represent the birthday of the author, Marshall Kirk McKusick.
* The Master Boot Record of bootable storage devices on almost all IA-32 IBM PC Compatibles has a code of AA 55 as its last two bytes.
* Executables for the Game Boy and Game Boy Advance handheld video game systems have a 48-byte or 156-byte magic number, respectively, at a fixed spot in the header. This magic number encodes a bitmap of the Nintendo logo.
* Zip files begin with "PK" (50 4B), the initials of Phil Katz, author of DOS compression utility PKZIP.

Tips: TCPDump

TCPDump prints our the headers of packets on a network interface that match the boolean expression. You can also use with option -w to save the packet data to a file for later analysis. You can also save the file with the .pcap extension so you can use either wireshark or other packet sniffing program.

TCPDUMP.org

Examples: tcpdump -i eth1 -XxvvnneS host 192.168.1.1 -w host_168.1.1.pcap

-i followed by sniffing interface name (eth0, eth1 , etc.)
-X when printing in hex, print ascii too
-x Print each packet in hex
-vv even more verbose output
-nn do not resolve host name and port service
-e Print the link-level header on each dump line
-S Print absolute, rather than relative, TCP sequence numbers


Few tips from TaoSecurity...


Understanding Tcpdump's -d Option, Part 2

In September I referenced a post by libpcap guru Guy Harris explaining outfrom from Tcpdump's -d switch. After looking at the original 1992 BSD Packet Filter (.pdf) paper and the subsequent 1999 BPF+ (.ps) paper, I understand the syntax for the compiled packet-matching code generated by the tcpdump -d switch. For example:

fedorov:/usr/local/etc/nsm# tcpdump -n -i em1 -d tcp

tcpdump: WARNING: em1: no IPv4 address assigned

(000) ldh      [12]

(001) jeq      #0x86dd          jt 2    jf 4

(002) ldb      [20]

(003) jeq      #0x6             jt 7    jf 8

(004) jeq      #0x800           jt 5    jf 8

(005) ldb      [23]

(006) jeq      #0x6             jt 7    jf 8

(007) ret      #96

(008) ret      #0

Here is what each instruction means:

• 000 says load (using 'ldh') the "half word" or two bytes starting at offset 12 of the Ethernet header. Since we begin counting at 0, bytes 0 to 5 are the destination MAC address and bytes 6 to 11 are the source MAC address. The name of the two bytes beginning at offset 12 differs according to the Ethernet format used.


• 001 compares the two bytes loaded in 000 with the value 0x86dd. That is the Ethertype of IPv6. A comparison is made (using 'jeq'); if equality is true, jump ('jt') to instruction 002. If false, jump ('jf') to 004.


• 002 loads the byte found at offset 20. If we are evaluating this instruction we are in an IPv6 header. Offset 20 holds the "next header" value.


• 003 compares the byte loaded in 002 with the value 0x6. This is the IP protocol code for TCP. A comparison is made (using 'jeq'); if equality is true, jump ('jt') to instruction 007. If false, jump ('jf') to 008.


• 004 compares the byte loaded in 000 with the value 0x800. That is the Ethertype of IPv4. A comparison is made (using 'jeq'); if equality is true, jump ('jt') to instruction 005. If false, jump ('jf') to 008.


• 005 loads the byte found at offset 23. If we are evaluating this instruction we are in an IPv4 header. Offset 20 holds the "protocol" value for the protocol following the IP header.


• 006 compares the byte loaded in 005 with the value 0x6. That is the protocol value for TCP. A comparison is made (using 'jeq'); if equality is true, jump ('jt') to instruction 007. If false, jump ('jf') to 008.


• 007 is the equivalent of "TRUE", meaning that the indicated number of bytes (96) of packet data will be copied to the calling application (in this case, Tcpdump). You reach this point if the packet being inspected is TCP, either using IPv4 or IPv6.


• 008 is the equivalent of "FALSE", meaning zero bytes of packet data will be copied to the application. You reach this point if the packet being inspected is not TCP.

Understanding this syntax is a way to troubleshoot BPFs that don't behave as you expect. You can run 'tcpdump -d' and inspect the code as explained above to see if it performs as you want.

For those of you wanting a definition of a packet filter, here is what I've come up with based on the original paper, The Packet Filter: An Efficient Mechanism for User-level Network Code (.pdf): a packet filter is a kernel-resident packet demultiplexer that provides a way for userland processes to tell the kernel what packets they want. For more detail, I recommend reading the three papers mentioned in this story. Guy Harris also posted a message to tcpdump-workers explaining BPF.