An Introduction to NTFS (New Technology File System)

Web Presentation and Text are Copyright©2018 by Daniel B. Sedory
(NOT to be reproduced in any form without Permission of the Author !)


We would caution anyone who needs accurate data concerning exactly what happens under a specific OS
or how data is in fact stored, to not only compare multiple sources, but also consult data from someone
who's actually debugged running code, since it will either work correctly or not!   You should, of course,
also perform your own verification of the information before considering it truly reliable; everyone makes
mistakes. If you wish to discuss NTFS, or any other topic, you can email us here.


Note: This page is still under construction . . . .
We are working on EXAMPLES to show how this data is actually used!





Purpose

Our main reason for publishing these notes on NTFS, and indeed much of the information here at The Starman's Realm, is an attempt to alleviate some of the confusion in the forensic community (and others) caused by typo errors, inaccurate statements[1] and even blunders regarding what is available online and in print concerning NT file systems[2] and other relevant data. Furthermore, we believe in using real-world data examples in order to explain a topic. Simply learning 'terminology' or listening to or reading someone's words as they attempt to teach a complex subject can keep many from truly understanding.

First, a few definitions (some words being necessary; others not so much) to help in understanding NTFS:

 

Definition of "File"

A file on a computer drive is essentially an artificial container for holding some kind of information. A file's boundaries (within a larger container often called a volume) are kept track of by its file system.

A file system (NTFS being one of a few well-known types; including FAT32 and ext4) must, of course, also store that boundary information (and any other associated data; such as a file's name and when it was created) somewhere on the same disk drive.  NTFS stores this metadata within that same volume.

Definition of "Metadata"

Metadata refers to data that provides information about other data. In the physical world, the old 'index cards' of a library that could be used to find a book on its shelves (according to the Dewey Decimal, LC or some other classification) also contained a great deal of metadata about each book: The book's author, when it was published, by what printing house(s), how many pages that edition had, etc. etc. A 'list of lists' is another example of metatdata. Metadata is most often found in a structured format which allows one to more easily retrieve some sought after data or to manage data resources.

 

The NT File System (NTFS)

Everything is a File!

One of the interesting things about NTFS is that even the instructions and system data used to manage the contents of its file system are also stored as files within its volumes; everything is a file in NTFS. So, NTFS volumes have meta files within them:

Even the NTFS Boot Record is a File!

In fact, even the 16 sectors of an NTFS partition's Volume Boot Record (or VBR); which always begins at Logical Sector zero (0), is a meta file under NTFS (its file name is: $Boot).

The files in which the structured metadata of a Windows™ NTFS volume are stored each have a dollar-sign ($) as their first character. Most users are unaware of this, because these files are kept hidden from them: Even if the Microsoft® Windows™ OS's "Folder Options" -> "View" tab settings are set to show both hidden and system files, users will never see any of these meta files! (Note: The '$-sign' character has nothing to do with how the OS hides these files; users can easily create a file with a $-sign as the first character.) Only a disk editor such as WinHex (download and use WinHex from here for free with limited functionality[3]) or some other forensic tool designed to do so, will allow you to easily view these files. One could of course search for and view the 'raw' bytes of these metafiles using the free hex and disk editor HxD, but that would require both the necessary experience and more time to find them. Here's a screenshot of a partition 'opened' in WinHex with only some metafile names showing in the partition's root directory:

In the window above, we've highlighted a few useful items inside the WinHex display for our study of meatafiles:

1) We point out again that the $Boot file always begins at Logical Sector 0 of the volume / partition.
2) The first 4 bytes of every MFT record entry should always be the ASCII characters: "FILE"[4].

3) When viewing the $MFT file, WinHex shows the record/entry number in focus; note the "(#0)" next to the $MFT file name (in the dark gray area of the display).

4) For every file (or directory) entry in the MFT, its Unicode name technically begins 2 bytes preceding (i.e., ahead of or earlier) than its characters: For example, in the display above, where the $MFT name begins at offset 0xF2, the byte at offset 0xF0 (04 in this case) indicates the length of the name; and since it uses only a single byte for its length, that means no Windows NTFS file name can be longer than 255 characters (0xFF = 255). The second byte (03 here) indicates the type of name[5]. It should also be noted that the location of a file name within each MFT record is not always the same! For example, both the $AttrDef and the Root Directory file names in our sample MFT records (see below), begin at offset 0xD8 in MFT record numbers 4 and 5 (see Table 1 below for the file names of the first 16 MFT records): "08 03 24 00 41 00 74 00 74 00 72 00 44 00 65 00 66 00 ..$.A.t.t.r.D.e.f." and "01 03 2E 00 ...." (the actual name within the MFT for the Root Directory being a single 'dot' or period; hex: 0x2E). Nor will it always be the same location in different volumes for the same record: For example, in the $MFT file on a different drive, we found its Root folder name ('.'; 0x2E) at offset 0x138. In fact, when the contents of an MFT record changes (which happen every time a file is modified or simply read), any number of its attributes may have their locations shifted within that record.

 

A Brief History of the NTFS Versions

As others have rightfully noted, apart from the version numbers listed in the table below, if we consider all the Windows service packs and updates which have affected the NTFS.SYS file over the course of its history, then "numerous versions of NTFS exist."[6] [In the future, we'll be listing many of the different versions of NTFS.SYS and various details about each file.]

NTFS
Version

When

Operating System

Comments

1.0

July 1993

Windows NT 3.1

 

1.1

May 1995

Windows NT 3.51

 

1.2

July 1996

Windows NT 4.0

Due to the OS version being NT 4.0,
sometimes called NTFS v. 4.0*

3.0

DEC 1999

Windows 2000

Since Win2k was thought of as NT 5,
sometimes called NTFS v. 5.0*

3.1

AUG 2001

Windows XP

Per logic above, this NT version has
often been called NTFS v. 5.1*

* Somewhat understandable, since the same system file on an NT 4, Win 2000 and Win XP
  install had file version numbers: 4.0.13x1.x, 5.00.2195.xxxx and 5.1.2600.x, respectively.
  But these numbers are indicative of the OS version, not the NT file system version.

And it has officially remained at version 3.1 ever since Windows XP was released, even though many new features were added in 2008, and at other times. The fsutil file system management command-line utility, first included with Windows XP, can be used to display a PC's NTFS version by entering: fsutil fsinfo ntfsinfo X: ; where X is a drive letter. The last screen shown here is from a Windows™ 10 machine:

 

  fsutil is a complex utility which will be described in
  more detail below; for both locating an NTFS volume's
  $MFT metafile and also managing some aspects of its
  file system.

 

 

Layout of An NTFS Partition

The $MFT file is normally located near the middle of an NTFS Volume; it can, however, also be fragmented, though rarely will it be split into more than only 2 fragments. It's possible for a good third-party software program to move the location of the $MFT file; for example, when shrinking a partition, or to defragment it. The Windows OS will never move the $MFT file if you attempt to shrink its partition; that's the reason one can rarely, if ever, shrink an NTFS partition by more than half (1/2) of its original size! The $MFTMirr file may be located anywhere within the NTFS Volume, just like any of the other metafiles, system files or user files. The volume's Boot Sector is critical for locating the $MFT file, so a backup copy of this sector is stored at the end of the NTFS partition if needed.

 

The Most Important Metafile (After the VBR)

The most important file within an NTFS partition (apart from any code which helps an operating system manage it) is the $MFT (Master File Table). Why? Because it's basically the master index to the location and size of every other file within the whole volume. In fact, it's so important, there's a backup copy of the MFT's most important records called the $MFTMirr file. (Note that $MFTMirr is not a complete backup copy of the $MFT file! In fact, it contains only the $MFT, $MFTMirr, $LogFile and $Volume entries; which are the first 4 records of the MFT.)

Note: If a file's size is small enough, NTFS will store its entire contents only within its MFT entry! This would be true, for example, for many .ini files. For 1 KiB NTFS record lengths (the norm), files of about 700 bytes or less are normally found only within the $MFT file. Under NTFS, these are known as resident data attribute records. Many directories which have only a few entries may also be stored as resident records within the MFT. (The various attributes of MFT record entries will be explained further below.)

Note: Each normal file record inside an MFT file, including an entry for the $MFT file itself; which is always the first record (ID # 0), begins with the header: FILE at offsets 00 - 03. And the ID # of the entry is found at offsets 0x2C through 0x2F.

 

The First 16 Entries of the $MFT File

The first 16 records of every NTFS Volume's MFT (Master File Table) are always reserved for the same metafiles and the Root Directory, and are always found in this order:

$MFT
Record
Number

Metafile

Common
Name

Description

0

$MFT

Master File Table

The file which contains all the records of every file on the volume!

1

$MFTMirr

MFT Mirror File

Backup copy of the $MFT file's most important entries:
$MFT, $MFTMirr, $LogFile and $Volume records.

2

$LogFile

Log File

The Log File might be useful in recovering files after power losses or OS lock-ups.

3

$Volume

Volume Data

Among other data, contains the volume's name.

4

$AttrDef

Attribute Definitions

The Attribute Definition Table. See Attributes.

5

      .

Root Directory

Contains an index of all files in the Root folder/directory.
Note: 'Filename' is a single dot (0x2E) character.

6

$Bitmap

NTFS Bitmap

Allocation status of every cluster on the Volume.

7

$Boot

Boot Record

16 sectors always reserved; boot code size is only about half that.

8

$BadClus

Bad Cluster File

Listing of all 'bad clusters' on the volume.

9

$Secure

Security

[Still under construction.]

10

$UpCase

UpperCase Table

[Still under construction.]

11

$Extend

Extend Table

[Still under construction.]

12-15

Reserved by NTFS

Unnamed

These 4 records are reserved by the file system.
Their function(s) are unknown at this time.

 

Header Layout for MFT "FILE" Records

The first 48 bytes of each MFT record, comprise its "header" which contains such vital information as ... .

Offset
in
Record

Size
in
Bytes

Description

0x00

4

Always: FILE (or Magic Number = 0x454C4946;
or as they appear in a disk editor: "46 49 4C 45")

0x04

2

Offset to 'Update Sequence' (from start of FILE header)

0x06

2

Size in WORDs of the 'Update Sequence Number and Array' (USA)

0x08

8

$LogFile Sequence Number (or LSN)

0x10

2

Sequence Number

0x12

2

Hard Link Count

0x14

2

Offset to first Attribute

0x16

2

Flags

0x18

4

Actual Size of this FILE Record.

0x1C

4

Allocated Size of this FILE Record; normally 0x400 (1 KiB; two sectors).

0x20

8

File Reference to Base FILE Record

0x28

2

Next Attribute ID

0x2A

2

Used to align to 4-byte boundary.

0x2C

4

MFT Record Number (since version 3.1; Win XP)

 

Sample $MFT File's Metafile Records

There's a sample $MFT file's first 17 records here; in both binary and hex editor view (text) format. This is from a Windows XP Professional OS install on a real physical disk drive, and includes a BAAD record.

 

Attributes

Within the $MFT file, everything, including all data, is an attribute. The $AttrDef file contains a list of all the attributes available for use in that volume. Each attribute has a type number associated with it which we've highlighted in this $AttrDef Sample File. The attributes found in this file are as follows:

Attribute
ID in Hex
& (Decimal)

ATTRIBUTE

0x10 ( 16)

$STANDARD_INFORMATION

0x20 ( 32)

$ATTRIBUTE_LIST

0x30 ( 48)

$FILE_NAME

0x40 ( 64)

$OBJECT_ID

0x50 ( 80)

$SECURITY_DESCRIPTOR

0x60 ( 96)

$VOLUME_NAME

0x70 (112)

$VOLUME_INFORMATION

0x80 (128)

$DATA

0x90 (144)

$INDEX_ROOT

0xA0 (160)

$INDEX_ALLOCATION

0xB0 (176)

$BITMAP

0xC0 (192)

$REPARSE_POINT

0xD0 (208)

$EA_INFORMATION

0xE0 (224)

$EA

0x100 (256)

$LOGGED_UTILITY_STREAM

Note: ID # 0xF0 is not defined on this volume.

 

 


Footnotes

1[Return to Text] For example, The Complete Guide to Windows Server 2008, John Savill (Pearson Education, 2008), which was invaluable to many admins working with this server, still had this error (on the 2nd page of Chapter 5): "The MFT2 is a mirror of the first 16 records of the MFT, which describe the metadata files themselves and are, therefore, most vital." Well, that number (16) is incorrect. $MFTMirr is a copy of only the first 4 records of the MFT. Whoever wrote '16' probably was thinking about the fact the first 16 records of the MFT are always the same metafile records; and (expected?) that all 16 should be copied. But no one has seen any $MFTMirr that does, even on a Windows 2008 Server.

2[Return to Text] "Systems" plural, because both the linux community and Apple® developers along with a number of commercial software houses like Paragon Technologie GmbH and Tuxera, each have their own versions of NTFS drivers for many different OSs; all of which may be helpful in understanding and/or implementing successful NTFS drivers, but not necessarily true of Microsoft® drivers. For example, one can read about NTFS-3G here and various versions of Apple's NTFS drivers here. Our main focus on this page will be verifiable data concerning the NTFS versions of Microsoft® Windows™.

3[Return to Text] Note: You may evaluate WinHex free of charge for 45 days without licensing it. However, in this mode WinHex will not save files larger than 200 KB, nor write to disk sectors or edit virtual memory; and unfortunately for our purposes here, the automatic color highlighting of different areas within each $MFT record are not functional!
It will also show this reminder:




4[Return to Text] The exception would be if one or more of the records had an unreadable sector(s), in which case, the Windows OS (or more specifically, running an error scan with a utility such as chkdsk) will replace the "FILE" header with a "BAAD" header, and all of its remaining bytes will be zero-filled ('00' bytes only); with the exception of the last two bytes in each sector called the 'Update Sequence Number'. (Note: We've included a BAAD record from a physical drive running Windows XP Professional in our $MFT Sample Metafile Records; it's the last record, MFT # 16, in Sample 1. However, we've found this same record (MFT # 16) in other NTFS volumes to also be a BAAD record; even in Virtual PCs with no physical damage! It is not found in newly created volumes, but has been seen in some after CHKDSK has been run on them. It appears this 'BAAD record' (in MFT # 16) may be used in some way, a kind of 'template' perhaps, for BAAD records? Comments?

5[Return to Text] The name types are: 01 - Indicates a LONG filename, 02 - Indicates a SHORT filename, 03 - Indicates a LONG filename that also meets the requirements for a SHORT filename.

6[Return to Text] John Savill, The Complete Guide to Windows Server 2008 (Pearson Education, 2008), Chapter 5, 2nd page.




Updated: 30 JAN 2018 (30.01.2018); 3 FEB 2018 (03.02.2018); 5 FEB 2018 (05.02.2018); 6 FEB 2018 (06.02.2018); 7 FEB 2018 (07.02.2018); 11 FEB 2018 (11.02.2018); 12 FEB 2018 (12.02.2018); 17 FEB 2018 (17.02.2018).

Last Update: 19 February 2018 (19.02.2018).


You can write to me using this: online reply form. (It opens in a new window.)

MBR and Boot Records Index

The Starman's Realm Index Page