Memory Mapping

Memory Mapping:

Memory mapping allows a file to be accessed in the same manner as a memory. This type of service is available in both UNIX and Windows. The system calls do differ. Review how PMTs (Page Map Tables) work.

Why do we use memory mapping?

Simplifies code. There is not need to track where you are in a file or do reads and writes. Think of an editor. Can use the same structure used to represent data both in the file and in memory.
Real-time in file processing can be dramatically improved. In UNIX this is by a factor of 2. CPU usage does not change.
Allows use of memory based algorithms on a file. Example, sort and serach. Mention danger in this. Example, large files.
Buffering of data automatic. Problem, in that data not posted immediately. So, there is a bit of risk if the system crashes.
Allows sharing of data among multiple processes. Give example on why this feature is useful. Give example from high frequency trading simulations.
Cannot use STL with this feature unless you do a lot of work.

Using memory mapping:

A memory map is an operating systems object. Therefore, it needs a handle. The following are the functions that you need to do memory mapping:

CreateFile - to create and/or open a file. We need this if we want to use the file for memory mapping.
CreateFileMapping - given the handle of an open file, this function will create a handle for the memory mapping. It will also allow you to specify the size of the mapping and to name the mapping. It does not create the mapping. Note: you do not need a file to create a mapping. You use this possibility for shared memory.
OpenFileMapping - creates a handle for a mapping given the name of the mapping. Used to share mappings. It is not needed if only one process is accessing the memory map.
MapViewOfFile - converts the file mapping handle into an address. Can now reference the file using memory operations and the address.
FlushViewOfFile - posts dirty pages from a file mapping to the disk. Normally, they hang out for a while. Why is memory mapped files a bad choice if you are implementing a credit card system?
UnmapViewOfFile - releases mapped memory. Why is this important? To reclaim address space so it can be reused.

Note: In that 2GB are available for user address space in Windows, the biggest a memory mapped file can be is 2GB. (You can make this a bit bigger.) In 64 bit versions of windows, we can map a lot more.

Let us walk through the help pages for memory mapping:

CreateFile

Creating and opening files. We have one system call to do this. With Win32 this system call will do a lot more than just open files, as we will see later. For example, we will be able to create a handle to communicate between processes running on the same or different computers.

The following is the prototype for this function. See the help page for this function for more detail.

HANDLE CreateFile(
LPCTSTR lpFileName,          // pointer to name of the file
DWORD dwDesiredAccess,       // access (read-write) mode
DWORD dwShareMode,           // share mode
LPSECURITY_ATTRIBUTES lpSecurityAttributes,
                              // pointer to security attributes
DWORD dwCreationDisposition,// how to create
DWORD dwFlagsAndAttributes, // file attributes
HANDLE hTemplateFile         // handle to file with attributes to copy
);

Return Values

If the function succeeds, the return value is an open handle to the specified file. If the function fails, the return value is INVALID_HANDLE_VALUE. To get extended error information, call GetLastError .

My first response at seeing all this is a hardy ARGH. This function definitely a lot of parameters. I will describe them carefully.

lpFileName – (books has this as lpszName. I took the prototype from the help pages. Apparently the documentation has changed a little bit.) This is a pointer to a '\0' character string. It is a pathname. Question:Why not use the string data type rather than the older char * notation.

DwDesiredAccess – specifies the type of access permitted for the file. This is a bit map. (Also referred to as flags.) What does that mean? The following are the legitimate values for the map.


GENERIC_READ	Specifies read access to the object. Datacan be read from the file and the file pointer can be moved. Combine withGENERIC_WRITE for read-write access.
GENERIC_WRITE	Specifies write access to the object. Data can be written to the file and the file pointer can be moved. Combine with GENERIC_READfor read-write access.

If you want both read and write access, you type:: GENERIC_READ | GENERIC_WRITE

DwShareMode - a bit-map to specify how the file may be shared. If 0 is specified for this option, the first program that opens it can only access the file. That is it can't be shared. Any attempt to open the file a second time will fail. (Even within the same program that first opened it.
: To share the object, use a combination of one or more ofthe following values:


Value	Meaning
0	File cannot be shared.
FILE_SHARE_READ	Subsequent open operations on the object will succeed only if read access is requested.
FILE_SHARE_WRITE	Subsequent open operations on the object will succeed only if write access is requested.

LpSecurityAttributes – specifies the security for this file. We will talk about this later (maybe). We will use NULL to choose the default security.

dwCreationDisposition - Specifies which action to take on files that exist, and which action to take when files do not exist. This parameter must be one of the following values or have them or'ed together:


Value	Meaning
CREATE_NEW	Creates a new file. The function fails if the specified file already exists.
CREATE_ALWAYS	Creates a new file. If the file exists, the function overwrites the file and clears the existing attributes.
OPEN_EXISTING	Opens the file. The function fails if the file does not exist.

OPEN_ALWAYS	Opens the file, if it exists. If the file does not exist, the function creates the file as if dwCreationDisposition were CREATE_NEW.
TRUNCATE_EXISTING	Opens the file. Once opened, the file is truncated so that its size is zero bytes. The calling process must open the file with at least GENERIC_WRITE access. The function fails if the file does not exist.

dwFlagsAndAttributes - Specifiesthe file attributes and flags for the file. These are ignored if the file already exists. Any combination of the following attributes is acceptable for the dwFlagsAndAttributes parameter, except all other file attributes override FILE_ATTRIBUTE_NORMAL. (Guess what value this is?) Note: I will only list a subset of the attributes.

FILE_ATTRIBUTE_NORMAL - The file has no other attributes set. This attribute is valid only if used alone.

FILE_ATTRIBUTE_READONLY - The file is read only. Applications can read the file but cannot write to it or delete it.

FILE_FLAG_DELETE_ON_CLOSE - Indicates that the operating system is to delete the file immediately after all of its handles have been closed, not just the handle for which you specified FILE_FLAG_DELETE_ON_CLOSE. Why is this great! Used for temporary files.

FILE_FLAG_WRITE_THROUGH - Instructs the system to write through any intermediate cache and go directly to disk. The system can still cache write operations, but cannot lazily flush them. Used to make data safely stored as soon as possible.

FILE_ATTRIBUTE_TEMPORARY - The file is being used for temporary storage. File systems attempt to keep all of the data in memory for quicker access rather than flushing the data back to mass storage. A temporary file should be deleted by the application as soon as it is no longer needed.

FILE_FLAG_RANDOM_ACCESS - Indicates that the file is accessed randomly. The system can use this as a hint to optimize file caching.

FILE_FLAG_SEQUENTIAL_SCAN - Indicates that the file isto be accessed sequentially from beginning to end. The system can use this as a hint to optimize file caching. If an application moves the file pointer for random access, optimum caching may not occur; however, correct operation is still guaranteed. Specifying this flag can increase performance for applications that read large files using sequential access. Performance gains can be even more noticeable for applications that read large files mostly sequentially, but occasionally skip over small ranges of bytes.

hTemplateFile - Specifies a handlewith GENERIC_READ access to a template file. The template file supplies fileattributes and extended attributes for the file being created. It takes the place of the dwFlagsAndAttributes argument. Use NULL to have this guy ignored. Why would you use? You use it to say, I want a file just like this one.

CreateFileMapping

The CreateFileMapping function creates a named or unnamed file-mapping object for the specified file.

HANDLE CreateFileMapping(

HANDLE hFile, // handle to the file to map
LPSECURITY_ATTRIBUTES lpFileMappingAttributes,
// optional security attributes
DWORD flProtect, // protection for mapping object
DWORD dwMaximumSizeHigh, // high-order 32 bits of object size
DWORD dwMaximumSizeLow, // low-order 32 bits of object size
LPCTSTR lpName // name of file-mapping object

);

hFile - if this is 0xFFFFFFFF there is no file that corresponds to the mapping. That is we are creating shared memory.
flProtect allows you to specify how the data may be accessed. This is useful if sharing mapping. You can make mapping read only.
If the size is the maximum size that the file mapping object can be. Zero implies the current file size.
lpName names the mapping object. It can be NULL. It allows other processes to access the mapping. If name exists, you will be able to access an existing mapped file.

Parameters:

hFile

Handle to the file from which to create a mapping object. The file must be opened with an access mode compatible with the protection flags specified by the flProtect parameter. It is recommended, though not required, that files you intend to map be opened for exclusive access.
If hFile is (HANDLE)0xFFFFFFFF, the calling process must also specify a mapping object size in the dwMaximumSizeHigh and dwMaximumSizeLow parameters. The function creates a file-mapping object of the specified size backed by the operating-system paging file rather than by a named file in the file system. The file-mapping object can be shared through duplication, through inheritance, or by name.

lpFileMappingAttributes

Pointer to a SECURITY_ATTRIBUTES structure that determines whether the returned handle can be inherited by child processes. If lpFileMappingAttributes is NULL, the handle cannot be inherited.
Windows NT: The lpSecurityDescriptor member of the structure specifies a security descriptor for the new file-mapping object. If lpFileMappingAttributes is NULL, the file-mapping object gets a default security descriptor.

flProtect

Specifies the protection desired for the file view, when the file is mapped. This parameter can be one of the following values. There are others that I have not mentioned.
PAGE_READONLY

Gives read-only access to the committed region of pages. An attempt to write to or execute the committed region results in an access violation. The file specified by the hFile parameter must have been created with GENERIC_READ access.

PAGE_READWRITE
Gives read-write access to the committed region of pages. The file specified by hFile must have been created with GENERIC_READ and GENERIC_WRITE access.

PAGE_WRITECOPY
Gives copy on write access to the committed region of pages. The files specified by the hFile parameter must have been created with GENERIC_READ and GENERIC_WRITE access.

In addition, an application can specify certain section attributes by combining (using the bitwise OR operator) one or more of the following section attribute values with one of the preceding page protection values: Value Description
SEC_COMMIT

Allocates physical storage in memory or in the paging file on disk for all pages of a section. This is the default setting.

SEC_IMAGE
The file specified for a section's file mapping is an executable image file. Because the mapping information and file protection are taken from the image file, no other attributes are valid with SEC_IMAGE.

SEC_NOCACHE
All pages of a section are to be set as non-cacheable. This attribute is intended for architectures requiring various locking structures to be in memory that is never fetched into the processor's. On 80x86 and MIPS machines, using the cache for these structures only slows down the performance as the hardware keeps the caches coherent. Some device drivers require noncached data so that programs can write through to the physical memory. SEC_NOCACHE requires either the SEC_RESERVE or SEC_COMMIT to also be set.

SEC_RESERVE
Reserves all pages of a section without allocating physical storage. The reserved range of pages cannot be used by any other allocation operations until it is released. Reserved pages can be committed in subsequent calls to the VirtualAlloc function. This attribute is valid only if the hFile parameter is (HANDLE)0xFFFFFFFF; that is, a file mapping object backed by the operating sytem paging file.

dwMaximumSizeHigh

Specifies the high-order 32 bits of the maximum size of the file-mapping object.

dwMaximumSizeLow

Specifies the low-order 32 bits of the maximum size of the file-mapping object. If this parameter and dwMaximumSizeHig are zero, the maximum size of the file-mapping object is equal to the current size of the file identified by hFile.

lpName

Pointer to a null-terminated string specifying the name of the mapping object. The name can contain any character except the backslash character (\).
If this parameter matches the name of an existing named mapping object, the function requests access to the mapping object with the protection specified by flProtect.

If this parameter is NULL, the mapping object is created without a name.

If lpName matches the name of an existing event, semaphore, mutex, waitable timer, or job, the function fails and the GetLastError function returns ERROR_INVALID_HANDLE. This occurs because these objects share the same name space.

Return Values

If the function succeeds, the return value is a handle to the file-mapping object. If the object existed before the function call, the function returns a handle to the existing object (with its current size, not the specified size) and GetLastError returns ERROR_ALREADY_EXISTS.

If the function fails, the return value is NULL. To get extended error information, call GetLastError.

OpenFileMapping

The OpenFileMapping function opens a named file-mapping object. The big thing about this is we don't need the file handle. We can use the name given when CreateFileMapping was used. A nice way of accessing a previously created file.

HANDLE OpenFileMapping(

DWORD dwDesiredAccess, // access mode
BOOL bInheritHandle, // inherit flag
LPCTSTR lpName // pointer to name of file-mapping object

);

dwDesiredAccess

Specifies the access to the file-mapping object.
Windows NT: This access is checked against any security descriptor on the target file-mapping object.

This parameter can be one of the following values: Value Meaning
FILE_MAP_WRITE Read-write access. The target file-mapping object must have been created with PAGE_READWRITE protection. A read-write view of the file is mapped.

FILE_MAP_READ Read-only access. The target file-mapping object must have been created with PAGE_READWRITE or PAGE_READ protection. A read-only view of the file is mapped.

FILE_MAP_ALL_ACCESS Same as FILE_MAP_WRITE.

FILE_MAP_COPY Copy-on-write access. The target file-mapping object must have been created with PAGE_WRITECOPY protection. A copy-on-write view of the file is mapped.

bInheritHandle

Specifies whether the returned handle is to be inherited by a new process during process creation. A value of TRUE indicates that the new process inherits the handle.

lpName

Pointer to a string that names the file-mapping object to be opened. If there is an open handle to a file-mapping object by this name and the security descriptor on the mapping object does not conflict with the dwDesiredAccess parameter, the open operation succeeds.

Return Values

If the function succeeds, the return value is an open handle to the specified file-mapping object.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.

MapViewOfFile

The MapViewOfFile function maps a view of a file into the address space of the calling process. This function is necessary if we are to actually access the file.

LPVOID MapViewOfFile(

HANDLE hFileMappingObject, // file-mapping object to map into
                              // address space
DWORD dwDesiredAccess,      // access mode
DWORD dwFileOffsetHigh,     // high-order 32 bits of file offset
DWORD dwFileOffsetLow,      // low-order 32 bits of file offset
SIZE_T dwNumberOfBytesToMap // number of bytes to map

);

dwDesiredAccess allow you to specify permissions. These permissions must be allowed in the CreateFileMapping.
You may also specify the offset to the beginning of a mapped file. Must be match memory allocation granularity. 0 is a great choice.
Number of bytes indicates the extent of the mapping. 0 means the entire file. Not

Parameters:

hFileMappingObject

Handle to an open handle of a file-mapping object. The CreateFileMapping and OpenFileMapping functions return this handle.

dwDesiredAccess

Specifies the type of access to the file view and, therefore, the protection of the pages mapped by the file. This parameter can be one of the following values: Value Meaning
FILE_MAP_WRITE Read-write access. The hFileMappingObject parameter must have been created with PAGE_READWRITE protection. A read-write view of the file is mapped.
FILE_MAP_READ Read-only access. The hFileMappingObject parameter must have been created with PAGE_READWRITE or PAGE_READONLY protection. A read-only view of the file is mapped.
FILE_MAP_ALL_ACCESS Same as FILE_MAP_WRITE.
FILE_MAP_COPY Copy on write access. If you create the map with PAGE_WRITECOPY and the view with FILE_MAP_COPY, you will receive a view to file. If you write to it, the pages are automatically swappable and the modifications you make will not go to the original data file.

If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is propagated to the other process. The original file does not change.

Windows NT: There is no restriction as to how the hFileMappingObject parameter must be created. Copy on write is valid for any type of view.

If you share the mapping between multiple processes using DuplicateHandle or OpenFileMapping and one process writes to a view, the modification is not propagated to the other process. The original file does not change.

dwFileOffsetHigh

Specifies the high-order 32 bits of the file offset where mapping is to begin.

dwFileOffsetLow

Specifies the low-order 32 bits of the file offset where mapping is to begin. The combination of the high and low offsets must specify an offset within the file that matches the system's memory allocation granularity, or the function fails. That is, the offset must be a multiple of the allocation granularity. Use theGetSystemInfo function, which fills in the members of a SYSTEM_INFO structure, to obtain the system's memory allocation granularity.

dwNumberOfBytesToMap

Specifies the number of bytes of the file to map. If dwNumberOfBytesToMap is zero, the entire file is mapped.

Return Values

If the function succeeds, the return value is the starting address of the mapped view.
If the function fails, the return value is NULL. To get extended error information, call GetLastError.

FlushViewOfFile

The FlushViewOfFile function writes to the disk a byte range within a mapped view of a file. That is it posts the dirty pages to the disk.

BOOL FlushViewOfFile(

LPCVOID lpBaseAddress, // start address of byte range to flush
DWORD dwNumberOfBytesToFlush // number of bytes in range

);

Parameters:

lpBaseAddress

Pointer to the base address of the byte range to be flushed to the disk representation of the mapped file.

dwNumberOfBytesToFlush

Specifies the number of bytes to flush. If dwNumberOfBytesToFlush is zero, the file is flushed from the base address to the end of the mapping.

Return Values

If the function succeeds, the return value is nonzero.

If the function fails, the return value is zero. To get extended error information, call GetLastError.

UnmapViewOfFile

The UnmapViewOfFile function unmaps a mapped view of a file from the calling process's address space. That is, it recovers the PMT and post dirty pages.

BOOL UnmapViewOfFile(

LPCVOID lpBaseAddress // address where mapped view begins

);

Parameters:

lpBaseAddress

Pointer to the base address of the mapped view of a file that is to be unmapped. This value must be identical to the value returned by a previous call to the MapViewOfFile or MapViewOfFileEx function.

Return Values

If the function succeeds, the return value is nonzero, and all dirty pages within the specified range are written "lazily" to disk.
If the function fails, the return value is zero. To get extended error information, call GetLastError.

See example: mmap.cpp

Notes:

Note to me: don't forget to create a duck.txt file.
Try making the size of the memory map bigger.
Try accessing the file outside of the map.
Try dispensing with the file.
Give the memory map a name and try accessing it from somewhere else.