In the past, this material was the starting point for the advanced topics course. Starting in Fall 2006, because the course sessions are longer, we will be able to cover this material in the UNIX environment course.
These three seeming unrelated topics are presented together in they are based on a common structure. The system calls are referred to as IPC (interprocess communications) calls. The IPC calls are not dependent on the file system. This differs from so many other system elements which are tied to files descriptors. The most important of these 3 services is shared memory.
Keys
are used to name and access the IPC structures. A key is a positive
integer. A
key is used to name an IPC structure in the operating system.
From the key, you can get an ID which most system calls need to reference the
IPC structures. Note: IDs are different from file descriptors in that they
are global.
There are two commands associated with IPC structures. The first, ipcs, is used to get general information (status) about IPC structures. The second, ipcrm, is used to remove IPC structures. It is important to remove them or they stay around. You must make sure that you do this, in that these resources may be limited.
The
following is a description of each command with a basic set of options.
ipcs
[-m] [-q] [-s] [-a] (-m = shared
memory, -q = message queues, -s = semaphores, -a = list all information)
ipcrm
[-m shmid] [-q msgid] [-s semid] [-M shmkey] [-Q msgkey] [-S semkey]
Each IPC structure is referred in systems calls by an identifier. This is a nonnegative integer. The identifier is assigned to the structure when it is created.
When
an IPC structure is created or accessed, a key must be used. This is of type key_t (usually a long
int). The shmget, semget, and msgget
functions (recall that there are 3 IPC structures) are used to create and prepare to access an IPC structure. These return the IPC identifier, which is
used for all other calls. Why have but
a key and an ID?
IPC
structures have owners and groups.
These permissions are similar to file permissions. For example, shared memory may be set up so
that it has read/write access for owner processes and it has read only by any process that does not belong to the owner of the
shared memory. (This is assuming that
the operating system/hardware supports access control.)
Read
IPC_PIVATE (value of 0) as a key.
Using
IPC structures requires some management.
BIG problems can occur if two independent processes chose the same key. So, either keys must be reserved in some way or every program must be
required to use the ftok function to generate a key. The
ftok function generates a unique key from a pathname and a one-byte project
id. For large systems with many
applications, a good tactic is to keep a write-protected directory for file
names for ftok.
The format of the ftok system call is as follows:
#include <sys/types.h>
#include <sys/ipc.h>
key_t ftok( char *path, char id );
Returns a unique key if successful and (key_t) -1 if it fails. It fails if the path does not exist.
We
will start with shared memory. This is the most
useful of the 3 structures. All the
other IPC structures have similar system calls. Shared memory is memory that is accessible to a number of
processes. By several orders of
magnitude, it is the quickest way of sharing information among a set of
processes.
Examples
of use:
1. To implement very big pipes whose status and content may be viewed by other processes. Recall that pipes were only 16K
2.
To
share current day's stock data among a bunch of processes.
3.
To
maintain dynamically updated statistics on a set of running processes.
In class, we will discuss how shared memory is implemented in an operating system. This hopefully will be a review of what you learned in operating systems.
Note: shared memory is persistent. It does not go away when no program is referencing it. This can be a good thing, but it can tie up system resources. Please make sure that you clean up after yourself. This is why the ipcrm command is important.
shmget system call. Gets or creates a shared memory segment. The call has the following format:
#include
<sys/types.h>
#include
<sys/ipc.h>
#include
<sys/shm.h>
int
shmget ( key_t key, size_t size, int shmflg)
Returns shared memory ID if successful and -1 if
error.
key has been discussed before.
size is used when creating the shared memory segment
to specify the size. If the shared
memory segment exists, it may be 0 or used to specify the minimum size of the
shared memory. The size of the shared memory you request is less than or
equal to the amount you actually get. That is, due to the fact that shared memory is allocated in
pages, so, the actual amount of shared memory allocated may be bigger than the
size requested.
shmflg are flags specifying options for this command. Right-most 9 bits are permissions. SHM_R and SHM_W for owner and these shifted 3 bits to the right for group and these shifted 6 bits to the right for other. Also have IPC_CREAT and IPC_EXCL bits. These are similar to O_CREAT and O_EXCL for the file system. There are additional flags that are not as general. For example, on Solaris, there is the ability to make the shared memory dynamically resizable. This feature was not available on older UNIX systems.
Note: shmget does not make the memory available to your program. You need another call to do this.
shmat system call. Attaches the shared memory segment to our process. This is separate from shmget because there is a limit on the number of shared memory segments to which we can attach. This is a system-imposed limit. (Let's try and figure this out when I do an example.) So, if you are dealing with a large number of shared memory segments, a good strategy would be to get the IDs for all that we need and then attach only to those shared memory segments that we are actively using. We can detach from a shared memory segment.
The system call shmat is defined as follows:
#include
<sys/types.h>
#include
<sys/ipc.h>
#include
<sys/shm.h>
void
*shmat ( int shmid, const void *shmaddr, int shmflg );
Returns a pointer to the shared memory segment or -1
if error.
shmid is the shared memory ID
shmaddr should be NULL. This argument can allow you to specify the address to associate
with the shared memory. For modern
computers, it is hard to find a reason for using this argument. Do no specify an address unless you have a
very good reason.
shmflg are flags for this call. The only one that we will care about is
SHM_RDONLY. This makes the shared
memory segment read only. This is like opening a file to be read only.
shmdt
detaches a shared memory segment from a process. There is a limit to how many shared memory segments you may
attach to. shmdt is used so that
another segment may be attached. The
format of this call is as follows:
#include
<sys/types.h>
#include
<sys/ipc.h>
#include
<sys/shm.h>
int
shmdt (const void *shmaddr );
shmaddr is the address of the shared memory
segment.
Returns 0 if successful and -1 if fails.
shmctl
performs a bunch of utility functions on shared memory. Its description follows:
#include
<sys/types.h>
#include
<sys/ipc.h>
#include
<sys/shm.h>
int
shmctl ( int shmid, int cmd, struct shmid_ds *buf)
Returns 0 if successful and -1 if fails.
cmd is the command that we wish to have executed.
IPC_RMID removes the shared memory segment when no
one is using it.
IPC_STAT gets data associated with shared memory
segment into buf
IPC_SET allows 3 fields in buf to be changed.
buf is used by certain commands. The most often used command is IPC_RMID.
See
example for much clarification. shm1.cpp,
shm2.cp
Notes on example:
Uncomment the code an show that the shared memory can be deleted.
Use ipcrm to remove the shared memory while the program is still attached to it. Do this in the debugger so you can see that the shared memory did not go away. The shared memory does not show up but it is there. Prove it by removing it and have the program call shmdt and look at it again.
Show that ftok will give a different key if the file is removed and added.
Show that readonly permission works.
If
there is a lot of information to be recorded in shared memory, a struct is used
to hold the data. For example:
#define
STUDENTS_SIZE 2056
typedef
struct {
long int date;
struct students
st[STUDENTS_SIZE];
int duck; /*
There always has to be a duck. */
} SharedMemory;
SharedMemory *SHM;
int
sid = shmget ( 23412, sizeof( SharedMemory), 0 ); // Assume already
created.
SHM = shmat( sid, NULL, 0 );
Then can reference duck by “SHM->duck”.
NOTE: It is very difficult to put STL objects in shared memory. Why??
How do we protect against multiple versions of the shared memory structure?
We will discuss in class the problem of needed an agreement between programs that use shared memory on the structure of shared memory.
Always remember to clean up any shared memory that you have allocated.
Write a program to implement a
stream
oriented queue using shared memory. Make the
queue 100K. You
should have two functions:
int sendQueue( char *buff, int buffsz ); Returns
-1 for error and 0 if successful.
int readQueue(char *buff, int buffsz);
Returns the number of bytes read and -1 if error.
Why
not a bool return value for the first function? What
kind of errors must we handle? How do
we set up the queue structure?
Let's show a risk in using shared memory by creating a race condition. How would we do this? We will do it in class.
Message Queues
Message
queues provide another means of inter-process communications that is
independent of the file system. Our
textbook (Stevens) is not a fan of message queues.
It claims that they are too complicated. They are a little tricky, but I like them sometimes for the following reasons:
1. They are record oriented.
2. They do not require family relationship among communicating
processes.
3. Allow multiple processes to write to and read from the queues
in a manner that is thread safe.
4. Allows prioritizing of messages.
5. Allows directed messages.
System
calls to support messages queues: msgget, msgsend, msgrcv, and msgctl.
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>
int msgget ( key_t key, int msgflg )
Returns the message queue ID associated with the key.
msgflg may be IPC_CREAT,
IPC_PRIVATE, IPC_EXCL, MSG_R, MSG_W.
The permission flags work in a manner similar to those used for shmget.
#include
<sys/types.h>
#include
<sys/ipc.h>
#include
<sys/msg.h>
int
msgsnd ( int msgqid, const void *msgp, size_t
msgsz, int msgflg)
Returns: 0 if successful and -1 if fails.
msgqid - ID for the message queue.
msgp - pointer to the message. The message is in the format:
struct {
long
mtype; /* Positive int describing message. */
char
mtext[1]; /* The message data. */
}
msgsz
msgsz - the
size of the data area to be sent.
Msgflg - flags.
IPC_NOWAIT means nonblocking.
(errno = EAGAIN if would block.)
int
msgrcv ( int msqid, void *msgp, size_t msgsz, long msgtype, int
msgflg)
Returns: size of data portion of message if
successful and -1 if fails.
msgqid - ID for the message queue.
msgp - pointer to the message. The message is in the format:
struct {
long
mtype; /* Positive int describing message. */
char
mtext[1]; /* The message data. */
}
msgsz -
maximum size data area that we are ready to receive.
msgtype - specifies which message we want.
msgtype == 0 Want the any message.
msgtype > 0
Want the first message with mtype == msgtype.
Msgtype < 0
Want the first message with mtype <= |msgtype|
msgflg - flags.
IPC_NOWAIT means nonblocking.
errno == ENOMSG if queue empty.
Talk
of how we can use mtype to client server applications. Talk of priority messages.
msgctl
system call. Used to perform various
operations on a message queue.
#include <sys/types.h >
#include <sys/ipc.h>
#include <sys/msg.h>
int msgctl( int msgqid, int cmd, struct msqid_ds
*buf );
Returns 0 if successful and -1 if fails.
cmd - IPC_STAT, IPC_SET (can set various elements
associated with the message queue. E.g. the permissions), and IPC_RMID
Notes
on Message Queues:
They continue to live until
they are explicitly deleted.
Message queues have a size limit and a surprisingly small one. 16K for Linux and 4k for Solaris. I guess the pathetic size shows that SUN does not consider these important.
Individual messages have size limits. 8K for Linux and 2K for Solaris.
Using positive type values in msgrcv is dangerous. We will discuss why in class
Usually faster than pipes. The books statistics don't show this however. They send 100,000 blocks of 2K and get 4.22 seconds for pipes and 3.71 seconds for message queues. What is going on here?
Example
concept. We will discuss in class how two message queue can be used to
support the interaction between a set of clients and a server.
See examples. send.cpp, rec.cpp
Write two programs.
RCsend and RCreceive. Both will
have a file name as a command-line argument.
RCSend will use messages queues to send the contents of its file to
RCreceive. RCreceive will record in
its file the data sent to it. Rcsend
and RCreceive will terminate when they have completed their tasks.
Discuss race conditions. Show how they can happen. We will write code to cause some race conditions in class. Please try and think of some way to cause this problem. (I put this comment here in case I forgot to discuss race conditions earlier.)
Semaphores
are used to control access to shared resources. Idea for semaphores due to Dijsktra. He introduced the concept semaphores. These are global variables used for access control. He further introduced the P and V
operators. The P and V operators are
defined (not implemented) as follows:
void P(sem)
{
while
( sem < 1 ) ;
sem--;
}
void V(sem)
{
sem++;
}
Do example on how to protect a critical area of code.
Discuss
why the above implementation will not work. Semaphores usually live in the
operating system or have special machine language instructions to guarantee that
they really work.
IPC semaphore are a more general and more confusing implementation of semaphores. Care has to be taken with semaphore in that there is some risk associated with them. There are alternatives to semaphores. These are referred to as mutexes. They are simpler and much more efficient when working with threads. They will be discussed in the advanced course.
The following are the system calls for semaphores.
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
int semget( key_t key, int nsem, int semflg );
Returns: the semaphore ID for the set of
semaphores. -1 if error.
nsem - the number of semaphores requested. Most
other operating systems only allow you to request one semaphore at a time.
Semflg - specifies options: IPC_CREAT, IPC_PRIVATE, IPC_EXCL, SEM_R, SEM_A
Note: semaphores in a set are referenced by the numbers
0 to nsems - 1.
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
int semop( int semid, struct sembuf sops[], int nsops )
returns 0 if successful and -1 if fails.
semid - the ID of the semaphore set.
nsops - the number of semaphore operations (remember
we have multiple semaphores.)
sops - the operations to be performed on each of the
semaphores. These are described by the
structure:
struct sembuf {
short sem_num; /* semaphore number */
short sem_op; /*
semaphore operation */
short sem_flg; /*
operation flags, IPC_NOWAIT, SEM_UNDO. */
};
sem_op - an integer that describes the semaphore
operation:
> 0 means increment semaphore by sem_op.
== 0 wait until the semaphores value becomes zero.
< 0 if |sem_op| <= semaphore value, add to semaphore. Otherwise, block until it is..
#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/sem.h>
int semctl( int semid, int semnum, int cmd, union
semun arg)
Returns: -1 if fails, 0 or command specific value.
union semun {
int val;
struct
semid_ds *buf;
ushort_t
*array;
} arg;
cmd - command: IPC_STAT, IPC_SET, IPC_RMID, GETVAL (returns value of semaphore), SETVAL (sets value of semaphore to arg.val ), SETALL set all semaphores to arg.array, etc.
Examples
semtest.c semtest2.c
There is a limit on how many semaphore you can have on the system. It is relatively small. In Solaris you can only have 10 sets. So, don't forget to clean up after yourself.
The SEM_UNDO is pretty cool. It allow a recovery from a program that terminates. Otherwise an aborted program can leave all others hanging on a semaphore.
Create a shared memory segment for a long int variable initially 0. Then us fork to schedule ten processes. Each of the process will add 10 onto the
variable, a sufficient number of times to see race conditions. Then display
the final result. Rewrite a second
version of the program using semaphores to protect the variable.