CS 570 --- Operating Systems
Dr. John Carroll - carroll@edoras.sdsu.edu - GMCS537

Text: Modern Operating Systems, Third Edition, Andrew S. Tanenbaum, available
through internet booksellers; type in ISBN 0136006639 at addall.com to get prices.
Notes: A collection of the annotated programs and diagrams presented in class.
Available by mail from Cal Copy (619-582-9949)

These Notes are written so that the [more expensive] Fourth Edition of the text
could be used instead. (ISBN 013359162X, available for about $140 through
internet booksellers; the international paperback edition is significantly
cheaper.) The Second Edition of Tanenbaum may be cheaper, but is NOT recommended.

Course Content:
File Systems,
Processes and Threads,
Process Scheduling and IPC,
Memory Management and Virtual Memory,
Deadlocks,
I/O Devices,
File Systems,
Concurrency Issues and
Security

These topics will be discussed in the context of general operating systems,
and will also be illustrated using UNIX as a specific example.  The following
areas of UNIX programming will be addressed:
fork(),
exec(),
pipe(),
constructing a shell (like csh),
interprocess communication,
concurrent programming

Prerequisites:
Knowledge of the C programming language,
CS310 (Data Structures), and CS370 (Computer Architecture).
You MUST know the material in these courses, or you will be lost.

################################################################################
#  You MUST write and successfully implement a simple C program within         #
#  the first week of class or you will be FORCIBLY DROPPED from my roster.     #
################################################################################

Grading:
Assignments will comprise 30% of your grade,
the final will be worth 45%, and the midterm will account for 25% of your grade.

Letter grades:
cumulative scores of 90% (and above) are guaranteed an A- (or possibly better);
80% and above: B- (or better);
70% and above: C- (or better);
60% and above: D- (or better)

Policies:

Homework and programming assignments are intended to help you learn.
Talking over your ideas and problems with other people in the class is
very helpful.  You may discuss ideas, but you must do your own work and
write up your own solutions and programs.  In particular, you may NOT work
on an assignment (or a program) as a team.

Using another person's work is cheating.  
Copying a program from a book is plagiarism, just like copying from a paper
for a humanities class, unless you give an appropriate citation.
If you are in doubt about where the border line is for our assignments, ASK ME.
It should go without saying (but past experience suggests saying it)
that copying on exams, homework, or other forms of cheating will be dealt
with severely.  Automatic failure of the course is guaranteed in such cases,
and sanctions can include expulsion from the university.
If an assignment is copied (or developed as a team), BOTH parties will fail the
course (so, if someone asks to copy your work, point them at this paragraph :-)

Written assignments are due at the BEGINNING of class on the day
specified on the assignment.  Programming assignments will be collected
electronically.  To maintain fairness and uniformity of grading, I
cannot accept late assignments.  Similarly, there will be no make-up
exams.  In unusual circumstances (to be determined by me), you might
be allowed to take an oral makeup at the end of the semester.  If you
know in advance that you will miss an exam, see me about it in advance.  Note
the date of our final exam now; don't make plans that conflict with the final.
Note in particular that university policy prohibits taking the final early.
(See, for example,
https://registrar.sdsu.edu/calendars/final_exam_schedule/fall_2020_final_exam_schedule)

You will be assigned edoras (edoras.sdsu.edu) accounts for use
in this course; assignment and exam announcements and spreadsheet entries
of your grade scores will be sent to this account (so you may wish to add
a .forward file if you don't plan to check it for mail regularly).

Note: if you are new to the system, edoras has many help files.  A good place
to start is:
http://edoras.sdsu.edu/doc/unixtut/

Lots of free software is available to you; peruse
http://edoras.sdsu.edu/software/
You'll need ssh to communicate with edoras; if you are running WinBlows,
you can get ssh functionality (and more) from:
http://mobaxterm.mobatek.net/download.html
[If you're running a real operating system at home (MacOS or Linux, which
are both versions of UNIX), don't bother -- you already have a version of ssh
with the latest security features.]

On-line Source: 
http://edoras.sdsu.edu/~carroll/cs570home.html

The above is a summary of the syllabus material; below are my specific notes
regarding Chapter 1 of Tanenbaum.

Chapter 1 -- Operating Systems Introduction

Edition 3 is a 2008 textbook, so it is essentially up to date.  Even with the
previous 2001 edition, the overarching principles discussed in this book have
been well-established for some time, so it is almost as relevant as the
shiny-new fourth edition.  The third edition covers current Linux versions
as well as Windows Vista; if you need a reference for Windows 8, then you may
want the fourth edition.  We will be using UNIX (Linux) in our examples and
programming, so you do NOT need to know anything about WinBlows for this course.
The fourth edition has even REMOVED some material that we ARE going to cover;
so the third edition is definitely the best choice for this course.

From your Assembly Language class and your Computer Architecture class,
you should already have a good idea of what the main components of the
hardware are, and how they fit together.  This course will build on those
concepts.

The main task of an operating system is primarily to manage competing entities,
which broadly fall into two categories: users and devices.  The operating
system must arrange for multiple processes (and multiple users) to peacefully
coexist, without interfering with each other.  It must also handle various
devices which might be simultaneously clamoring for attention.  Part of its
function is to protect the user from the complexity of those devices by
providing a simplified interface (as well as to protect the devices from
the typical bone-head user :-).

To protect users from interfering with each other, we need some safeguards
built into the hardware: hence the concept of permissions.  To prevent one
user from overwriting another user's files, we cannot allow a user to
directly give directions to a disk drive.  Instead, the user must ask the
operating system (OS) to carry out a write command on his/her behalf; one
of the responsibilities of the OS is to determine whether the write should
be allowed, and if so, to instruct the disk drive to carry on with the
operation.  The operating system also hides all the ugly details, so that
the user can just say 'write this block to disk', rather than having to
explicitly deal with the tons of directions that the disk needs to carry out
the operation.  A device *driver*, typically part of the OS, handles
these ugly details.  Tanenbaum gives a more in-depth discussion of such
a driver (for a floppy disk) on Page 4.

Page 6 discusses the duties of a printer driver, which forms an excellent
generalized example of the management issues.  If users were free to simply
send text to the printer, different output jobs could be intermixed with
one another, producing gibberish that is of no value to anyone.  At the
very least, the OS must ensure that one job has finished printing before
the next job starts.  More typically, the OS will order the printer requests,
perhaps on a FCFS (First-Come-First-Served = FIFO = First-In-First-Out) basis,
or perhaps use a shortest-job-first strategy, or perhaps rank print jobs
on the basis of the relative importance of the user who submits the job
(which in turn might depend on how much the user is willing to pay :-).
If there is a charge for printing, the OS would take care of keeping track
of how many pages each user prints, etc.

Other resources are managed in a similar way.  For example, the OS tracks
how many disk blocks belong to each user, and is responsible for enforcing
quotas (denying further requests if a user is currently over his individual
block limit).

At this point, the Fourth Edition and the Third Edition diverge.  These notes
will give the relevant section and page numbers for BOTH editions, so that you
can use either in this course.  Being effectively obsolete, the third [and
second] edition is substantially cheaper to buy than the newer edition.  Keep
in mind that you can sell back the fourth edition (until the fifth edition
comes out :-), but no bookstore will buy back the older editions; you'd have to
sell it on the internet.  I recommend keeping Tanenbaum as a reference, though.

The notation 'Page 6/7' below means that the topic is on page 6 of the third
edition, but on page 7 of the fourth edition.  Similarly, 'Section 4.5.3/4.5.2'
means 'Section 4.5.3 in the third edition, and Section 4.5.2 in the fourth
edition'.

Section 1.2 (Page 6/7) give an overview of the evolution of operating systems,
beginning with purely gear-driven early attempts (Babbage's design was on
the order of hand-crank mechanical adding machines or cash registers), to
the first successful attempts that used electromechanical relays (a relay
contains an electromagnet, which, when powered, magnetically flips a switch
in another part of the circuit).  These were very slow (by electronic
standards), because flipping a 1 to a 0 involved physically moving the switch
arm.  (And these famously gave rise to the term 'hardware bug', when a problem
with one early computer was traced to an actual insect that had become fried
and stuck in a switch arm, preventing the contacts from touching.)

When relays were replaced by vacuum tubes (such as you would find in the
radios or televisions of the time), the process was sped up enormously,
but was still impossibly slow by today's standards.  Programming could only
be done by using true machine language, and I/O was almost non-existent.
Answers were read out via little lights, and input was accomplished by
essentially rewiring the machine for your particular program.  Since the
machines were too valuable to lay idle while some slow human did the wiring,
this (relatively cheap) part of the computer (a "plugboard") was detachable and
duplicable.  A programmer would wire up his program off-line on his own board,
and then plug it in to the computer when it was time to run. You can see one here:
http://en.wikipedia.org/wiki/Plugboard
(This page also has a nice picture of a relay mechanism.)

Card readers made input a bit more convenient.  As things progressed, cards
were punched on a machine that was basically a cross between a typewriter
and a hole punch.  Programmers now carried around a box of cards rather
than a plugboard (and you were very careful to avoid spilling the box and
scrambling the order of your cards).  A paper tape puncher was similar in
concept, where six columns of holes/non-holes encoded a single character on
each row of a paper tape.

Transistors were a major improvement over vacuum tubes, and not just because
of speed: reliability was vastly improved.  A vacuum tube that lasts for 3 years
on average sounds pretty reliable; you would have to replace it only once every
three years.  But if you have 20,000 such tubes, something is burning out on
the average of about once an hour.  Transistors are MUCH more reliable.
(Transistors also are far more compact, require much less power to function,
and therefore generate much less heat.)

Prior to the transistor, even the idea of an assembly language was absurd,
and something as complicated as a compiler was unthinkable.  It was far more
cost-effective to have humans create the machine-language instructions
for a program than to waste valuable computer time doing something
a lowly human could do at a much lower price.  Each different
machine architecture needed its own special assembly language, of course.
FORTRAN was one of the first machine-independent languages:  a FORTRAN
compiler produced assembly language for a particular machine, and then the
assembler reduced this to the particular machine code for the target hardware.

Things had progressed to the point where peripherals could come into play.
It no longer made sense to idle a hugely expensive mainframe while it took
care of reading in punched cards at a painfully slow (mechanical) rate, nor
to use the mainframe to directly control the printer.  Instead, a separate
computer/cardreader was used to read in cards and prepare an input stream
for the mainframe.  These peripherals were not physically connected as we
would expect today; there were no wires running between the mainframe and
the peripherals.  The computer/cardreader(s) collected data and spooled it onto
a tape.  The analog of today's communication cable was at that time a human
operator who dismounted this 'input' tape and mounted it on the mainframe.
Similarly, the mainframe produced an output tape, which was then schlepped
over to a tapereader/computer/printer for printing -- which was not the
compact device you see today; it was the size (and price) of a small tank.

Computers at this time tended to fall into two categories: number-crunchers
and data processors.  The number-crunchers featured a fast CPU and didn't
bother too much about I/O.  The character-stream processors could get by
with a wimpy central processor since this CPU was mostly idle while waiting
for input or output to take place.

IBM's 360 line of computers aimed at being good at both, which led to a
need to keep the expensive CPU busy during I/O-bound jobs, so this operating
system introduced multiprogramming: if several jobs were in memory at once,
the CPU could be working on one while another was waiting for I/O to complete.
One result was that the hardware now had to have additional hardware features
to be able to protect one job from interfering with the memory space allocated
to another.

Another speedup was achieved by SPOOLing (Simultaneous Peripheral Operation
On Line), which had earlier been done by separate machines (off-line),
eliminating the use of magnetic tapes for this purpose.  Instead, a card
reader directly read input onto a disk (or drum), and the 360 would then
read a new program from the disk into main memory whenever an old program
completed and relinquished its memory space.

This was still a batch system: while programs were pre-loaded into memory
so that they could be ready to run as soon as another program completed,
a single job ran from start to finish uninterrupted before the next job
started.  This meant that small jobs could be held up for hours if they
happened to be behind a number-cruncher.

The need for better response times was the impetus for timesharing, which,
as a pleasant side effect, also often increased the utilization of all
that expensive equipment.  The idea, of course, is to run several jobs
pseudo-simultaneously (within a second, allow several jobs to have the CPU
for a little bit of time), so that jobs that require only a little CPU time
won't be backed up for a long time waiting for a number-cruncher.  More
specialized hardware was needed to support this, plus some very fancy OS
programming to support the context switches between one job and another.

MULTICS was a far-sighted project that introduced many innovations,
including timesharing, intending to be all things to all users.  It was
written in PL/1 (Programming Language One) which likewise was intended
to have every possible feature imaginable dumped into one compiler (like
ADA on steroids).  PL/1 never did work completely, which meant that
MULTICS was delayed by both the language and the challenge of implementing
all the new ground-breaking concepts.

Ken Thompson and Dennis Ritchie at Bell Labs began writing an efficient,
stripped down, one-user version of MULTICS, which formed the basis of UNIX
(UNIX is a pun on MULTICS: it was intended to do one thing, and do it really
well).

The result was a very efficient and small OS of exquisite design (so good,
in fact, that research into alternate operating systems was, and still is,
retarded -- most researchers concentrated on improving UNIX rather than
setting out in new directions).  Over the years, the spartan kernel was
expanded to support more and more capabilities, all without losing the
original efficiency.  (Chapter 10 of Tanenbaum contains the details,
which we will cover as needed for our programming assignments.)  The
distribution limitations of the UNIX variants motivated Linus Torvalds 
to begin writing an unrestricted version of UNIX called Linux (Lee-nucks).
Linux/UNIX in its current forms can run on anything from a supercomputer
to a palmtop.

Originally, individual transistors were wired together on a circuit board,
requiring the CPU to be spread out over a large area, which meant that signals
had a long way to go between components.  Integrated circuits (where many
transistors were on the same chip) allowed CPUs to shrink in size, giving
a nice speedup as a result.  LSI (Large-Scale Integration) circuits, grouping
thousands of transistors on a single chip, improved things dramatically.
Minicomputers and then personal computers became a reality.  The Intel 8080
and the DEC LSI-11 (a small PDP-11) were among the first.  In 1980, an
LSI-11 could be bought for about $10,000, consisting of a processor, drive,
keyboard, and screen.  ('Drive' meant an 8-inch floppy drive; an actual
hard drive was outlandishly expensive at the time. 'Screen' meant an 80x24
monochrome character display.)

IBM only reluctantly entered the personal computer field; they were pretty sure
that if a business could buy a $10,000 machine that would meet their needs,
they might choose that instead of a multi-million-dollar machine from IBM.
However, since minicomputers and personal computers from other companies
were starting to really impact their sales, they entered the field, but late.
This meant that they did not have time to build a new machine from scratch;
happily (for the world in general, but not so much for IBM) that meant they
had to cobble something together from existing parts, rather than take the
time to design a custom system.  Everything was off-the-shelf parts, except
for the BIOS on the motherboard.  (The BIOS is briefly discussed on Page 33/34.)

Once the firmware BIOS was reverse-engineered, just about any company could
build an IBM clone.  That is why today, rather than having dozens of hardware
platforms with the sort of incompatibilities that exist between the PC and
the Mac, the world has a (fairly) uniform set of hardware to build upon.
The downside is that most of these PCs run one of the abominations peddled
by Microsoft.

We will next look at Section 1.3 (Computer Hardware Review, Page 19/20), but,
since it *is* a review and the text is fairly comprehensive, we will primarily
follow the text's exposition, rather than repeating pretty much the same points
here in these notes.

Make sure you have a good grasp of the concepts of program counter, stack
pointer, and PSW (Program Status Word); review these as necessary, and check
the texts from the prerequisite courses if you need to brush up on them.

The figure on Page 21/22 illustrates pipelining.  This concept aims to speed
up the fetch/decode/execute cycle by attempting to do these phases concurrently.
While one instruction is being fetched, the one in front of it is being decoded,
while the one in front of that one is being executed.  Having a fetch unit,
a decode unit, and an execute unit working simultaneously has the potential to
speed things up by (not quite) a factor of three.  However, as the book says:
"Pipelines cause compiler writers and operating system writers great headaches
because they expose the complexities of the underlying machine to them."
You may have only a vague idea of what this might mean at present, but at the
appropriate point in the course we will look at some very specific examples
which will make it quite clear.

Part (b) of the figure on Page 21/22 shows a way to get even more parallelism
(using superscalar architecture), in which instructions might even get executed
out of their intended order (if the hardware determines that it really doesn't
affect the ultimate results of the calculation), which leads to infinitely
greater headaches for the operating system writers.

All but the simplest computers have two modes, user mode and kernel mode.
All instructions are available in kernel mode, but 'dangerous' operations
(such as writing a block of data to the disk drive) are blocked in user mode.
User programs run in user mode, but this does not mean that users cannot write
to disk.  As mentioned before, the user must ask the operating system (OS)
to carry out a write command on his/her behalf by means of a system call
(such as write()); one of the responsibilities of the OS is to determine
whether the write should be allowed, and if so, to instruct the disk drive
to carry out the operation.  The mechanism for doing this is embedded in
the system call: the system call causes a change from user mode to kernel mode,
and then (after making the appropriate checks) the instructions to the disk
drive will not be rejected by the hardware, because we are now running in the
(enhanced) kernel mode in which disk writes are allowed.

Section 1.3.2 Memory (Page 23/24)

The figure on Page 23/24 shows the typical 'memory hierarchy',
which is driven by the obvious economic reasons.
Anything that is successful in the market must have some redeeming aspect:
if you manufacture something that is expensive, slow, and holds very little,
no one will buy it.  Disk drives are slow, but they have the advantage of
being (relatively) cheap and non-volatile.  Cache memory is very fast, but
also very expensive, so it is used for only the most critical aspects of
computing, where it will make the most difference.  In between these extremes
is RAM, far faster than hard disk storage, but not as expensive as cache memory.

We will discuss cache concepts later in the course, and move ahead to:

Section 1.5 Operating System Concepts (Page 37/38)

The general concepts in this section will be reviewed briefly in class.
Make sure you are familiar with the bold-face terms (process, address
space, etc.)

Section 1.5.7 Recycling of Concepts / Ontology Recapitulates Phylogeny
(Page 46/47)

Read Pages 46-47/47-48 carefully, to ensure you understand the point that is
being made: Technology changes can make some concepts obsolete, but further
changes might make them relevant again.  A recent shift (not discussed
adequately in either edition) is the emergence of Solid-State drives,
which are just now becoming cheap enough to replace spinning-disk drives.

Section 1.6 System Calls (Page 49/50)

The UNIX system calls in this section will be reviewed briefly in class.
As we need the nitty-gritty details in the programming assignments,
we'll go over the relevant concepts when I discuss the assignment.

Starting on page 3 of this booklet, you will find programming examples that
make use of the system calls we will be using.  Electronic copies of these .c
files can be found on edoras, under the ~cs570 directory.

Pages 265-268/273-276 discuss a C program that illustrates reading and writing
on a UNIX system.  I recommend reading that over carefully before starting your
first programming assignment.  (Details of the first programming assignment are
near the end of this booklet.) NOTE: Figure 4-5 on Page 266/274 uses the read()
system call, but getchar() is a far better way to read data in Program 1.

We will cover the concrete material in Chapter 10 of Tanenbaum in bits and
pieces, at the time that they become relevant to our more theoretical
discussions.  (This chapter gives specific examples of the UNIX implementation
of Operating System concepts.)

Section 10.2.3 The Shell (Page 731/683)

This section describes features of a typical command-line shell.  Understand
this material well, because you will soon be writing a program that carries
out much of the functionality that is discussed here.

Section 10.2.4 Linux Utility Programs (Page 734/686)

Many of the utilities described here will be useful to you during the course
of the semester.  Figure 10-2 (Page 736/688) contains a handy table.

Section 10.3 Processes in Linux (Page 739/733)

You will need to have a basic grasp of the material in this section in order
to handle some of our subsequent programming projects.  For now, you should
understand the concepts of fork() and exec().  The discussion of the program
in Figure 10-7 (Page 744/738) shows how these system calls interact.
Figure 10-8 (Page 749/743) illustrates the ideas.

We next skip to Chapter 2 (though Section 1.8, Page 72/73,
"The World According To C", is worth reviewing if you have not
had to handle large C projects before).  We will use makefiles
in each of our projects, so it is important to understand how they work.
I have included several sample makefile questions (with answers!) in
Problem Sheet 1 (near the end of these notes).

At some point in your career, you will probably want to have a C language
reference: C by Discovery by Foster is a good choice for learning C,
because it explains things thoroughly and contains tons of sample programs to
illustrate the concepts.  The current edition is expensive, but the earlier
editions used to be dirt cheap (we seem to have sucked up all the old copies,
though).  A good place to do comparison pricing is addall.com; the ISBN number
for the second edition is 1881991296 (also 1881991298).

On edoras, ~masc0000/foster.tar.z is a tarball containing all the sample
programs [from the second edition, but most of these are the same as in the
other editions, too].  The individual Foster programs can be found under
~masc0000/CbyDiscovery/ch*. A good template to start your program0 and program1
is ~masc0000/CbyDiscovery/ch2/inout2.c , for example.  Also, ~masc0000/Foster4th
is a text file, much like these notes, explaining what I think are the key
concepts in the Foster text.  (The page numbers are keyed to the Fourth edition,
but all the editions follow more or less the same exposition, and use the same
sample programs.)

Our CS570 due dates will be publicly available in ~cs570/calendar, and
will also be announced via email.  Rather than checking your edoras account
incessantly, you should set up a ~/.forward file to send a copy of the edoras
email to the address of your choice.  The contents of the .forward file
will then be something like:

\cssc00nn,
someone@somewhere.com

The backslash (back, not forward!) in front of your edoras username is
very important!