System Call Support

Mark Smotherman
Last updated: May 2025

Early history - including memory protection, multiple modes of execution, and privileged instructions.

Many of the seminal ideas, however, were widely spread by word of mouth or internal memorandum rather than by journal publication, and historical accuracy is sometimes difficult to obtain. In addition, some ideas related to protection were originally conceived in other contexts.
...
The concepts of base-and-bound register and hardware-interpreted descriptors appeared, apparently independently, between 1957 and 1959 on three projects with diverse goals. At M.I.T., J. McCarthy suggested the base-and-bound idea as part of the memory protection system necessary to make time-sharing feasible. IBM independently developed the base-and-bound register as a mechanism to permit reliable multiprogramming of the Stretch (7030) computer system. At Burroughs, R. Barton suggested that hardware-interpreted descriptors would provide direct support for the naming scope rules of higher level languages in the B5000 computer system.

-- from Saltzer and Schroeder, "The Protection of Information in Computer Systems," CACM, July 1974.

"[John] McCarthy's essential insight was that both hardware interrupts and memory protection were needed to provide safe and efficient timesharing service. Memory protection also makes it easier to debug programs and to run programs with unkown integrity."

-- Les Earnest

Hardware Developments

• IBM Stretch (late 1950s)

  Stretch used bounds registers to protect memory, and it used interrupts being enabled or disabled as essentially the mode bit. That is, the bounds registers can be changed only when interrupts are disabled. However, the instruction to disable interrupts (branch disabled) can be executed by a user program. The Stretch designers relied on the compiler to prevent this instruction from being generated for user programs. (This same philosophy influenced the IBM 801 RISC design.)

• Burroughs B5000 (1961)
  • The 1961 B5000 overview, "The Descriptor," lists four modes of execution, but only one for the OS:
    • Arithmetic or normal mode
    • Subroutine mode
    • Data Manipulation or editing mode
    • Control mode, "which is in effect whenever the Master Control Program is being used"

  • The B5000 Operational Characteristics Manual lists seven instructions that are only available in control mode, including initiate I/O and initiate processor (see pages 5-14 to 5-15)

  • It appears that the COM instruction is the general method to make a system call to MCP by passing information and triggering an interrupt. (Other instructions, such as program release, can also trigger interrupts under certain conditions.)

    from The Descriptor:
    Processor-Dependent Interrupts
    PROGRAM COMMUNICATION. This interrupt condition may indicate one of a variety of conditions: end of a program segment, end of a program, or the need for additional memory to accommodate data. The information required by the MCP to handle the situation is found in the top of the Stack.

    from the B5000 manual:
    Communication Operator-Communicate information to MCP.
    COMMUNICATION OPERATOR. If operating in Processor A, store the contents of the word at the top of the stack in a specific location. Delete the word from the stack. Set the communication bit in the interrupt register.

    from a later B5500 manual:
    Communication operator. This interrupt is set by a communication operator and is used to prove a means of entry to the MCP.
    COM Communication Operator (COML) 1011 6-201. The word at the top of the stack is stored in R plus nine (11 octal). The word is deleted from the stack. The communication interrupt bit is set. The operator is a No-Op in the control state.

  • Robert Bock describes how MCP returns control to a user program:

    Whichever program is to be initiated, the Master Control Program will select from the Mix Table the entry corresponding to that job (Figure 3). From this Mix Table entry, the PRT [Program Reference Table] base address is procured. Using the base address of the PRT, the Initiate Control Word will be placed in the top of the Control State Stack and the Initiate Processor syllable will be executed.

• Atlas (1962)
  • interrupts and "extracodes" invoked supervisory subroutines held in read-only memory
  • ATLAS had three execution modes, each with a separate program counter:
    • M-state (main or user control)
    • E-state (extracode control)
    • I-state (interrupt control)
  • Iain Stinson writes:

    The I-state was the most privileged state and this was used for only very small parts of the supervisor and in this mode of operation the one-level store was never used. The E-state was the next most privileged mode on the machine and it was in this mode that much of the supervisor and all the extracodes were obeyed. User programs always ran in M-state, except, of course, for the execution of extracodes they initiated.

  • these three states were encoded using two bits: M/E and I/ME, the first for M-state or E-state and the second for I-state
  • for exiting extracode routines, two encoding patterns of the first four bits of the ten-bit function code will pair a non-branching instruction with a return to M-state after the instruction had been executed
  • for exiting interrupt routines, it appears that the interrupt mode bit I/ME was directly reset; Stinson writes:

    At the end of each interrupt routine the control was transferred to the beginning again to check for any further interrupt requests to allow these to be dealt with. If no interrupts required servicing the exit from the interrupt servicing was performed. This simply involved setting the I/ME bistable back to the ME state. Control would then continue obeying orders from the specified cells in either B126 or B127 depending on the machine being in E-state or M-state.

  • Atlas documentation describes how areas of memory were protected from normal user program access:

    It is not possible for either the subsidiary store or the V-store to be used by a programmer accidentally. A lockout is provided to ensure that if these stores are referred to by main control (i.e. when a programme in the main store is being obeyed) an automatic interrupt occurs. For this reason these stores are referred to as the "private store". The use of these stores is only permitted when a programme in the fixed store is being obeyed (more precisely using either the extracode or interrupt controls).

• BBN modifications to PDP-1 (1962)

  "The modifications to the PDP-1 to effect timesharing were embodied in the 'restricted mode' of operation. They matched the above requirements in the following way:

  1. Memory protection. Switching between the two 4-Kword areas required the use of an I/O instruction.
  2. Program and data relocatability. Because only one user was resident at one time, this was not needed.
  3. A supervisor program. The channel 17 clock routine fulfilled this function.
  4. A timed return to the supervisor. The channel 17 clock generated an interrupt every 20 milliseconds.
  5. Interpretive execution of I/O instructions. Whenever the PDP-l was in restricted mode, an attempt to obey an I/O instruction caused a sequence break [i.e., an interrupt]."

  -- from Bell, et al., "The PDP-1 and other 18-bit computers," Bell, Mudge, and McNamara, chapter 6, pp. 140-141.

• Burroughs D825 (1962)

  The occurrence of any interrupt causes one of the system computer modules to leave the program it has been running and branch to the suitable AOSP entry, entering a control mode as it branches. The control mode differs from the normal mode of operation in that it locks out the response to some low-priority interrupts (although recording them) and enables the execution of some additional instructions reserved for AOSP use (such as setting an interrupt mask register or memory protection registers, or transmitting an I/O instruction to an I/O control module).

  -- from Anderson, et al., "D825 - a multiple-computer system for command and control," FJCC 1962, reprinted in Bell and Newell, chapter 36, p. 452

• MIT 7094 modified for CTSS (1963, described by Corbato at 1962 SJCC)

  "The hardware RPQs that made CTSS run on MIT's 7094s added several instructions and a memory boundary register to the processor: in addition, the machines had two 32K core memory banks instead of one. Core bank A held the CTSS supervisor, and B Core was used for user programs. ... More importantly, the RPQ made the 7094 a two-mode machine. When the machine was running a program in B-core, many instructions were forbidden: executing them caused the machine to trap and take its next instruction from A-core. In particular, I/O instructions were forbidden, as well as attempts to switch core banks.

  By convention, user programs called on supervisor services, such as the file system, by executing a TSX ADDR,4 where ADDR held TIA =HWRFLXA. The TIA instruction was illegal in B-core mode, so the CPU trapped into the A-core supervisor module named PMTI (Protection Mode Trap Interpreter), which looked up the BCD name of the supervisor entrypoint, found its location, and made the transfer."

  -- from Tom Van Vleck, "The IBM 7094 and CTSS"

• GE-635 (1963)
  • master and slave execution modes
  • MME (master mode entry) and DRL (derail) instructions for making a system call
  • TSS (transfer and set slave) instruction or return instruction with bit 28 set to 0 to leave master mode

• PDP-6 (1963-1964)
  • executive and user execution modes
  • 64 opcodes are reserved for "programmed operators" that trap into the monitor program and provide system calls
  • JRST (jump and restore) instruction to leave executive mode

• IBM S/360 (1964)
  • supervisor and user execution modes
  • SVC (supervisor call) instruction for making a system call
  • LPSW (load PSW) instruction to leave supervisor mode

• Multics - rings of protection and gates for up-calls
  • Mike Schroeder and Jerry Saltzer's 1971 paper, "A hardware architecture for implementing protection rings," includes these statements in the acknowledgements section:
    • "The original idea of generalizing the supervisor/user relationship to a multiple ring structure was suggested by R.M. Graham, E.L. Glaser and F.J. Corbató."
    • "The hardware implemented call and return, and automatically managed stacks, were at least partly inspired by similar mechanisms which have long been used on computer systems of the Burroughs Corporation."
  • Robert Graham's paper, "Protection in an information processing utility," first appeared in January 1967 at ACM SOSP and was then republished in CACM in 1968.
  • https://multicians.org/mgr.html#ring states Although there were 64 rings (an illusion created by software) on GE 645 Multics, there are 8 rings, supported in hardware, on the Honeywell 6180 in its various models.
  • https://multicians.org/fjcc2.html describes the three modes of execution implemented in the GE 645: absolute, master, and slave
  • The first Honeywell 6180 was delivered to MIT in 1972, according to https://multicians.org/mga.html#6180
  • Schroeder and Saltzer also mention the 1969 paper on the Hitac 5020, which cites the Multics work and describes its memory protection scheme using 4-bit locks and keys, with actions defined for key==0, key<lock, key==lock, and key>lock.

• capability history [e.g., see Hank Levy's book, now on-line]

• see the reflections on interrupts, capability addressing, and OS design by Peter J. Denning, "Principles for Reliable Operating Systems," January 28, 2003

Examples

• (TBD) running RSX-11 on low end PDP-11 models with no protection - use EMT (emulator trap) to make a system call, with the "execution mode" recorded in the STKDP variable

• IBM S/360

• VAX

some comp.arch traffic

Paul Repacholi writes

> Didn't the DEC KL-10 have "call gates" that were implemented
> in just this fashion...?

The PORTAL intruction? That worked by triggering a MM fault on the fetch, which the PORTAL than stepped on before it got out of the Mbox. If you jumped to other than the PORTAL, the MM fault happened and got to the kernal. It was a bit odd in that it could controll access to code in user mode, that you could only execute, not read, and further, you could only enter it at specific points with out MM faulting.

Dale Morris writes:

PA-RISC implements this sort of mechanism in its GATEWAY instruction, which can be used to branch and promote to any of 3 privilege levels. The instruction TLB mapping (for the page containing the GATEWAY instruction) provides information on what privilege level to promote to, and whether the GATEWAY instruction is legal (GATEWAY instructions on normal code pages fault).

Itanium also implements a similar mechanism with the EPC instruction. It works similarly, but doesn't branch - it only does the privilege promotion. In Itanium, privilege can also be changed on a normal br.ret, allowing system call code to simply return to the calling process. However, since the privilege to return to is stored in an unprivileged register, we also had to guard against an application trying to spoof a system call into returning at high privilege. Therefore, the EPC also validates the return privilege level (to ensure that it's no higher than the current executing privilege level) before it promotes.

added Jan. 2010: "SYSENTER: a good idea, taken too far..." [Andy Glew's description of x86 SYSENTER/SYSEXIT]

Acknowledgement: My thanks to Earl Killian for his help in identifying early machines with multiple modes of execution.

[History page] [Mark's homepage] [CPSC homepage] [Clemson Univ. homepage]

mark@cs.clemson.edu