Draft Specification NDMP Version 4 Protocol April 2003 Network Working Group Harald Skardal INTERNET DRAFT Network Appliance, Inc. Category: Applications James Bunnell Document: draft-skardal-ndmpv4-04.txt Spectra Logic Sudakar V. Chellam IBM Tim Gardner Chewcoba Systems, Inc. Clive Hendrie BlueArc Corporation Kiyoshi Komatsu Network Appliance, Inc. Greg Linn Network Appliance, Inc. Dave Manley Independent Gordon Waidhofer Traakan Jim Ward Reliaty, Inc. Network Data Management Protocol Version 4 Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or become obsolete by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Expires October 2003 [Page 1] Draft Specification NDMP Version 4 Protocol April 2003 Abstract The Network Data Management Protocol (NDMP) defines a mechanism and protocol for controlling backup, recovery, and other transfers of data between primary and secondary storage. The NDMP architecture separates the network attached Data Management Application (DMA), Data Servers and Tape Servers participating in archival or recovery operations. NDMP also provides low-level control of tape devices and SCSI media changers. The XDR and TCP/IP protocols are foundations for NDMP. The key goals of NDMP include interoperability, contemporary functionality, and extensibility. Copyright Copyright (C) The Internet Society (2001). All Rights Reserved. Table of Contents 1. Overview......................................................6 1.1. Motivation..................................................6 1.2. Audience....................................................6 1.3. Terminology.................................................6 1.4. Key Words...................................................9 2. Architecture.................................................10 2.1. Architectural Model........................................10 2.2. NDMP Topologies............................................10 2.2.1. Simple NDMP Configuration................................11 2.2.2. NDMP Two Drive Configuration.............................12 2.2.3. Tape Library Configuration...............................13 2.2.4. Three-Way Configuration..................................14 2.2.5. Data Replication Configurations..........................14 2.3. Key NDMP Concepts..........................................17 2.3.1. Session State............................................17 2.3.2. Control Streams..........................................17 2.3.3. Data Streams.............................................18 2.3.4. NDMP Services............................................18 2.3.4.1. Data Service...........................................19 2.3.4.2. Tape Service...........................................19 2.3.4.3. SCSI Pass Through Service..............................19 2.3.5. Other Mechanisms.........................................19 2.3.5.1. Tape Format and Mover Window...........................19 2.3.5.1.1. Mover Records........................................20 2.3.5.1.2. Tape Content Ownership...............................20 2.3.5.1.3. Control of the Tape Drive............................20 2.3.5.1.4. Mover Window.........................................20 2.3.5.1.5. Mover Window Usage While Writing.....................21 2.3.5.1.6. Mover Window Usage While Reading.....................22 2.3.5.2. File History...........................................22 2.3.5.3. Direct Access Recovery.................................23 Expires October 2003 [Page 2] Draft Specification NDMP Version 4 Protocol April 2003 2.4. Character and Role.........................................23 2.5. Protocol Interfaces........................................24 2.5.1. Messages from DMA to NDMP Server.........................24 2.5.2. Messages from the NDMP Server to the DMA.................25 2.5.3. Optional Interfaces and Messages.........................26 2.5.4. NDMP Server Extensions...................................27 2.5.4.1. Proprietary vs. Standard Extensions:...................27 2.5.4.2. The Class..............................................28 2.5.4.2.1. Class Versions.......................................28 2.5.4.2.2. Class Version vs. Core NDMP Version..................29 2.5.4.3. Discovery and Negotiation..............................29 2.5.4.4. Extension Management...................................30 2.5.4.4.1. The NDMP Class Space Allocation......................30 2.5.4.4.2. Extension Allocation and Management..................30 2.6. Messaging Protocol.........................................31 2.7. Message Header.............................................31 2.8. Error Reporting............................................33 2.8.1 Error Codes In Core NDMP..................................33 2.8.2 Error Codes in NDMP Extensions............................37 2.9. Message Numbers............................................37 2.10. Message Definitions.......................................39 2.11. Message Sequencing and State Tables.......................40 2.11.1. General Rules...........................................40 2.11.2. Connection..............................................40 2.11.3. Authentication..........................................41 2.11.4. SCSI and Tape Devices...................................42 2.11.5. Data State Diagram......................................42 2.11.5.1. Example Race Condition................................46 2.11.6. Mover State Table.......................................47 2.12. Supporting XDR Definitions for NDMP.......................51 2.13. Protocol Version Compatibility............................59 3. NDMP Server Interfaces.......................................60 3.1. Connect Interface..........................................60 3.1.1. NDMP_CONNECT_OPEN........................................61 3.1.2. NDMP_CONNECT_CLIENT_AUTH.................................63 3.1.3. NDMP_CONNECT_CLOSE.......................................66 3.1.4. NDMP_CONNECT_SERVER_AUTH.................................67 3.2. Config Interface...........................................69 3.2.1. NDMP_CONFIG_GET_HOST_INFO................................70 3.2.2. NDMP_CONFIG_GET_SERVER_INFO..............................71 3.2.3. NDMP_CONFIG_GET_CONNECTION_TYPE..........................73 3.2.4. NDMP_CONFIG_GET_AUTH_ATTR................................75 3.2.5. NDMP_CONFIG_GET_BUTYPE_INFO..............................77 3.2.6. NDMP_CONFIG_GET_FS_INFO..................................82 3.2.7. NDMP_CONFIG_GET_TAPE_INFO................................85 3.2.8. NDMP_CONFIG_GET_SCSI_INFO................................87 3.2.9 NDMP_CONFIG_GET_EXT_LIST..................................89 3.2.10 NDMP_CONFIG_SET_EXT_LIST.................................91 3.3. SCSI Interface.............................................93 3.3.1. NDMP_SCSI_OPEN...........................................94 3.3.2. NDMP_SCSI_CLOSE..........................................96 3.3.3. NDMP_SCSI_GET_STATE......................................97 3.3.4. NDMP_SCSI_RESET_DEVICE...................................98 3.3.5. NDMP_SCSI_EXECUTE_CDB....................................99 3.4. Tape Interface............................................102 Expires October 2003 [Page 3] Draft Specification NDMP Version 4 Protocol April 2003 3.4.1. Tape Model..............................................102 3.4.2. NDMP_TAPE_OPEN..........................................104 3.4.3. NDMP_TAPE_CLOSE.........................................106 3.4.4. NDMP_TAPE_GET_STATE.....................................108 3.4.5. NDMP_TAPE_MTIO..........................................111 3.4.6. NDMP_TAPE_WRITE.........................................115 3.4.7. NDMP_TAPE_READ..........................................118 3.4.8. NDMP_TAPE_EXECUTE_CDB...................................121 3.5. Data Interface............................................122 3.5.1. Data Interface Overview.................................122 3.5.1.1. Data Interface Variables & Constants..................123 3.5.2. Data Message Definitions................................127 3.5.2.1. NDMP_DATA_CONNECT.....................................128 3.5.2.2. NDMP_DATA_LISTEN......................................130 3.5.2.3. NDMP_DATA_START_BACKUP................................133 3.5.2.4. NDMP_DATA_START_RECOVER...............................136 3.5.2.5. NDMP_DATA_START_RECOVER_FILEHIST......................142 3.5.2.6. NDMP_DATA_GET_STATE...................................146 3.5.2.7. NDMP_DATA_GET_ENV.....................................147 3.5.2.8. NDMP_DATA_STOP........................................149 3.5.2.9. NDMP_DATA_ABORT.......................................150 3.6. Mover Interface...........................................151 3.6.1. Mover Interface Overview................................151 3.6.1.1. Mover Interface Variables & Constants.................152 3.6.2. Mover Message Definitions...............................158 3.6.2.1. NDMP_MOVER_SET_RECORD_SIZE............................160 3.6.2.2. NDMP_MOVER_SET_WINDOW.................................162 3.6.2.3. NDMP_MOVER_CONNECT....................................165 3.6.2.4. NDMP_MOVER_LISTEN.....................................168 3.6.2.5. NDMP_MOVER_READ.......................................172 3.6.2.6. NDMP_MOVER_GET_STATE..................................175 3.6.2.7. NDMP_MOVER_CONTINUE...................................176 3.6.2.8. NDMP_MOVER_CLOSE......................................178 3.6.2.9. NDMP_MOVER_ABORT......................................179 3.6.2.10. NDMP_MOVER_STOP......................................180 4. DMA Interfaces..............................................181 4.1. Notify Interface..........................................181 4.1.1. NDMP_NOTIFY_DATA_HALTED.................................182 4.1.2. NDMP_NOTIFY_CONNECTION_STATUS...........................183 4.1.3. NDMP_NOTIFY_MOVER_HALTED................................185 4.1.4. NDMP_NOTIFY_MOVER_PAUSED................................186 4.1.5. NDMP_NOTIFY_DATA_READ...................................187 4.2. Log Interface.............................................188 4.2.1. NDMP_LOG_MESSAGE........................................189 4.2.2. NDMP_LOG_FILE...........................................192 4.3. File History Interface....................................194 4.3.1. NDMP_FH_ADD_FILE........................................195 4.3.2. NDMP_FH_ADD_DIR.........................................198 4.3.3. NDMP_FH_ADD_NODE........................................200 5. Security....................................................201 7. Recognition of Prior Work...................................203 8. Authors and Contributors....................................204 8.1. Document Editor...........................................204 8.2. Authors' Addresses........................................204 8.3. Contributors..............................................205 Expires October 2003 [Page 4] Draft Specification NDMP Version 4 Protocol April 2003 Appendixes:....................................................206 Appendix A: NDMP Extension Management..........................206 Appendix B: NDMP Extensions Test Message.......................209 Appendix C: XDR for an NDMP Implementation.....................212 Appendix D: Workflow...........................................232 D.1. Backup....................................................232 D.2. Data Recovery.............................................236 D.2.1. Recovery Exceptions.....................................239 D.2.1.1. End-of-file...........................................239 D.2.1.2. Media error...........................................240 D.2.1.3. User aborted..........................................241 D.3 Direct Access Recovery.....................................242 D.4 Loss of Data Connection....................................243 D.5 Using a Jukebox............................................244 D.5.1 Backing Up and Restoring Using a Jukebox.................244 D.5.2 Initializing a Jukebox...................................245 D.5.3 Jukebox Exception Handling...............................246 D.6 Tape File Duplication......................................246 D.7 Network Copy...............................................248 D.8 NDMP Exceptions............................................250 D.8.1 End-of-media.............................................250 D.8.2 Media Errors.............................................252 D.8.3 User Aborted.............................................253 D.8.4 Loss of Data Connection..................................254 D.8.5 Broken Connection........................................255 Expires October 2003 [Page 5] Draft Specification NDMP Version 4 Protocol April 2003 1. Overview 1.1. Motivation The purpose of this protocol is to allow a network backup application to control the backup and retrieval of an NDMP compliant server without installing third party software on the server. The control and data transfer components of the backup/recovery are separated. The separation allows complete interoperability at a network level. The file system vendors need only be concerned with maintaining compatibility with one, well-defined protocol. The backup vendors can place their primary focus on the sophisticated central backup administration software. The NDMP protocol is targeted towards the process of backup and recovery. There are extensive references to these tasks. The protocol is specifically intended to support tape drives. However, the protocol can be used for other applications and support other media in the future. 1.2. Audience This document is intended for use by software developers to implement Network Data Management Protocol. The reader is assumed to be familiar with network protocol specifications and with the general operation of backup software. The user is not expected to have knowledge of internal backup software behavior. 1.3. Terminology NDMP or Network Data Management Protocol An open protocol for enterprise-wide, network-based data management such as backup and recovery. NDMP is a control protocol used to control the NDMP services participating in the session. NDMP specifies the format and means of transmission of messages and payload data between a DMA and an NDMP server, and between two NDMP servers. NDMP is increasingly being used for replication/copying of data in primary or secondary storage systems. DMA or Data Management Application The Data Management Application (DMA) that controls the NDMP session. In NDMP there is a master-slave relationship. The DMA is the session master; the NDMP services are the slaves. In NDMP versions 1, 2, and 3 the term "NDMP client" was used instead of “DMA.” NDMP Host The host computer system that executes the NDMP Server application. Data is backed up from the NDMP host to either a local tape drive or to a backup device on a remote NDMP host. Expires October 2003 [Page 6] Draft Specification NDMP Version 4 Protocol April 2003 NDMP Service The state machine on the NDMP host accessed with the Internet protocol and controlled using the NDMP protocol. This term is used independently of implementation. There are three types of NDMP Services: Data Service, Tape Service, and SCSI Service. NDMP Server An instance of one or more distinct NDMP services controlled by a single NDMP control connection. Thus a Data/Tape/SCSI Server is an NDMP Server providing Data, Tape, and SCSI services. NDMP Session The configuration of one DMA and two NDMP services to perform a data management operation such as a backup or a recovery. Primary Storage System A storage system that stores live or "in production" data on an active file system. Examples are direct or SAN attached storage in application servers, or dedicated storage appliances such as filers. A primary storage system hosts an NDMP data service. Secondary Storage System A storage system used for archiving or data protection. Examples are application servers with direct attached tape drives, libraries or robots, or dedicated network attached archiving/data protection appliances. A secondary storage system hosts an NDMP tape service and often a SCSI service. Backup or Backup Operation Copying selected data from primary storage to secondary storage. Recovery or Recovery Operation Copying selected data from secondary storage to primary storage. Backup Data The resulting data from a Backup Operation. Recovery data The resulting data from a recovery Operation. Replication The copying of data between two services of the same type. Examples are data to data service replication or tape to tape service replication. Expires October 2003 [Page 7] Draft Specification NDMP Version 4 Protocol April 2003 Control Connection A bi-directional TCP/IP connection that carries XDR encoded NDMP messages between the DMA and the NDMP Server. Data Connection The connection between the two NDMP Servers that carry the data stream. The data connection in NDMP is either an NDMP interprocess communication mechanism (for local operations) or a TCP/IP connection (for 3-way operations). Data stream A unidirectional byte stream of data flowing over a data connection between two peer NDMP Services in an NDMP session. For example, in a backup, the data stream is generated by the data service and consumed by the tape service. The data stream can be backup data, recovered data, etc. Data Service A NDMP Service that transfers data between primary storage and the Data Connection. Tape Service A NDMP Service that transfers data between secondary storage and the Data Connection and allows the DMA to manipulate and access secondary storage. Mover An aspect of the Tape Service that transfers data between the secondary storage and the Data Connection. SCSI Service A NDMP Service that passes low-level SCSI commands to a SCSI device, typically used by the DMA to manipulate a SCSI or fibre channel attached media changer. DAR or Direct Access Recovery An optional capability of NDMP Data and Tape Services whereby only relevant portions of secondary media are accessed during Recovery Operations. Expires October 2003 [Page 8] Draft Specification NDMP Version 4 Protocol April 2003 1.4. Key Words The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. Expires October 2003 [Page 9] Draft Specification NDMP Version 4 Protocol April 2003 2. Architecture 2.1. Architectural Model The NDMP architecture is based on a client-server model. NDMP compliant backup software, which is referred to as the Data Management Application or DMA, is considered to be the client. A DMA interacts with one or more NDMP Servers, managing the transfer of data between server resident NDMP data and tape services. Each instantiation of a NDMP data or tape service is represented as a virtual state machine on the NDMP Server. Data services provide an abstracted interface to the file system or primary storage of the NDMP Server. A data service is the source of data during backup operations and the destination of data during recovery operations. Examples of data services are file servers and general compute platforms with direct or SAN attached storage. Tape services provide an abstracted interface to tape devices or other types of secondary storage attached to the NDMP Server. A tape library can implement its own NDMP Server and associated tape service or it can be connected through an external NDMP Server. A tape service is the source of data during recovery operations and the destination of data during backup operations. The tape service also provides a mechanism for tape positioning and I/O on behalf of the DMA. Examples of tape services are individual tape drives, tape libraries, or servers with one or more writeable CD ROM drives. An NDMP session is an instantiation of a pair of NDMP services with data connections between the two services and control connections between the DMA and each service. The DMA creates and controls the NDMP session. It is responsible for managing all session state required to fully or partially recover a file system including server topology, tape sets, numbering and so on. There is exactly one bi-directional TCP/IP control connection between the DMA and each NDMP Server. If the DMA is distributed in such a way that two or more DMA processes need to communicate to one NDMP service, the DMA commands MUST be merged into a single control connection to the NDMP Server. The NDMP protocol is a set of XDR encoded messages. These messages control and monitor the state of each NDMP service. The messages also report detailed information about the NDMP session and the data that is manipulated. 2.2. NDMP Topologies This section describes typical NDMP topologies and configurations in terms of the relationship between DMAs and NDMP Servers that provide Data and Tape services. Expires October 2003 [Page 10] Draft Specification NDMP Version 4 Protocol April 2003 2.2.1. Simple NDMP Configuration In the simplest configuration, a DMA backs up the data from the NDMP Server to a locally attached tape subsystem. The NDMP control connection exists across the network boundary. The NDMP data connection between the data and tape services exists within the NDMP Server implementation. +------------------------------+--------------------------------+ | DMA * NDMP Server | | * | | * | | +-------------+ * | | | NDMP Data | * | | | Management | <-----------------------+ | | | Application | * | | | +-------------+ * Network | | | * Boundary | | | * | | |******************************* | | | | NDMP Control | | | Connection | | V | | +-------------+ | | | NDMP | | | | Data & Tape | | | | Services | | | +-------------+ | | ^ NDMP | | | | Data V | | +-----------+ +-----------+ | | | File | | Tape | | | | System | | Subsystem | | | +-----------+ +-----------+ | | | +---------------------------------------------------------------+ Figure 1. Simple configuration Expires October 2003 [Page 11] Draft Specification NDMP Version 4 Protocol April 2003 2.2.2. NDMP Two Drive Configuration NDMP can also simultaneously back up to multiple backup devices physically attached to the NDMP Server. In this configuration, there are two instances of the NDMP data and tape services on the NDMP Server. The NDMP control connection exists across the network boundary. The NDMP data connections between the data and tape services exist within the NDMP Server implementation. +------------------------------+--------------------------------+ | DMA * NDMP Server | | * | | * | | +-------------+ * | | | NDMP Data | * | | | Management | <-----------------------+ | | | Application | * | | | +-------------+ * Network | | | ^ * Boundary | | | | * | | |*************|***************** | | | | NDMP Control | NDMP Control | | | Connection | Connection | | v V | | +-------------+ +-------------+ | | | NDMP | | NDMP | | | | Data & Tape | | Data & Tape | | | | Services | | Services | | | +-------------+ +-------------+ | | ^ NDMP | ^ NDMP | | | | Data V | Data V | |+-----------+ +-----------+ +-----------+ +-----------+ | || File | | Tape | | File | | Tape | | || System | | Subsystem | | System | | Subsystem | | |+-----------+ +-----------+ +-----------+ +-----------+ | | | +---------------------------------------------------------------+ Figure 2. Two-drive configuration Expires October 2003 [Page 12] Draft Specification NDMP Version 4 Protocol April 2003 2.2.3. Tape Library Configuration NDMP can back up data to a tape library that is physically attached to the NDMP Server. In this configuration a separate instance of the NDMP Server control the robotics within the tape library. +------------------------------+--------------------------------+ | DMA * NDMP Server | | * | | * | | +-------------+ NDMP Control +-------------+ | | | NDMP Data | Connection | NDMP | | | | Management | <------------------>| Tape & Data |--+ | | | Application | * | Service | | | | +-------------+ * +-------------+ | | | ^ * Network ^ | | | | * Boundary | | | |************|****************** +---------+ | | | | NDMP Control | File | | | | | Connection | System | | | | v +---------+ | | | +------------+ | | | | NDMP Tape | | | | | Service | NDMP Data | | | | | Connection | | | +------------+ | | | | | | | | +---------------------------------+ | | | | | Tape Library | | | | | | +-----------+ +-----------+ | | | | +----->| Robotic | | Tape |<---------+ | | | | Control | | Subsystem | | | | | +-----------+ +-----------+ | | | +---------------------------------+ | | | +---------------------------------------------------------------+ Figure 3. Tape library configuration Expires October 2003 [Page 13] Draft Specification NDMP Version 4 Protocol April 2003 2.2.4. Three-Way Configuration One may back up an NDMP Server that supports NDMP but does not have a locally attached backup device by sending the data through a TCP/IP connection to another NDMP Server. In this configuration, the NDMP data service exists on one NDMP Server and the NDMP tape service exists on a separate server. Both the NDMP control connections (to server 1 and server 2 and the NDMP data connection (between server 1 and server 2 exist across the network boundary. +-------------------+----------------------+--------------------+ | NDMP Server 1 * DMA * NDMP Server 2 | | * * | | * * | | NDMP Control * +--------------+ * NDMP Control | | Connection * | NDMP Data | * Connection | | +------------>| Management |---*---------+ | | | * | Application | * | | | | * +--------------+ * | | | | * * | | | | * Network Boundary * | | | | ************************ | | | | * | | | V * V | |+------------------+ NDMP Data +------------------+ | || NDMP Data | Connection | NDMP Tape | | || Service | -------------------->| Service | | |+------------------+ * +------------------+ | | ^ * | | | |Backup * | Backup | | |Data * | Data | | | * Network V | | +-----------+ * Boundary +-----------+ | | | File | * | Tape | | | | System | * | Subsystem | | | +-----------+ * +-----------+ | | * | +---------------------------------------------------------------+ Figure 4. Three-way configuration 2.2.5. Data Replication Configurations In addition to backup and recovery operations, NDMP supports replication of data between two services of the same type. The two cases in NDMP are: - Tape to tape replication: replicating a set of backup tapes. - Data to data replication: replicating data between two primary storage systems. Expires October 2003 [Page 14] Draft Specification NDMP Version 4 Protocol April 2003 In the tape replication configuration one tape service performs a read operation while the other performs a write operation. This allows tape data to be copied from one NDMP tape service to another. The tape to tape copy is useful when duplicating backup tapes for offsite storage. +-------------------+----------------------+--------------------+ | NDMP Server 1 * DMA * NDMP Server 2 | | * * | | * * | | NDMP Control * +--------------+ * NDMP Control | | Connection * | NDMP Data | * Connection | | +------------>| Management |---*---------+ | | | * | Application | * | | | | * +--------------+ * | | | | * * | | | | * Network Boundary * | | | | ************************ | | | | * | | | V * V | |+------------------+ NDMP Data +------------------+ | || NDMP Tape | Connection | NDMP Tape | | || Service | -------------------->| Service | | |+------------------+ * +------------------+ | | ^ * | | | |Recovery * | Backup | | |Data * | Data | | | * Network V | | +-----------+ * Boundary +-----------+ | | | Tape | * | Tape | | | | Subsystem | * | Subsystem | | | +-----------+ * +-----------+ | | * | +---------------------------------------------------------------+ Figure 5 Tape to tape configuration Expires October 2003 [Page 15] Draft Specification NDMP Version 4 Protocol April 2003 NDMP also supports replication between two primary storage systems. In this configuration, one data service performs a backup operation while the other performs a recovery operation on the same data stream. This capability is useful for performing a logical duplication of a portion of a file system (data migration). +-------------------+----------------------+--------------------+ | NDMP Server 1 * DMA * NDMP Server 2 | | * * | | * * | | NDMP Control * +--------------+ * NDMP Control | | Connection * | NDMP Data | * Connection | | +---------*-->| Management |---*---------+ | | | * | Application | * | | | | * +--------------+ * | | | | * * | | | | * Network Boundary * | | | | ************************ | | | | * | | | V * V | |+------------------+ NDMP Data +------------------+ | || NDMP Data | Connection | NDMP Data | | || Service | -------------------->| Service | | |+------------------+ * +------------------+ | | ^ * | | | |Backup * | Recovery | | |Data * | Data | | | * Network V | | +-----------+ * Boundary +-----------+ | | | File | * | File | | | | System | * | System | | | +-----------+ * +-----------+ | | * | +---------------------------------------------------------------+ Figure 6 Data to data configuration Expires October 2003 [Page 16] Draft Specification NDMP Version 4 Protocol April 2003 2.3. Key NDMP Concepts The NDMP architectural model is focused around the creation and management of control connections and data streams in an NDMP session. Data from a volume or file system is turned into a data stream by a data service, a tape service takes a stream, converts it into a tape format and writes it to tape, or vice versa. The role of the NDMP protocol is to allow the DMA to set up, configure, and control an NDMP session of NDMP Servers and services. 2.3.1. Session State In NDMP, the DMA is responsible for capturing and managing all state needed to provide the desired capabilities such as enabling partial recoveries of data. In addition, the DMA is responsible for media management. The NDMP service only keeps local running state. The DMA will poll the state from the NDMP services in order to record sufficient information to enable optimized access to subsets of the backed up data on the tapes. There are several benefits to an architecture where state is centralized to one place, and the other components are “state lean:” - The architecture is simpler. - The protocol commands and event notifications are clearer and simpler. - The state diagrams are simpler. - The code becomes simpler and more supportable. The net result is more robust products, and therefore a more robust data management environment. 2.3.2. Control Streams Between the DMA and every NDMP Server is one and only one control connection. The DMA uses this control connection to manage each NDMP service associated with the server. The control connection is implemented as a TCP/IP connection. Messages flow in both directions on a control connection. The DMA sends messages to the NDMP Server for the purpose of managing the operations of its NDMP services. The NDMP Server sends notifications to the DMA when a NDMP service requires the DMA’s attention. The NDMP Server also uses the control connection to send file history information about the content of the data stream. Expires October 2003 [Page 17] Draft Specification NDMP Version 4 Protocol April 2003 2.3.3. Data Streams NDMP data streams convey implementation specific backup or recovery data between NDMP services. In the native or general case the transport for NDMP data streams is TCP/IP over any IP supported network media. In NDMP each service may return multiple IP addresses to the DMA, addresses on which the service will be listening for a connection from the other service. This is done to accommodate the following: - Many servers have multiple connections to an IP network. These connections may offer different performance characteristics. - Different network segments may be created for different purposes: one for communication between users and servers, one for communication between applications servers and dedicated storage, one for specific data management such as backup, recovery and replication, etc. - Specific network segments may provide important service level characteristics such as guaranteed minimum bandwidth etc. that would guarantee a maximum time for an NDMP session. - Fiber Channel SANs are established, and VI and Infiniband are emerging as future high-speed technologies for data management. IP is the global addressing mechanism; it is also being used to address non-IP node connections such as for FC and VI. NDMP sessions must be able to take advantage of the ways the networking infrastructure is constructed. It is therefore the recommendation that the DMA, possibly with the assistance of a general directory service, understands the capabilities and purpose of the network topology, and thus makes the determination about which network connections to use. The NDMP services will make a simple recommendation for use of connections based on basic information such as the bandwidth of a NIC. Notice that each server still MUST be connected to the LAN for TCP/IP based NDMP control connections. 2.3.4. NDMP Services The NDMP services are the NDMP interfaces to the storage devices. Data services interface to primary storage devices such as servers with storage subsystems or filers, tape services interface to secondary storage devices such as tape drives or writeable CDROMs. The NDMP services are controlled by the DMA through a set of service parameters. There are two types of service parameters, service parameters that impact the NDMP protocol or state, and “NDMP opaque” service parameters that only impact vendor specific state in a NDMP service. Expires October 2003 [Page 18] Draft Specification NDMP Version 4 Protocol April 2003 2.3.4.1. Data Service A data service provides the NDMP interface to a primary storage device such as a filer, a compute server with direct or SAN attached storage, or a read-only CDROM library. The data service allows a DMA to read or write all or a subset of a volume or a file system, for the purpose of a backup or a recovery. The data service reads or writes one single data stream for backup or recovery, respectively. 2.3.4.2. Tape Service A tape service provides the NDMP interface to a secondary storage device, such as a tape drive or a writeable CDROM. The tape service reads or writes a single data stream, for backup or recovery respectively. Note that a tape service only handles the reading or writing of one “cartridge,” a single tape or CD. The tape service when notify the DMA when a cartridge is read or written, the DMA will provide the necessary media management. 2.3.4.3. SCSI Pass Through Service The SCSI pass through service allows a DMA to issue SCSI commands to a device such as a tape robot. This service allows the DMA to control other devices such as a tape or CD media changer. 2.3.5. Other Mechanisms Two additional important NDMP concepts are the mover window, the file history, and their use in direct access recovery. 2.3.5.1. Tape Format and Mover Window Tapes recorded by NDMP-conformant software contain one of two types of data: - backup image data, generated by the data service and recorded by the mover, and - metadata, generated by the DMA and recorded by the Tape service The portions of the tape to which backup image data are written, and from which those data are read, are specified by the DMA. Though this placement is under DMA control, content of backup image data is controlled by the data service. Placement and content of metadata are controlled exclusively by the DMA. Expires October 2003 [Page 19] Draft Specification NDMP Version 4 Protocol April 2003 2.3.5.1.1. Mover Records Backup image data is recorded by the mover in units of mover records. The size of a mover record is equal to the size of a tape block if the tape drive is operating in variable block size mode. (That is, specifying mover record size using NDMP_MOVER_SET_RECORD_SIZE implicitly defines the tape block size.) For tape drives operating in fixed block size mode, the mover record size is an integer multiple of that fixed block size. 2.3.5.1.2. Tape Content Ownership Each tape data block contains either backup image data or metadata, but never both. Backup image data consists solely of tape data blocks. Metadata may include tape data blocks, file marks, and other out-of-band information, the specifics of which are beyond the scope of this specification. Thus, each tape data block or instance of out- of-band information is "owned" by exactly one of the mover or the DMA. It is the DMA's responsibility to manage this distinction and ensure that the mover operates only on tape data blocks it owns. 2.3.5.1.3. Control of the Tape Drive Operationally, when a tape drive is opened, it is controlled by the DMA and accessed via NDMP Tape interface requests. When the DMA is ready for backup image data to be transferred between a data service and a mover, presumably after transferring metadata and positioning the tape, it passes control of the tape drive to the mover. This occurs when the DMA issues a NDMP_MOVER_LISTEN or NDMP_MOVER_CONNECT request. Once the DMA hands control of the tape drive to the mover, the DMA is not permitted to perform NDMP Tape requests until the mover returns control to the DMA. This occurs when the mover pauses or halts. 2.3.5.1.4. Mover Window A mover window is the means by which the DMA constrains the mover to use a certain, contiguous set of tape blocks. A mover window identifies a range of tape data blocks that a mover may operate on when it is given control of the tape drive. It is expressed in terms of an offset in bytes from the beginning of the backup image data, and a length in bytes. The offset corresponds to the first byte of the first mover record to be transferred when the DMA passes control of the tape drive to the mover. The length, when divided by the mover record size, identifies the total number of mover records represented by the window. Mover windows always begin and end on mover record boundaries; they do not span tapes. Exactly one mover window is in effect at all times a mover is operating. (Certain events, however, cause that window to be made invalid.) A mover suppresses any attempt to operate outside the bounds of the mover window. If such attempt is made, the mover transitions to the PAUSED state. Expires October 2003 [Page 20] Draft Specification NDMP Version 4 Protocol April 2003 2.3.5.1.5. Mover Window Usage While Writing A mover writing to a tape (e.g., performing a backup operation) considers only the mover window length. Although the offset is specified in requests and replies, its value is ignored except when checked by the mover to ensure it aligns with a mover record boundary. Should a mover writing to a tape encounter the end of the mover window, it pauses with the NDMP_MOVER_PAUSE_EOW reason code. Should such mover encounter logical end of medium before the end of the mover window, it pauses with the NDMP_MOVER_PAUSE_EOM reason code. Consider the following sample tape content. Each "d" indicates that a 10 byte data block -- containing one mover record -- is recorded; "f" indicates a file mark. "m" or "i" identifies the tape content as metadata or backup image data. file# 0 0 0 0 0 0 0 0 0 1 1 1 block# 0 1 2 3 4 5 6 7 8 0 1 2 BOT EOD | d d d d d d d d f d d f | | m m m i i i i m m i m m | Assume for the sake of example that the DMA wishes to write metadata, consisting of one data block and a file mark, every 4 mover records. Assume, too, that the data service has 50 bytes to write. (The mover, though, has no a priori knowledge of this.) After opening the tape and writing 3 blocks of metadata, the DMA sets the mover record size to 10 bytes and the mover window to offset 0, length 40. It then requests the data service to begin the backup operation. The mover writes the 1st 40 bytes of the backup image data as 4 consecutive 10 byte data blocks. Attempting to write the 5th data block causes the mover to pause with an end-of-window reason code. Note that the EOW condition is detected only when an attempt is made to write beyond the window limit. Following the EOW, the DMA regains control of the tape drive, writes a block of metadata and a file mark using the NDMP Tape interface. It establishes a new mover window by again setting the offset to 0 and length to 40, then requests the mover to continue. After writing one more record, the data service and mover complete, ultimately producing a NDMP_NOTIFY_MOVER_HALTED. The DMA regains control of the tape drive and writes a final data block and file mark. Expires October 2003 [Page 21] Draft Specification NDMP Version 4 Protocol April 2003 2.3.5.1.6. Mover Window Usage While Reading A mover reading from a tape (e.g., performing a recovery operation) correlates the mover window offset to the first byte to be read at the tape position when the mover is passed control of the tape drive. Thus, it honors mover "read" and "seek" requests by reading and seeking among the tape data blocks within the mover window, pausing for assistance from the DMA only when it needs data outside of the window. Consider the example tape format above. To read the 50 bytes of backup image data, the DMA first reads the 3 blocks of metadata at BOT. It sets the mover record size to 10, window offset to 0 and length to 40, then begins the recovery. The mover reads the 1st 40 bytes of the backup image data as 4 consecutive 10 byte data blocks. When attempting to read the 5th record, the mover pauses with a mover seek notification, indicating that the seek position is byte 40. As with a write operation, the EOW condition is detected only when an attempt is made to read beyond the window limit. Following the EOW, the DMA regains control of the tape drive, processes the metadata in tape block 7 and spaces over the file mark that follows. The DMA resets the mover window to offset 40, length 10, and requests the mover to continue. Upon arrival at the end of this window -- and the end of the backup image -- the mover will pause (reason: seek), or remain in the active state but operationally idle. 2.3.5.2. File History Backup data formats such as tar and cpio include metadata in the backup data stream. This includes file name, access control lists, etc. In addition NDMP enables the data service to send file history notifications to the DMA as the backup data is written to the data stream. The file history includes the following information: file name and path, file status information, and file positioning information (the address of the file in the backup data stream). The file locator data in the file history record is in a data service (OS) specific format. To the DMA this information is an opaque string. This means that the DMA will not attempt to interpret it. In order to determine the location of a file in the backup data stream, the DMA will send the complete file history record for the corresponding file history record to the data service, the data service will calculate the starting location and the length of the byte string to be read from the original backup data stream. The DMA will use this data to manipulate the tape service to retrieve the selected data. Expires October 2003 [Page 22] Draft Specification NDMP Version 4 Protocol April 2003 The two typical applications of the file history are: to provide a human readable user interface to the backup data and to provide a basis for direct access recovery. The file history enables the DMA to build a database over all the files in a backup database. This database enables users to locate which backup has the file, when (before the backup time) the file was modified, and other similar information. Notice that the file locator data is not in a DMA readable form. Direct access recovery depends upon accurate file positioning information in the backup data stream. For more on this see the section on direct access recovery. 2.3.5.3. Direct Access Recovery Direct access recovery (DAR) is an optimized data recovery operation based on the use of the mover window function. DAR is used to allow the DMA to directly access backed up data in the middle of a tape set without having to parse the tape set sequentially. This is very useful when backup data tape sets can take many hours to read or write. Direct access recovery is typically achieved as follows. The DMA uses the mover window tool to partition the backup data into segments that are written to tape. The DMA records where these segment are located on the tapes, as well as their start and end address relative to the start of the backup data stream. The DMA receives file history notifications from the data service; these notifications include (in DMA opaque format) the address of the file relative to the start of the backup data image. By using the file addresses the data service can cause NDMP_NOTIFY_MOVER_PAUSED post (reason seek) from the mover service which the DMA can use to compute which segment contains the starting point of a requested file. It can therefore start the recovery process from the beginning of the tape that has the segment, use the mover to move across segments and start reading through the segment to locate the beginning of the file. Direct access recovery requires a tape drive that can generate accurate tape block numbers. Some tape devices do not support this. 2.4. Character and Role An NDMP Server may provide a number of services, for example: a Data Service, a Tape service, and a SCSI Service. An NDMP Server may provide one or more of these services simultaneously. In the most common case of a transfer of data between a disk and a local tape library the NDMP Server might perform all three roles. Expires October 2003 [Page 23] Draft Specification NDMP Version 4 Protocol April 2003 An NDMP Server providing a Data Service is called a Data Server. During the backup, the Data Server reads the data from disk, generates an NDMP data stream using a specified backup format, and returns the file history information, if requested, back to the DMA. During the retrieval, the Data Server reads the NDMP data stream and recovers it back to the disk. The Data Server SHOULD NOT be aware of any backup device or medium issues, e.g. tape size, block size, end of medium and so on. An NDMP Server providing a Tape service is called a Tape Server. During a backup the Tape Server reads data from an NDMP data stream and writes it to tape. During a recovery the Tape Server reads from tape and writes to the NDMP data stream. The Tape Server SHOULD NOT be aware of the backup format, for instance dump or tar. All tape handling functions, such as split image issues MUST be dealt with by this service. An NDMP Server providing a SCSI Service is called a SCSI Server. The SCSI Server is usually but not necessarily co-located with the Tape Server. It passes SCSI control commands from the DMA to a SCSI device that is usually a media auto-changer device. 2.5. Protocol Interfaces NDMP messages are grouped by functionality into several interfaces. An NDMP Server implementation is NOT REQUIRED to implement all of the listed messages. See 2.5.3 for details about optional interfaces and messages. 2.5.1. Messages from DMA to NDMP Server The NDMP Server MUST implement a consistent subset of the following interfaces: Connect Interface This interface allows the NDMP Server to authenticate the client and negotiate the version of protocol used. The Connect Interface is used after a client first establishes a connection to an NDMP Server. Config Interface This interface allows a DMA to discover the configuration of the NDMP Server. The Config Interface discovers NDMP Server configuration and attributes. The Config Interface includes commands for discovering extensions in the server. Extensions are groups of requests used to control server functionality that is not part of the core NDMP Server specification. See section 2.5.4 for details on server extensions. Expires October 2003 [Page 24] Draft Specification NDMP Version 4 Protocol April 2003 SCSI Interface This interface passes SCSI CDBs through to a SCSI device and retrieve the resulting SCSI status. The DMA uses the SCSI Interface to control locally attached tape library media changer. Software on the DMA will construct SCSI CDBs and interprets the returned status and data. The SCSI Interface MAY also exploit special features of SCSI backup devices. Tape interface This interface supports tape positioning and tape read/write operations. The DMA typically uses the Tape interface to write tape metadata. This includes tape labels and information identifying and describing backup data included on the tape. The DMA also uses the Tape interface to position the tape during backups and recoveries. Data Interface This interface initiates backup and recover operations. The DMA provides all the parameters that affect the backup or recovery using the Data Interface. The DMA does not place any constraints on the format of the backup data other than it MUST be a stream of data that can be written to the tape device. Mover Interface This interface controls the reading and writing of backup data from and to a tape device. During a backup the MOVER reads data from the data connection, buffers the data into tape records, and writes the data to the tape device. During a recover the Mover Interface reads data from the tape device and writes the data to the data connection. The MOVER handles tape exceptions and notifies the DMA. 2.5.2. Messages from the NDMP Server to the DMA The NDMP Server implementation MAY send the following messages to the DMA. All the messages that the DMA accepts are asynchronous. None of these messages generates a reply message. Notify Interface These messages enable the NDMP Server to notify the DMA that the NDMP Server requires attention. File History interface These messages enable the NDMP Server to make entries in the file history catalog for the current backup. The DMA uses file history information to track the files contained in each backup and to select and locate files for recovery. Log Interface These messages enable the NDMP Server to make entries in the backup and recovery log. The operator uses the log to monitor the progress and completion status of the backup and recovery operations. The log MAY also be used to diagnose problems. Expires October 2003 [Page 25] Draft Specification NDMP Version 4 Protocol April 2003 2.5.3. Optional Interfaces and Messages An NDMP server MAY omit implementation of certain messages as described below. However, if the NDMP server receives a request that it does not implement, it MUST generate a reply containing the NDMP_NOT_SUPPORTED_ERR error code. This section describes optional large-scale features and lists the associated messages. Certain individual messages are also optional. Where this is the case it is noted in the detailed description of the message in section 3. If a request is not explicitly indicated as optional in its description in section 3 and it is part of the feature set supported by the NDMP Server, the NDMP Server MUST implement that request. If an NDMP Server does not implement one of the features described below then it MUST reject any of the associated requests with error code NDMP_NOT_SUPPORTED_ERR. The NDMP messages can be partitioned functionally into the following four subsets: Core messages The core subset of NDMP requests applicable to all NDMP servers include the following: - All Connect Interface requests. - General-purpose Config Interface requests including NDMP_CONFIG_GET_HOST_INFO, NDMP_CONFIG_GET_SERVER_INFO, NDMP_CONFIG_GET_CONNECTION_TYPE, NDMP_CONFIG_GET_AUTH_ATTRIB, NDMP_CONFIG_GET_EXT_LIST and NDMP_CONFIG_SET_EXT_LIST. All NDMP servers MUST be able to generate the initial NDMP_NOTIFY_CONNECTION_STATUS message and MAY generate NDMP_LOG_MESSAGE. Data service messages The data service is responsible for the interfaces to the file system that is being backed up or recovered. The Data Server feature consists of the following messages: - All Data Interface requests. - Config Interface NDMP_CONFIG_GET_BUTYPE_INFO and NDMP_CONFIG_GET_FS_INFO requests. The Data Server MUST be able to generate an NDMP_NOTIFY_DATA_HALTED message. It MUST be able to generate NDMP_FH_ADD_FILE, and/or NDMP_FH_ADD_DIR/NDMP_FH_ADD_NODE messages if the data it returns in an NDMP_CONFIG_GET_BUTYPE_INFO reply indicates that it will. It MAY also generate NDMP_LOG_FILE notification messages. Expires October 2003 [Page 26] Draft Specification NDMP Version 4 Protocol April 2003 Tape service messages The Tape service provides access to tape drives. It MAY also provide access to media changer devices. The tape access feature consists of the following messages: - All Tape interface and Mover Interface requests. - The Config Interface NDMP_CONFIG_GET_TAPE_INFO request. - The server MUST be able to generate NDMP_NOTIFY_MOVER_PAUSED and NDMP_NOTIFY_MOVER_HALTED messages. SCSI Service messages The SCSI Service provides access to media changer devices. The SCSI Service consists of the following messages: - All SCSI Interface requests. - The Config Interface NDMP_CONFIG_GET_SCSI_INFO request. 2.5.4. NDMP Server Extensions NDMP provides for a server extension mechanism that enables the following: - The NDMP community can develop and standardize new functionality in NDMP without requiring a revision of core NDMP - Implementers can expose proprietary functionality in NDMP Server implementations through NDMP Server extensions - DMAs can discover and negotiate the use of these extensions - Extensions are managed at two levels: standard extensions developed or ratified by the NDMP community, and proprietary extensions developed for the individual implementations - Extensions are versioned, and can evolve over time The following sections describe the architecture of NDMP extensions. 2.5.4.1. Proprietary vs. Standard Extensions: The architecture provides for two classes of extensions. These are proprietary NDMP extensions and standard NDMP extensions. Proprietary extensions are used for exposing proprietary functionality. These extensions are owned by the implementers of NDMP Servers. The functionality is specific to the implementation. There is no requirement to this functionality other than it MUST comply with the NDMP extension architecture. Expires October 2003 [Page 27] Draft Specification NDMP Version 4 Protocol April 2003 Standard extensions are standardized by the NDMP community into separate NDMP standards specifications. The specifications of standard extensions are owned by the standards community. 2.5.4.2. The Class The basic building block in NDMP extensions is the class. All NDMP extensions are implemented in classes and managed on a per class basis. The NDMP code space is 32 bit. The class is a set of 64k NDMP messages with the same value in the upper 16 bits, i.e. there are 64k NDMP classes. The complete message code is defined as "class.message," where "class" and "message" are 16 bit each. Notice that "core NDMP" is technically known as "class 0x0". For convenience the messages in a class is grouped into "interfaces." Interfaces are groups of messages that operate on the same functional module, for instance NDMP Server configuration. Observe that "interface" is only a conceptual and organizational tool, there is no architectural element called interface in NDMP. Therefore the implementer is free to organize the messages in a class as he/she prefers. 2.5.4.2.1. Class Versions Classes are versioned. The version number is a 16 bit unsigned integer. A DMA MUST select only one version of each class it selects to use. The version is decided during the discovery and negotiation phase. (See 2.5.4.3.) The version mechanism is used for several purposes. First it gives implementers a tool to communicate a change in the feature set exposed to NDMP. Secondly it imposes some discipline in that extensions are built and advanced in a structured process. The implementer MAY use versioning at his/her desire. However, in order to get a consistent handling of versions in NDMP implementers SHOULD comply with the following guidelines: The version of a class SHOULD be revised ONLY when the semantics of the class changes. This includes the following cases: - New functionality is added to the class. For instance, a tape library implementer adds a set of tape library management functions to an existing tape library extension. - A bug has been found and it cannot be fixed without changing the semantic definition of one or more messages or message replies in the class. Expires October 2003 [Page 28] Draft Specification NDMP Version 4 Protocol April 2003 - When a function is changed in terms of the number or type of parameters. The class version SHOULD NOT be changed if a bug is detected, and the fix does not change the semantics of any part of the class. 2.5.4.2.2. Class Version vs. Core NDMP Version The versioning scheme for NDMP extensibility is orthogonal to the version of core NDMP. The only requirement is that core NDMP MUST be of version 4 or later. Future versions of core NDMP MUST NOT depend upon any particular extension, standard or proprietary. Core NDMP MUST be a complete functional implementation of a sufficient and necessary set of functionality to allow for the most common data management operations. 2.5.4.3. Discovery and Negotiation Discovery and negotiation is used by the DMA to probe which extensions are supported in the NDMP Servers, and to select a subset of these extensions in the NDMP session. The Discovery and Negotiation (D+N) messages MUST be implemented in core NDMP starting in NDMP version 4. The server MUST be able to handle D+N requests according to the specification. If D+N calls are made, the D+N exchange MUST occur before ANY extension requests are issued by the DMA. The D+N exchange between the requesting DMA and the NDMP Server SHOULD proceed as follows: 1: The DMA requests the list of supported extensions in the NDMP Server by issuing the message NDMP_CONFIG_GET_EXT_LIST. 2: The NDMP Server replies with a list of all extensions, standard and proprietary, that are supported and should be exposed. (*) 3: The DMA selects a subset of the extensions and sends the list of selected extensions to the NDMP Server with the command NDMP_CONFIG_SET_EXT_LIST. 4: The NDMP Server acknowledges the selected extensions. * - Implementers may decide to hide extensions, or to require a more sophisticated authentication or negotiation scheme before an extension can be accessed. This is specific to the implementations. Expires October 2003 [Page 29] Draft Specification NDMP Version 4 Protocol April 2003 The requester (DMA) SHOULD discover and negotiate classes of extensions before attempting to use any extensions. This explicitly determines the set of extension that will be used by the two parties in this session. If a server allows a requester to use extensions without first going through the D+N steps, the server SHOULD assume a default version of a class. It is recommended that the default version is the most recent version of the class. It is highly recommended that the discovery and negotiation process is completed such that the classes and versions to be used are explicitly known by both parties. 2.5.4.4. Extension Management Standard extensions are managed by the standards community. A group of NDMP implementers can propose an extension for standardization, the community will evaluate the proposal in the same way as this specification is evaluated for standardization. Proprietary extensions are owned by the NDMP implementer. Implementers will use multiple extensions, some for internal prototyping, and some for publicly exposing functionality. A small set of classes is therefore allocated to each implementer; this is considered an "extension sandbox". See Appendix A. 2.5.4.4.1. The NDMP Class Space Allocation The class space is selected with the upper 16 bits of the 32-bit NDMP message code. In order to maintain core NDMP, and provide for standard and proprietary extensions, the class space is allocated into separate ranges as follows: Class = 0x0000: Core NDMP. Class = 0x0001 - 0x0007: Standard NDMP extensions. Class = 0x0008 - 0x1fff: Reserved. Class = 0x2000 - 0x7fef: Proprietary extensions. Class = 0x7ff0 - 0x7ffe: Reserved for test use. Class = 0x7fff - 0xffff: Reserved. Notice that a reserved area is allocated at the separation point between standard and proprietary extensions. 2.5.4.4.2. Extension Allocation and Management The code space allocation and management for proprietary extensions is described in Appendix A. Expires October 2003 [Page 30] Draft Specification NDMP Version 4 Protocol April 2003 2.6. Messaging Protocol NDMP consists of NDMP request, reply and post messages sent over a TCP/IP connection. Request messages are sent from the DMA to the NDMP Server, and have corresponding reply messages. Post messages are used by the NDMP Server to pass information to the DMA, and hence have no associated reply messages. NDMP uses the RPC Record Marking (RM) Standard [4]. An NDMP message consists of a message header optionally followed by a message body. A message sequence number identifies each message. This message sequence number is sent as part of the header. Each message is XDR encoded and sent within a single RM record. (See [1] for details of XDR.) Implementation note: The XDR libraries available on UNIX/LINUX platforms include a set of xdrrec functions that provide RPC RM and XDR translation functionality. All NDMP requests (except NDMP_CONNECT_CLOSE) from the DMA to the NDMP Server have associated NDMP reply messages that MUST be returned by the server to indicate success or failure. NDMP post messages from the NDMP server to the DMA do not have associated replies. When a DMA sends a request to the NDMP server it SHOULD wait to receive the reply before sending its next request. If the DMA sends multiple requests without waiting for the reply to a previous request, the NDMP server may either queue the requests and deal with them sequentially or handle them asynchronously. Because it is RECOMMENDED that the DMA wait for a reply before sending the next request, the NDMP server MUST make every effort to reply to requests. In particular, if it receives an unrecognized message or has problems decoding a request with a valid message header it MUST send an NDMP reply message reporting the error. If the NDMP server receives a message for which it cannot decode the message header it MUST discard the message. (This might happen if the RM record is too short to contain the full NDMP header.) If the NDMP Server determines that the session is in an unrecoverable error state then it SHOULD disconnect the TCP connection. This would be the case if the NDMP server received a sequence of malformed messages. 2.7. Message Header A message header starts each message. The message header identifies the message and defines how to de-serialize the arguments and dispatch the message. Expires October 2003 [Page 31] Draft Specification NDMP Version 4 Protocol April 2003 ---------------------------------------------------------- | | | | ndmp_header | message_request | | | | ---------------------------------------------------------- ---------------------------------------------------------- | | | | ndmp_header | message_reply | | | | ---------------------------------------------------------- The following XDR block defines the message header: ndmp_header_message_type { NDMP_MESSAGE_REQUEST, NDMP_MESSAGE_REPLY }; const NDMP_MESSAGE_POST = NDMP_MESSAGE_REQUEST; struct ndmp_header { u_long sequence; u_long time_stamp; ndmp_header_message_type message_type; ndmp_message message_code; u_long reply_sequence; ndmp_error error_code; }; Message header data definitions: sequence The sequence number is a connection local counter that starts at one and increases by one for every message sent. The client and the server both start with one and increase independently. time_stamp The time_stamp identifies the time, in seconds since 00:00:00 GMT, Jan 1, 1970, that the message was sent. message_type The message_type enum identifies the message as either a request/post or a reply message. Note that in order to minimize changes from version 3, request and post messages use the same message_type identifier. Expires October 2003 [Page 32] Draft Specification NDMP Version 4 Protocol April 2003 message_code The message_code field identifies the message. reply_sequence The reply_sequence field MUST be 0 in a request message. In reply messages, the reply_sequence MUST be the sequence number from the request message with which the reply is associated. error_code The error_code field MUST be 0 in request and post messages. In reply messages, the error_code field identifies any problem that occurred receiving or decoding the message. If the error_code value is nonzero, no message body will follow the message header. The complete list of error codes is in the next section. When the NDMP Server receives and decodes a request that has a message field indicating a function that the NDMP Server supports, it MUST create a suitable message_reply. In this case, errors SHOULD be reported by setting the error field in the ndmp_header to NDMP_NO_ERR and setting the error field in the message_reply to the relevant error value. However, DMAs MUST handle any error codes appearing in the error code field in the ndmp_header. 2.8. Error Reporting When the NDMP Server receives a request from the DMA it MUST generate a reply that indicates success or failure. If the NDMP Server does not recognize or support a request it MUST generate an error reply and ignore the request. The error reply in this case MUST either use the ndmp_header error_code field to report NDMP_NOT_SUPPORTED_ERR, or generate a valid message reply body with error set. (See section 2.6 for ndmp_header details.) Core NDMP has a set of error codes that are specified in section 2.8.1. These error codes are also used for errors that occur as a result of extension messages, but the error is best described in the context of core NDMP. In addition, there are error codes specific to each extension. 2.8.1 Error Codes In Core NDMP All possible error codes for core NDMP are listed below. Some of these error codes cover a range of cases. It is strongly RECOMMENDED that the NDMP Server use the Log Interface Log Message request (see 4.2.1) to provide further information. A log type of NDMP_LOG_ERROR SHOULD be used. The following error codes are defined: Expires October 2003 [Page 33] Draft Specification NDMP Version 4 Protocol April 2003 enum ndmp_error { NDMP_NO_ERR = 0, NDMP_NOT_SUPPORTED_ERR = 1, NDMP_DEVICE_BUSY_ERR = 2, NDMP_DEVICE_OPENED_ERR = 3, NDMP_NOT_AUTHORIZED_ERR = 4, NDMP_PERMISSION_ERR = 5, NDMP_DEV_NOT_OPEN_ERR = 6, NDMP_IO_ERR = 7, NDMP_TIMEOUT_ERR = 8, NDMP_ILLEGAL_ARGS_ERR = 9, NDMP_NO_TAPE_LOADED_ERR = 10, NDMP_WRITE_PROTECT_ERR = 11, NDMP_EOF_ERR = 12, NDMP_EOM_ERR = 13, NDMP_FILE_NOT_FOUND_ERR = 14, NDMP_BAD_FILE_ERR = 15, NDMP_NO_DEVICE_ERR = 16, NDMP_NO_BUS_ERR = 17, NDMP_XDR_DECODE_ERR = 18, NDMP_ILLEGAL_STATE_ERR = 19, NDMP_UNDEFINED_ERR = 20, NDMP_XDR_ENCODE_ERR = 21, NDMP_NO_MEM_ERR = 22, NDMP_CONNECT_ERR = 23, NDMP_SEQUENCE_NUM_ERR = 24, NDMP_READ_IN_PROGRESS_ERR = 25, NDMP_PRECONDITION_ERR = 26, NDMP_CLASS_NOT_SUPPORTED_ERR = 27, NDMP_VERSION_NOT_SUPPORTED_ERR = 28, NDMP_EXT_DUPL_CLASSES_ERR = 29, NDMP_EXT_DANDN_ILLEGAL_ERR = 30 }; The following list describes each error code. The errors that are returned in reply to specific requests are described in detail under the relevant message descriptions in section 3. However, there might be cases where the NDMP Server uses other error replies and the DMA MUST be implemented to accept these. In particular there are a number of error codes describing unexpected conditions that can affect any request. These are marked in the following list as Generic Errors. NDMP_NO_ERR No error. Expires October 2003 [Page 34] Draft Specification NDMP Version 4 Protocol April 2003 NDMP_NOT_SUPPORTED_ERR Specified message not supported. This error code is used in all of the following situations: The request forms part of a service (see 2.5.3) that is not implemented by the NDMP Server. The request is an optional message not supported by the NDMP Server. The specific operation included in request is not supported by the NDMP Server. This error code is also used if the message code is unrecognized by the NDMP Server. NDMP_DEVICE_BUSY_ERR Specified device is in use. This error is used in two circumstances. It is used when an NDMP_SCSI_OPEN request fails because the device is in use by an agent other than this NDMP Server. It is also returned by NDMP_TAPE_OPEN if the tape is currently in use by the MOVER. (The MOVER is in Active or Listen state.) NDMP_DEVICE_OPENED_ERR A device is already open. NDMP connections are limited to a single tape or SCSI device open at a time. NDMP_NOT_AUTHORIZED_ERR NDMP connection not yet authenticated. Prior to issuing most requests, the NDMP connection MUST first be authenticated through the NDMP_CONNECT_AUTH_CLIENT message. This error is returned if a message requiring connection authentication is received when the connection has not yet been authenticated. NDMP_PERMISSION_ERR The user name used to authenticate the connection does not have access permissions to execute this message. NDMP_DEV_NOT_OPEN_ERR Device not open. An attempt was made to access a device that was not open. NDMP_IO_ERR Device I/O error. This general error SHOULD only be used if none of the more specific device failure error codes apply. A Log Message SHOULD be sent to describe the error in more detail. NDMP_TIMEOUT_ERR Command timeout error. NDMP_ILLEGAL_ARGS_ERR Message received containing one or more invalid arguments. It is RECOMMENDED that a Log Message be sent to describe the unacceptable arguments. Expires October 2003 [Page 35] Draft Specification NDMP Version 4 Protocol April 2003 NDMP_NO_TAPE_LOADED_ERR Tape device could not be opened because no tape was loaded. Alternatively the tape has been unloaded since the open command. (If the Server cannot detect this specific condition an NDMP_IO_ERR SHOULD be reported.) NDMP_WRITE_PROTECT_ERR Tape device could not be opened in write mode because the tape is write protected. Alternatively the tape write protect state has changed since it was opened or the open was a raw mode open. NDMP_EOF_ERR The Tape command failed because end-of-file was encountered. See Tape/Mover Interface for details of usage. NDMP_EOM_ERR The tape command failed because the end of media mark was encountered. See Tape/Mover Interface for details of usage. NDMP_FILE_NOT_FOUND_ERR During a recover operation, a specified file was not found. This error code is used in a Log Interface File Recovered message. NDMP_BAD_FILE_ERR Error due to invalid file descriptor. NDMP_NO_DEVICE_ERR Specified device does not exist. NDMP_NO_BUS_ERR Specified SCSI controller does not exist. NDMP_XDR_DECODE_ERR (Generic Error) Error decoding message. NDMP_ILLEGAL_STATE_ERR Message cannot be processed in the current state. NDMP_UNDEFINED_ERR (Generic Error) This error code SHOULD only be used if no other error code describes the condition. A Log Message SHOULD be sent to describe the condition in detail. NDMP_XDR_ENCODE_ERR (Generic Error) Error encoding reply message. NDMP_NO_MEM_ERR (Generic Error) Memory allocation error. To avoid diagnostic errors, it might be useful to send a Log Message giving more information about the operation that failed. Expires October 2003 [Page 36] Draft Specification NDMP Version 4 Protocol April 2003 NDMP_CONNECT_ERR Data Server - Tape Server failed to establish a data connection. NDMP_SEQUENCE_NUM_ERR Request header received contains an invalid sequence number. NDMP_READ_IN_PROGRESS_ERR The mover read request was received while a previous mover read was in progress. Only one read request may be processed at any one time. NDMP_PRECONDITION_ERR The request was rejected because a required preparatory action has not been performed. For instance, an NDMP_MOVER_LISTEN or NDMP_MOVER_CONNNECT command would be rejected with this error code if the mover record size had not been set. NDMP_CLASS_NOT_SUPPORTED_ERR The list of selected class-version pairs includes one or more classes that the NDMP Server does not support. NDMP_VERSION_NOT_SUPPORTED_ERR The list of selected class-version pairs includes one or more class with unsupported version. NDMP_EXT_DUPL_CLASSES_ERR The list of selected class-version pairs includes two or more instances of one class with different versions. NDMP_EXT_DANDN_ILLEGAL_ERR The D+N requests are illegal at this point because extension requests have already been issued. 2.8.2 Error Codes in NDMP Extensions NDMP extensions need to provide error codes in the context of the extension. This is done by looking at the 32 bit error code as two 16 bit numbers: "class"."class_specific_error_code". Implicitly the error codes for core NDMP is "class_0x0"."core_NDMP_error_code". The definition of error codes for extensions is extension specific, and is specified together with the extension. For proprietary extensions the implementer provides the specification. For standard extensions the error codes are specified in the standard extension specification. 2.9. Message Numbers The following messages are defined: Expires October 2003 [Page 37] Draft Specification NDMP Version 4 Protocol April 2003 enum ndmp_message { NDMP_CONNECT_OPEN = 0x900, NDMP_CONNECT_CLIENT_AUTH = 0x901, NDMP_CONNECT_CLOSE = 0x902, NDMP_CONNECT_SERVER_AUTH = 0x903, NDMP_CONFIG_GET_HOST_INFO = 0x100, NDMP_CONFIG_GET_CONNECTION_TYPE = 0x102, NDMP_CONFIG_GET_AUTH_ATTR = 0x103, NDMP_CONFIG_GET_BUTYPE_INFO = 0x104, NDMP_CONFIG_GET_FS_INFO = 0x105, NDMP_CONFIG_GET_TAPE_INFO = 0x106, NDMP_CONFIG_GET_SCSI_INFO = 0x107, NDMP_CONFIG_GET_SERVER_INFO = 0x108, NDMP_CONFIG_SET_EXT_LIST = 0x109, NDMP_CONFIG_GET_EXT_LIST = 0x10A, NDMP_SCSI_OPEN = 0x200, NDMP_SCSI_CLOSE = 0x201, NDMP_SCSI_GET_STATE = 0x202, NDMP_SCSI_OBSOLETE1 = 0x203, NDMP_SCSI_RESET_DEVICE = 0x204, NDMP_SCSI_OBSOLETE2 = 0x205, NDMP_SCSI_EXECUTE_CDB = 0x206, NDMP_TAPE_OPEN = 0x300, NDMP_TAPE_CLOSE = 0x301, NDMP_TAPE_GET_STATE = 0x302, NDMP_TAPE_MTIO = 0x303, NDMP_TAPE_WRITE = 0x304, NDMP_TAPE_READ = 0x305, NDMP_TAPE_EXECUTE_CDB = 0x307, NDMP_DATA_GET_STATE = 0x400, NDMP_DATA_START_BACKUP = 0x401, NDMP_DATA_START_RECOVER = 0x402, NDMP_DATA_ABORT = 0x403, NDMP_DATA_GET_ENV = 0x404, NDMP_DATA_STOP = 0x407, NDMP_DATA_LISTEN = 0x409, NDMP_DATA_CONNECT = 0x40A, NDMP_DATA_START_RECOVER_FILEHIST = 0x40B, NDMP_NOTIFY_DATA_HALTED = 0x501, NDMP_NOTIFY_CONNECTION_STATUS = 0x502, NDMP_NOTIFY_MOVER_HALTED = 0x503, NDMP_NOTIFY_MOVER_PAUSED = 0x504, NDMP_NOTIFY_DATA_READ = 0x505, NDMP_LOG_FILE = 0x602, NDMP_LOG_MESSAGE = 0x603, Expires October 2003 [Page 38] Draft Specification NDMP Version 4 Protocol April 2003 NDMP_FH_ADD_FILE = 0x703, NDMP_FH_ADD_DIR = 0x704, NDMP_FH_ADD_NODE = 0x705, NDMP_MOVER_GET_STATE = 0xA00, NDMP_MOVER_LISTEN = 0xA01, NDMP_MOVER_CONTINUE = 0xA02, NDMP_MOVER_ABORT = 0xA03, NDMP_MOVER_STOP = 0xA04, NDMP_MOVER_SET_WINDOW = 0xA05, NDMP_MOVER_READ = 0xA06, NDMP_MOVER_CLOSE = 0xA07, NDMP_MOVER_SET_RECORD_SIZE = 0xA08, NDMP_MOVER_CONNECT = 0xA09, NDMP_EXT_STANDARD_BASE = 0x10000, NDMP_EXT_PROPRIETARY_BASE = 0x20000000 }; 2.10. Message Definitions Each message is described using XDR specifications. These form either a request/reply message pair as shown below or a single post message constructed in a similar fashion. struct message_name_request { type request_argument1; ... type request_argumentN; }; struct message_name_reply { ndmp_error error; type reply_argument1; ... type reply_argumentN; }; Expires October 2003 [Page 39] Draft Specification NDMP Version 4 Protocol April 2003 Each XDR specification conforms to the format given in [1] and can be processed using a program such as rpcgen. No XDR specification is provided for the request message if the request message does not contain any arguments. No XDR specification is provided for the reply message if the reply message does not contain any arguments or if no reply message is defined. Following the XDR specification is a description of each argument in the message or messages. Each reply message contains an error code. If an error code is returned that is not equal to NDMP_NO_ERR, some of the reply arguments MAY be meaningless. A list of errors that MAY typically be returned in the reply is provided for each message. This is not an exhaustive list. Generic errors, such as NDMP_NO_MEM_ERR, are not listed. 2.11. Message Sequencing and State Tables 2.11.1. General Rules This section describes the timing of NDMP messages. It describes when messages MUST be sent and when they MUST NOT be sent. The timing of messages is closely related to a number of state variables that describes the state of the NDMP Server. Some state variables are simple booleans. Two examples of Boolean variables are: "Is the client authorized?" or "Is the tape device open?" However, the Data and Mover Interfaces are more complex and are described below in state diagrams. The state machines are conceptually located in the NDMP Server and states changes are made in response to events in the NDMP Server. The DMA is informed of state changes by the receipt of NDMP messages. The DMA SHOULD recognize a state change when it receives a state change notification message from the NDMP Server or when it receives a reply from the NDMP Server accepting a previously issued request. The Data and Mover states may also be monitored using NDMP_DATA_GET_STATE and NDMP_MOVER_GET_STATE requests. If the DMA issues a request that would normally cause the NDMP Server to change state and this request is rejected, no state change is made. In normal conditions it is clear whether it is the responsibility of the DMA or the NDMP Server to send the next operational message. However, in error situations, abort messages may be sent by the party that is not currently in control. These situations can result in messages crossing in transmission causing race conditions. The usual result of this situation is that the request sent by the DMA is rejected by the NDMP Server with an NDMP_ILLEGAL_STATE_ERR error code. The DMA MUST handle this situation in a reasonable fashion. (That is, it SHOULD continue to tidy up the session and it MUST NOT treat the NDMP Server as if it had caused the error.) An example race condition is shown in section 2.11.5.1. 2.11.2. Connection When an NDMP session first starts the DMA and NDMP server MUST ensure that they can communicate successfully. Expires October 2003 [Page 40] Draft Specification NDMP Version 4 Protocol April 2003 The TCP/IP connection is initiated by the DMA, which must know the port on which the NDMP Server is listening. Port 10,000 is reserved for NDMP. NDMP Servers SHOULD typically listen on port 10,000. However, to accommodate conflicts caused by another service already using port 10,000, both DMAs and Servers SHOULD be implemented so that they may be configured to use a different port. The first message sent on the connection MUST be an NDMP_NOTIFY_CONNECTION_STATUS message from the NDMP Server. The server MAY either refuse the connection because of some local difficulty or it MAY suggest that the DMA use a particular version of NDMP. The DMA SHOULD then send an NDMP_CONNECT_OPEN request specifying an NDMP version it wishes to support. If the NDMP Server accepts the NDMP_CONNECT_OPEN request the specified protocol version is used thereafter by the client and server for all successive messages. If the DMA uses the same NDMP version as specified in the original NDMP_NOTIFY_CONNECTION_STATUS message it MAY omit sending the NDMP_CONNECT_OPEN request. Sending any other request implicitly indicates acceptance of the NDMP version specified in the NDMP_NOTIFY_CONNECTION_STATUS message. If the client does not support the protocol version specified in the NDMP_NOTIFY_CONNECTION_STATUS message, the client SHOULD continue to send NDMP_CONNECT_OPEN requests with successively lower version numbers until the server accepts a message. Consider a server that supports versions 2 and 4 and a client that supports versions 2 and 3. The server SHOULD specify version 4 in the NDMP_NOTIFY_CONNECTION_STATUS message. Because the client does not support version 4, the client SHOULD send an NDMP_CONNECT_OPEN request containing a version of 3. Because the server does not support version 3, the server MUST reject the request by returning an NDMP_CONNECT_OPEN reply containing an NDMP_ILLEGAL_ARGS_ERROR error code. The client SHOULD then send an NDMP_CONNECT_OPEN request containing a version of 2. Because the server supports version 2, it SHOULD accept the request by returning an NDMP_CONNECT_OPEN reply containing an NDMP_NO_ERR error code. When the DMA finishes using the connection it SHOULD send an NDMP_CONNECT_CLOSE message prior to closing the TCP connection. The NDMP Server SHOULD not close the connection until requested to do so by the DMA. If forced to close the connection due to a local error or shutdown it SHOULD first send an NDMP_NOTIFY_CONNECTION_STATUS request containing an NDMP_SHUTDOWN reason code. 2.11.3. Authentication The NDMP Server stores user data that MUST be protected from unauthorized access. The DMA MUST be authenticated by using the NDMP_CONNECT_CLIENT_AUTH request before it is allowed to use most of the NDMP requests. Following are the only requests that the DMA MAY use prior to authentication: Expires October 2003 [Page 41] Draft Specification NDMP Version 4 Protocol April 2003 NDMP_CONNECT_OPEN, NDMP_CONNECT_CLOSE, NDMP_CONNECT_CLIENT_AUTH, NDMP_CONFIG_GET_SERVER_INFO, NDMP_CONFIG_GET_AUTH_ATTR. Because the DMA is in control of establishing the TCP/IP connection and does not have any resources to protect there is less need to authenticate the server. There is an NDMP_CONNECT_SERVER_AUTH request that MAY be used. However, this is optional and the NDMP Server MAY choose not to implement it. 2.11.4. SCSI and Tape Devices The SCSI Interface accesses media changer devices. A single media changer device can be associated with the NDMP connection using the NDMP_SCSI_OPEN request. After finishing with the device the association is removed by issuing an NDMP_SCSI_CLOSE request. Other SCSI Interface commands MAY only be issued when the SCSI device is open. The Tape interface is similar, using NDMP_TAPE_OPEN and NDMP_TAPE_CLOSE to associate and disassociate the device with the NDMP connection. An NDMP Server is restricted to having a single device (SCSI or tape) associated with a connection at a time. The NDMP Server MUST return an NDMP_BUSY_ERROR upon receiving an NDMP_TAPE_OPEN or NDMP_SCSI_OPEN request if a device is already open. The Tape interface prepares the tape for use by the MOVER which writes and reads the actual backup data. For the two to work together, a number rules must be followed about when requests can be issued: - The TAPE device MUST be open when the MOVER is activated by one of NDMP_DATA_CONNECT, NDMP_MOVER_CONNECT or NDMP_MOVER_CONTINUE requests. - When the MOVER state is NDMP_MOVER_STATE_ACTIVE, no Tape interface requests can be issued except NDMP_TAPE_GET_STATE. - When the MOVER state is NDMP_MOVER_STATE_PAUSED, the Tape interface can be freely used. This might even involve closing the current tape device and opening the same or another device before issuing a NDMP_MOVER_CONTINUE request. 2.11.5. Data State Diagram In the following diagram the states are shown in boxes of "*"-s. Requests received from the DMA are shown through their message identifier (for example, NDMP_DATA_CONNECT). The state transitions occur as the events take place at the NDMP Server. The DMA is informed of the transitions in the following ways: Expires October 2003 [Page 42] Draft Specification NDMP Version 4 Protocol April 2003 - The transition to NDMP_DATA_STATE_HALTED state is always indicated by an NDMP_NOTIFY_DATA_HALTED message. This is true even if the transition was instigated by an NDMP_DATA_ABORT request. - The transition from NDMP_DATA_STATE_LISTEN state to NDMP_DATA_STATE_CONNECTED state is only indicated indirectly through the NDMP_MOVER_CONNECT reply on the Mover Interface. - All other state transitions are related to direct requests on the Data Interface and are complete when the corresponding reply mes