KNMP Overview

This white paper was never completed. However we feel that the content is useful.

Version: March 1, 2010

Introduction

KNMP is a new network management protocol.

KNMP is intended to replace its predecessor, SNMP, with a protocol that is more efficient and easier to implement.

Rather than being a break with the past, KNMP builds on the large existing investment in MIB definitions.

KNMP is a RESTful client-server architecture in which a standard connection transport is used to convey requests and responses. Security is provided by the transport. A typical transport would be SSH.

In KNMP, as in SNMP, Netconf, and CMIP the managed-device is the server and the manager is the client.

KNMP uses a client-server interaction approach similar to that used between web browsers and web servers. The target of KNMP operations are identified by a textual URI (URL.) All KNMP data is textually encoded.

KNMP defines a set of operations – such GET, UPDATE, CREATE, DELETE, NOTIFY – that are applied to the target defined by the KNMP URI. A typical KNMP URI may be as simple as:

 knmp://host.example.tld/interfaces

KNMP data is encoded using JSON, a text-based encoding that is widely used in the world wide web and is supported by nearly every programming language.

KNMP supports data queries as well as operations to change, create, and delete portions of the target system's management data.

KNMP conceives of the management data in a way similar to SNMP: Management data is organized into a tree with every element identified using the unified, IANA operated, object identifier hierarchy used for SNMP MIBs. Data is organized into groups, tables and rows in the same way as in SNMP MIBs, but with a significant simplification.

Management actions are mechanized through the update of action variables within the management data tree.

There are some differences between classic SNMP MIBs and the hierarchy used by KNMP. To reduce confusion, KNMP uses the term “MDT”, or Management Data Tree, in lieu of “MIB” or “Management Information Base”.

The general structure of KNMP transactions is to identify one or more sub-trees within the MDT and then to apply an operation to those sub-trees. Thus, using Internet Standard MIB-2 as an example of an MDT, a KNMP GET transaction would select rows from the Interfaces Table as a set of sub-trees, obtain the contents of those sub-trees, and return those contents as a unit to the client.

The KNMP SET operation works much like the GET operation in that a set of target sub-trees is identified. However, in SET a collection of update information is provided in the URI and those updates are applied to each sub-tree before the contents are obtained and returned to the client.

KNMP allows SET, CREATE, and DELETE operations to be coerced so that they apply to internal MDT nodes (typically tables or rows within tables) so that the these transactions may be performed atomically and with full consistency testing. The complex “row-status” dances of SNMP are not necessary in KNMP.

Implementation experience with KNMP and SNMP indicates that a KNMP client and server can be constructed much more easily and with a much smaller memory and processing footprint than SNMP. KNMP network performance is much greater than that of SNMP. As a consequence management transactions occur more quickly with KNMP than SNMP.

Background

In the mid 1980's the young internet community looked at various alternative network management approaches. The three primary candidates were an internet version of ISO/OSI CMIP, a research project called HEMS, and the already operational system of SGMP (which was evolving into SNMP.)

For reasons political and technical, the choice was made to adopt both CMIP and SNMP. CMIP, as was typical of ISO/OSI protocols was difficult to implement and thus, as a practical matter, the field was left to SNMP.

SNMP was premised on the notion that TCP would be unusable when network conditions were poor and thus when the need for network management would be highest. Consequently, SNMP was based on UDP and all transaction data had to be contained in each separate UDP packet. The premise that TCP would be unusable has not been sustained in practice. Rather, it has been learned that when the net is so bad that a TCP connection can not be maintained it is also unlikely that a sequence of UDP packets can be maintained. When the net becomes so ill that TCP fails then it is time for diagnostic and troubleshooting tools rather than network management tools.

The use of UDP requires SNMP to place all transaction information in each and every packet. That significantly reduces the data capacity and adds a large per-packet processing overhead. In addition, the lack of a reliable, sequenced transport forced the designers of SNMP to find a way of dealing with arbitrarily large amounts of management data. SNMP meets this through a Get-Next operator that demands that management data be maintained in a lexicographic order. Lexicographic ordering and Get-Next require server devices to maintain management data in an unnaturally sequenced form that often requires a significant amount of complex, error-prone code that can add a significant computational burden to managed devices.

SNMP's UDP foundation leads to a situation in which most SNMP interactions require a significant number of packet exchanges, a lot of time spent waiting for those packets to traverse the network, and a fair amount of server processing overhead.

SNMP adopted the ISO/OSI encoding mechanism, ASN.1/BER. That encoding was designed in the early 1980's to be very economical in terms of bits in packets but it came at a large cost of CPU cycles. RFC 1052 - IAB Recommendations for the Development of Internet Network Management Standards - http://tools.ietf.org/html/rfc1052

SNMP also adopted the ISO/OSI object identifier naming system. Because of the way that SNMP accesses data items and performs its “Get-Next” operation, SNMP added to the ISO/OSI object identifier system a requirement that the binary object identifiers be decoded from ASN.1/BER and sequenced in lexicographic order. These decoding and sequencing tasks can consume many CPU cycles and often involve complex and error-prone code.

KNMP retains the object identifier hierarchy used by SNMP but uses them in a relaxed way that eliminates the high cost of decoding and sequencing them.

SNMP went through several generations of increasing complexity to reduce the inefficiency of the original Get-Next mechanism and to add security. Because of the UDP datagram constraint these enhancements became intricate and difficult to implement or use.

(There are ongoing efforts to re-host SNMP onto a connection-oriented transport, such as SSH. That would give SNMP the means to push many security issues onto that transport in the same way as KNMP and Netconf. However, a SNMP- over-SSH would continue to be burdened with its obligation to coerce management data into an object-identifier based lexicographic ordering and continue to bear the processing overhead that comes from that ordering.)

KNMP has built on the lessons of SNMP.

KNMP uses connections rather than independent UDP packets. KNMP uses the IANA rooted naming hierarchy for object identifiers but does not use the ASN.1/BER encoding. Nor does KNMP use lexicographic ordering or Get-Next transactions to step though management data. KNMP uses the efficient and widely deployed JSON encoding standard rather than ASN.1/BER.

KNMP, being connection based, can obtain the entire aggregate MDT of a device, or any subtree of that MDT, in a single transaction with the delay being little more than a single packet round-trip time. KNMP has learned lessons from other protocols as well.

KNMP, like Netconf, leaves the burden of security to the enclosing transport mechanism. Like Netconf, KNMP will most often be operated over SSH or SSL/TLS. However, for development and testing KNMP can be run, without security, over a raw TCP connection.

KNMP status

KNMP is operational today – the design of KNMP has gone hand-in-hand with actual interoperating client and server implementations.

The implementations confirm that KNMP is efficient: It imposes only a small processing and memory footprint on client and server devices. And it uses only a small amount of network bandwidth and transaction processing requires a minimum of network round-trip-times.

KNMP features

Connection based session with client/server identification, authentication, and privacy provided by the connection mechanism (SSH, SSL/TLS, etc).
Lightweight code footprint and low processing overhead.
Can be used by an AJAX web application.
Minimal number of packet round-trip times, i.e. quick response.
Uses data in a form very similar to SNMP MIBs thus preserving existing instrumentation and allowing re-use of management station architectures. However, KNMP neither needs nor uses lexicographic ordering. Because KNMP is based on a connection-oriented transport KNMP has no need to chop data into small UDP datagram sized chunks: KNMP can transfer the entire body of management data, or any sub-tree of that data, in a single, easily parsed unit.
Transactions use a URL-like format to express the target data; the data itself is encoded using industry standard JSON, a format supported by nearly every computer language and platform including most web browsers.
Transactions may select one or more sub trees from the MDT (Management Data Tree). Thus, for example, a single query transaction could retrieve each interface of type “ethernet”; a single update transaction could disable all interfaces of type “802.11”.
Expressions may be embedded into the URL so that useful selections may be made. For example, a single query transaction could retrieve each interface in which the error rate exceeds 1% of the total packets. A single update transaction could shut down all interfaces in which the error rate exceeds 5% of the total packets.
Transactions that create, update, or delete MDT data can be atomic without the need for complex “row-status” protocol exchanges.
Transactions that create, update, or delete MDT data can be fully evaluated for consistency before they are performed.
MDT data may grow and contract in accord with events such as the insertion or removal of an interface line card.
KNMP may be operated via a proxy.
Can operate across Network Address Translation (NAT) devices.
Etc **TODO**

KNMP details

Most KNMP transactions consist of a request message from a client (the manager) and a subsequent reply message from the agent (the managed device.) These occur on the same transport connection.

The one transaction that operates differently is the event NOTIFY, which can be conceived as a kind of reverse operation in which what is normally the server initiates the transaction.

Requests and replies are associated through a transaction ID field that is established by the sender of the request and echoed back in the reply.

Request messages and reply messages for different transactions may be interleaved on the transport. In other words, a client is free to issue several requests without waiting for replies. And the server may answer them in whatever order it chooses. However, a well behaved server will not dally too long and keep the client waiting.

Messages

KNMP messages are all ASCII text except that string data encoded in JSON is in Unicode.

In order to facilitate parsing by the receiver, every KNMP message is terminated with a carriage-return, line-feed (CRLF, 0x0D 0x0A) pair. CR and LF should not be used within a KNMP message except within JSON encode strings (in which they are escaped by the JSON encoding process into different bit patterns.)

Each message conforms to the following overall structure:

Version (string)
Verb (string)
Request/Reply Indicator (string)
Transaction ID (number)
Content (string)

Verbs are one of “GET”, “CREATE”, “UPDATE”, “DELETE”, “LOCK”, “UNLOCK”, “ERROR”, and “INFORM”.

The Request/Reply indicator is either “REQ” (for requests) or “REP” (for replies.)

The transaction ID is a number created by the sender of a request message. The transaction ID is echoed in the matching reply message. The content varies according to the particular message verb and request/reply indicator. A typical JSON encoded message inside a TCP connection would appear like.

["0.1","GET","REQ",12345,[”knmp://host.example.tld/interfaces”]]CRLF

The KNMP Management Data Tree (MDT)

The KNMP Management Tree is much like the collection of SNMP MIBs supported by an SNMP agent.

The MDT is, as its name indicates, a tree. The tree has a single root, internal nodes, and leaves. Leaves hold the data and have no subordinate nodes.

The root and internal nodes may have zero or more subordinated (child) nodes.

Following the SNMP MIB structure, certain internal nodes are designated as tables that contain zero or more subordinate nodes known as rows. In SNMP MIBs rows may contain only data-bearing leaf subordinate nodes. KNMP relaxes that constraint and allows rows to contain both data-bearing leaf and table subordinate nodes.

Management data types

KNMP MDTs support leaf nodes of the following data types:

ASCII string
Unicode string (to be done)
8-bit signed integer
8-bit unsigned integer
16-bit signed integer
16-bit unsigned integer
32-bit signed integer
32-bit unsigned integer
64-bit signed integer
64-bit unsigned integer
Single precision floating point number
Double precision floating point number
48-bit MAC address
64-bit MAC address
IPv4 Address
IPv6 Address

KNMP schema

The MDT is shaped in conformity with a schema. That schema defines the relationship of various nodes, the names of nodes, data type information, access criteria, whether particular nodes are simple inner nodes, tables, rows, or leaves, etc.

The MDT contains the data while the schema contains information about the MDT and its data. The schema is also a tree structure. But unlike the MDT, in which rows and the children of rows may occur multiple times, the schema information about a row and its children occurs only once.

For each internal node of the MDT the schema defines the following:

The parent node, if any, of the node
Whether the node defines a table row
DNId (see below)
Alias name
Whether UPDATE may be performed against the node
Whether CREATE may be performed against the node
Whether DELETE may be performed against the node
Description (in text form)

For each leaf node of the MDT the schema defines the following:

The parent node of the node
Whether the node forms part of a key set for a row of which the node is an element
The management data type (e.g. 16-bit unsigned integer, IPv4 Address, etc)
DNId (see below)
Alias name
Whether UPDATE may be performed against the node
Whether CREATE may be performed against the node
Whether DELETE may be performed against the node
Default value to be used in a CREATE operation if a value is not otherwise provided.
Description (in text form)

The canonical form for the schema is JSON. However, an agent implementation may, for convenience, support alternative forms such as a .ini format file or an XML file.

The YANG format used for Netconf might also provide a syntactical structure to represent a KNMP schema.

The server's schema is available as a standard part of the MDT (and schema) supported by all KNMP-compliant agents.

A client may obtain the server's schema through a simple GET transaction with a URI that looks like this:

  knmp://hostname/schema/

The schema is returned as a text string that contains a JSON encoded representation of the schema.

Node names (DNId)

Each node in the MDT has a name. For all nodes except row nodes and nodes to any degree subordinate to a row node the name is unique. Row nodes and their subordinates also have names, but those names are not necessarily unique and are given context through mechanisms to be described later in this note.

This name is a character string that is in the form of a sequence of labels separated by dots. By convention the labels are numeric and the name looks like a textual object identifier, e.g. “1.3.6.1.2.1.2.2.1.18”.

These names are sometimes called “DNId”s.

Because these names are difficult for humans, KNMP supports alias names. The alias for “1.3.6.1.2.1.2.2.1.18” is “ifOutNUcastPkts” - a name taken directly from an SNMP MIB definition.

KNMP also recognizes that SNMP object identifiers frequently begin with the same sequence. Consequently, KNMP adopts a shorthand form in which commonly repeated prefixes are replaced by a two character pair starting with the '@' character. Thus “ 1.3.6.1.2.1.2.2.1.18” becomes “@B2.2.1.18”.

The shorthand form may be used in lieu of the full name string in KNMP messages, in KNMP configuration data, and in any interaction with users.

The alias form may also be used in KNMP query messages, in configuration data, and user interaction. These aliases are defined by the schema.

(The reason that the alias form is not permitted in KNMP reply messages is to allow the existence of very simple KNMP managers that do not have the sophistication to fetch the schema and do the alias- to-name mapping indicated in the schema.)

**TODO**

Names are constructed according to a convention in which a node that is a direct subordinate takes the name of its parent node and adds one chunk. Thus if the parent is “1.2.3” the child will be something like “1.2.3.4”.

The KNMP URI

KNMP uses URI's to identify the managed device (the server) and the portions of the Management Data Tree within that device (MDT) upon which an operation is to act.

The KNMP URI contains enough information to allow KNMP to be operated via intermediary proxies. A KNMP URI has the following basic structure:

  knmp:<hostpart><mdt-part>

The <hostpart> designates the managed device. The <hostpart> is used in the same way as it is in a normal web, HTTP URL, it is two slash characters followed by a host domain name or IP address.

The <mdt-part> begins with a slash character and has the following structure:

  /<target>/<row-selector>/<row-selector>/ ...

There must be a <target>, the <row-selector>s are optional. The entire URI may optionally end with a slash character.

URI hostpart

The hostpart of the KNMP URI is used the same way that it is in a web, HTTP URL, it is simply the domain name or IP address of the target host.

URI target

The target part of the KNMP URI names a point in the Management Data Tree (MDT). **TODO**

URI row-selectors

The row-selector parts of a KNMP URI are optional and there may be more than one.

Each row-selector is separated from the preceding hostpart or row-selector by a single slash character.

Each row selector is a tiny program, a sequence of steps, that are used by a KNMP server to find the proper target (or targets) for an operation.

**TODO**

Queries (Verb “GET”)

Updates to Existing MDT Data (Verb “UPDATE”)

Row Creation (Verb “CREATE”)

Row Deletion (Verb “DELETE”)

Actions

Error Response

The Error Response message is unique in that there is no Error Request message. Rather, Error Response messages are sent from the server back to a client when one of the other imperative messages (GET, UPDATE, CREATE, DELETE, LOCK, UNLOCK) encounters an error condition.

Error Response messages have a slightly more generalized form than do the other response messages. This is to allow servers to send extra information back to the clients in a way that will not confuse clients that don't comprehend that extra information.

The information that comes back in all Error Responses are these:

A numeric error code
A short ASCII textual title that provides a quick classification of the error (e.g. “Message format error”)
An ASCII textual short description of the error that extends the information provided by the title. Optional information might include:
A textual stack traceback showing where in the server's code that the error occurred.
A boolean indicating whether the error should be considered transient such that it may not occur when processing a subsequent, but otherwise identical, request.

Examples

Obtaining the schema

  GET knmp://host.example.tld/schema/

The schema will be returned in JSON format.

Obtaining all interface names for interfaces with name beginning with “eth”

  knmp://hostname/ifDescr/=ifDescr,!VAL,^eth.*$,!RE/

A JSON encoded list will be returned with the names of all the interfaces that match the given regular expression. The returned list will contain context information identifying how each returned interface row can be directly referenced on subsequent KNMP actions.

Obtaining all interface rows for interfaces with name beginning with “eth”

This differs from the previous example in that it requests the entire row for the interfaces not merely the name.

  knmp://hostname/ifEntry/=ifDescr,!VAL,^eth.*$,!RE/

A JSON encoded list will be returned in which each item in the list represents an ifEntry row and is itself a nested list containing the items of that row. Each nested list will contain context information allowing direct access to the row on subsequent KNMP actions.

Obtaining all interface rows for all interfaces

This example obtains all interface rows without regard to their names. There are two ways that this may be done.

First the set of ifEntry rows may be obtained:

  knmp://hostname/ifEntry/

As in the prior example a JSON encoded list will be returned in which each item in the list represents an ifEntry row and is itself a nested list containing context information and the items of that row.

The second way is to obtain the enclosing ifTable in its entirety:

  knmp://hostname/ifTable/

This will return a JSON representation of the ifTable and all of its rows. No context information is returned because the ifTable is not a row in a table.

Setting the “doneness” value in the toaster MDT

The Toaster MDT defines a variable that controls how long to toast the bread. The following UPDATE transaction will set that variable to 99 and also set another variable called “toasterControl2” to a value of 2:

  knmp://hostname/toaster//toasterControl=2,toasterDoneness=99/

This will return a JSON representation of the toaster group after the changes are applied.

Differences between SNMP MIBs and KNMP Schema and MDT

KNMP largely preserves the structure and form of SNMP MIBs.

The data hierarchy, object identifier (OID )names, textual names (e.g. “sysUpTime”), data types, groups (e.g. TCP group), tables, and row structure are preserved.

In KNMP 64-bit integers (signed and unsigned), IPv4, IPv6, and MAC addresses are native data types.

However, because KNMP does not require lexicographic ordering the SNMP instance appendage to object identifiers dropped.

And in KNMP OIDs are simply structured text strings, so there is no translation of OIDs to ASN.1/BER or the decoding from ASN.1/BER form. KNMP does not use ASN.1 or ASN.1/BER at all.

KNMP allows table rows to contain tables (actually KNMP is even more liberal in that it allows tables to contain rows that contain sub-trees). This eliminates the need for SNMP object-identifier pointers or conceptual extensions of row contents.

KNMP, because, it does updates as atomic operations, removes the need for row-status variables and the choreography of using them.

**TODO** address test-and-set and LOCK**

More on actions

Why JSON rather than XML?

There are many who advocate XML for everything. And XML would have worked underneath KNMP. So why did KNMP adopt JSON?

XML is a general purpose markup language; JSON is focused on the single task of facilitating data interchange over a network between applications. JSON is more efficient on the wire and JSON can be generated and parsed by quite small bodies of code. KNMP does not need XML tools like XPATH. And because KNMP uses data named according to an existing hierarchy maintained by the Internet Assigned Numbers Authority (IANA), KNMP does not need anything like XML namespaces.

If one really wants and needs XML it would be quite easy to construct a KNMP client that produced output in XML format from the JSON that is actually used by the protocol.

Implementation status

Working implementations of most of the client and server (also called the “agent”) side of KNMP are operational. The server is hooked to real operational data in the operating system.

These implementations are in Python. This implementation is intended as a vehicle for protocol design and lacks some features, such as multi-threading, that one would want in a production implementation. The Python implementation for both client and server sides aggregates to approximately 7,000 lines of Python code, including comments, access to operating system data, and debugging scaffolding.

This code uses the open-source Python Twisted framework for asynchronous operation. Only the core portion of that framework is used.

The system and Python library support is relatively small – principally the normal os and sys modules plus the datetime, hashlib, json, re, socket, and time, modules.

LEARN MORE