P2P Tools
Erich Rast <erich ’at snafu.de>
1 The LUMP Messaging Protocol
(require (planet erast/p2ptools:1:=0/lump)) |
LUMP is a simple binary protocol for sending messages with arbitrary number of arguments over reliable stream ports.
1.1 Messages
The main data structure of LUMP are messages. These have an id, a referer field, some internal flags, and an argument list.
struct
(struct message (id seqnum referer flags version? args) #:extra-constructor-name make-message) id : (integer-in 0 65535) seqnum : (integer-in 0 4294967295) referer : (integer-in 0 65535) flags : (integer-in 0 15) version? : (integer-in 1 16) args : (listof/c lump-argument-type?)
Instead of the default message constructor, use new-message or new-response for creating new messages.
1.2 Opening and Closing Sessions
In LUMP messages carry a sequence number that is increased per session. You first need to create a new session before sending any messages.
After all messages associated to one session have been sent, you should use close-session to close it:
1.3 Creating and Writing Messages
procedure
id : (integer-in 0 65535) args : lump-argument-type? = none/c
Example: | ||
|
procedure
id : (integer-in 1 65535) referer-message : message? args : lump-argument-type? = none/c
procedure
(write-message session message [out-stream]) → number?
session : session? message : message? out-stream : port? = (current-output-port)
1.4 Receiving Messages
procedure
in-stream : port? = (current-input-port)
version-check-proc : (number? .->. any/c) =
(lambda (version) (when (> version protocol-version) (raise-argument-error 'read-message (format "LUMP protocol version ~a or lower" protocol-version) version)))
value
protocol-version : (integer-in 1 16)
1.5 Supported Data Types
LUMP supports numbers, strings, bytes, lists, symbols, and vectors as external argument types that are automatically serialized. In addition to these Racket data types, LUMP also supports the following low-level data types:
Type | Internal | Bytesize | Explanation |
bool | 0 | 1 | Boolean value as a byte |
int8 | 2 | 1 | Unsigned Byte |
int16 | 3 | 2 | Signed Short |
uint16 | 4 | 2 | Unsigned Short |
int32 | 5 | 4 | Signed 32-bit Integer |
uint32 | 6 | 4 | Unsigned 32-bit Integer |
int64 | 7 | 8 | Signed 64-bit Integer |
uint64 | 8 | 8 | Unsigned 64-bit Integer |
text | 9 | n/a | UTF-8 String (varying length) |
symbol | 10 | n/a | Racket symbol (varying length) |
list | 11 | n/a | List (varying length) |
number | 12 | n/a | Number (varying length) |
bytes | 13 | n/a | Bytes (varying length) |
vector | 14 | n/a | Vector (varying length) |
(The values in the Internal column are internal identifiers that are only needed if you want to implement the protocol in another language.)
To use an internal, low-level type you need to prefixed the identifier with type:, so for example type:uint32 is the type for an unsigned 32-bit integer. Use the following procedures to wrap Racket data into a typed structure that can be provided to new-message or new-response.
Example: | ||
|
Example: | ||
|
Examples: | ||||||||
|
Examples: | ||||||||
|
Examples: | ||||||||
|
1.6 Description of the LUMP Protocol
You do not need to know the internals of the protocol in order to use it, but the following information is useful if you would like to implement a receiver or sender in another programming language. All numeric values represented by multiple bytes are in little endian format. The structure of a serialized LUMP message is as follows:
- Header (7 bytes):
4 bits (MSBs of the byte): Protocol version - a value from 0 to 15 representing the internal protocol version number - 1
- 4 bits (LSBs of the byte): Flags - 4 bits of flags, currently 2 of them are used:
Bit 0: the message has a data portion
Bit 1: the message has a referer field
2 bytes: Message Id
4 bytes: Sequence number of the message
Referer (4 bytes, only present if referer flag is set): Sequence number of the referer message
- Data Portion (varying size, only present if data flag is set):
2 bytes: Number argnum of following arguments
- For each argument 1...argnum:
1 byte: LUMP internal argument type of the argument
4 bytes: length arglen of the following data in bytes
1....arglen bytes: data of the argument
2 The File Transfer Library
(require (planet erast/p2ptools:1:=0/filetransfer)) |
This library allows you to transfer files over TCP connections. To transfer a file, the receiver must first open a listener and then the sender must send the file. Sending and receiving is asynchronous, using procedures as event callbacks.
2.1 Receiving Files
procedure
(start-listen local-port save-path from-ip progress-proc final-proc [ timeout file-table listen-timeout]) → filetransfer? local-port : (and/c exact-nonnegative-integer? (integer-in 0 65535)) save-path : path? from-ip : (or/c string? boolean?)
progress-proc :
([phase (one-of/c 'listening 'preparing 'receiving)] [path path?] [progress number?] .->. any/c)
final-proc :
([error-code (one-of/c 'finished 'error)] [path path?] [total-milliseconds number?] [total-bytes-read number] .->. any/c) timeout : real? = 60.0 file-table : (and/c hash? hashtable-mutable?) = (make-hash) listen-timeout : real? = 604800.0
2.2 Sending Files
procedure
(send-file remote-hostname remote-port file-path suggested-filename progress-proc final-proc [ timeout]) → filetransfer? remote-hostname : string? remote-port : (and/c exact-nonnegative-integer? (integer-in 0 65535)) file-path : path? suggested-filename : string?
progress-proc :
([phase (one-of/c 'connecting 'preparing 'sending)] [path path?] [progress number?] .->. any/c)
final-proc :
([error-code (one-of/c 'finished 'error)] [path path?] [total-milliseconds number?] [total-bytes-read number] .->. any/c) timeout : real? = 60.0
2.3 Interrupting Transfers
procedure
(kill-transfer transfer) → any/c
transfer : filetransfer?
procedure
(finish-transfer transfer [timeout]) → any/c
transfer : filetransfer? timeout : real? = 259200.0
procedure
(wait-transfer transfer [timeout]) → any/c
transfer : filetransfer? timeout : real? = 259200.0
2.4 Resuming Transfers
File transfers are resumed automatically as long as the same file-hash table and suggested names are used for both transfers. When a file is sent by send-file the sender produces a hash value retrieved from the mid of the file and a suggested file name. These are stored by the receiver in the file-table hash. If the same file paths and file-table are used for a file transfer on the receiving side, and the sender uses the same suggested file name both times, a file transfer that has previously been interrupted and succeeded only partially will automatically be resumed.
2.5 Description of the Transfer Protocol
You do not need to know the internals of the protocol in order to use it, but the following information is useful if you would like to implement a receiver or sender in another programming language. All numeric values represented by multiple bytes are in little endian format. The protocol involves a single handshake reply from the receiver to indicate the position from which the file transfer is to be resumed. A complete transmission of a file works as follows:
The sender sends 20 bytes of checksum data based on an sha1 hash of 16384 bytes from the middle of the file or less if the file is shorter.
The sender sends a 2 bytes value namelen representing the length of the suggested file name, followed by namelen bytes of name data in UTF-8 format.
The receiver then replies with an 8 bytes value offset into the file that indicates the position from which to continue the transfer (i.e. offset will be 0 if no portion of the file has been received yet).
The sender then sends an 8 bytes value datalen, followed by datalen bytes of file content. The content sent starts at position offset of the file and the total size of the file should be datalen+offset.
3 Display Measures
(require (planet erast/p2ptools:1:=0/display-measures)) |
This library contains utility functions for displaying transfer speeds. The data rate functions using a power of 10 measure for the amount of data are recommended.
procedure
(bits/sec->data-rate-string bit sec [ decimals]) → string? bit : number? sec : number? decimals : positive? = 1
procedure
(bytes/sec->data-rate-string byte sec [ decimals]) → string? byte : number? sec : number? decimals : positive? = 1
procedure
(bytes/msec->data-rate-string byte msec [ decimals]) → string? byte : number? msec : number? decimals : positive? = 1
Produce a string that expresses the data rate measured in bits per second, bytes per second, and bytes per millisecond respectively up to the given decimals precision, using power of 10 measures for the amount of data transferred.
Examples: | ||||||||
|
procedure
(bytes/sec->binary-rate-string byte sec [ decimals]) → string? byte : number? sec : number? decimals : positive? = 1
procedure
(bytes/sec->binary-rate-string* byte sec [ decimals]) → string? byte : number? sec : number? decimals : positive? = 1
procedure
(bytes/msec->binary-rate-string byte sec [ decimals]) → string? byte : number? sec : number? decimals : positive? = 1
procedure
(bytes/msec->binary-rate-string* byte sec [ decimals]) → string? byte : number? sec : number? decimals : positive? = 1
Produce a rate string based on power of 2 measures of data with the appropriate scale such as e.g. "8.0 KiB/s" up to the given decimals precision, where the first two functions use seconds and the last two functions use milliseconds as the basis, and the starred versions produce the slightly incorrect but more common measures "KB/s", "MB/s", etc.
Examples: | ||||||||
|