JHUAPL/kvspool
1
0
Fork 0
mirror of https://github.com/JHUAPL/kvspool.git synced 2026-01-08 15:14:02 -05:00
Code Issues Packages Projects Releases Wiki Activity

doc bpub

Browse Source
This commit is contained in:
Troy D. Hanson
2014-04-12 10:38:14 -04:00
parent 7fbd53c1fd
commit 658d2493e9
3 changed files with 248 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
doc/Makefile
View File

@@ -10,6 +10,14 @@ pub-sub.png: pub-sub.txt
kvspool.html: kvspool.txt
asciidoc -a toc2 $<
TMP=/tmp/kvspool-gh-pages
stage:
mkdir -p ${TMP}
rm -if ${TMP}/*
cp LICENSE.txt ${TMP}
cp *.html ${TMP}
cp *.png ${TMP}
.PHONY: clean
clean:

206
doc/kvspool.html
View File

@@ -3,7 +3,7 @@
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.6" />
<meta name="generator" content="AsciiDoc 8.6.7" />
<title>kvspool: a tool for data streams</title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */
@@ -349,7 +349,7 @@ div.colist td img {
margin-bottom: 0.1em;
}
div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
div.toclevel0, div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
margin-top: 0;
margin-bottom: 0;
}
@@ -407,6 +407,8 @@ span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }
div.unbreakable { page-break-inside: avoid; }
/*
* xhtml11 specific
@@ -414,7 +416,7 @@ span.line-through { text-decoration: line-through; }
* */
tt {
font-family: monospace;
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
@@ -453,7 +455,7 @@ div.tableblock > table[frame="vsides"] {
* */
.monospaced {
font-family: monospace;
font-family: "Courier New", Courier, monospace;
font-size: inherit;
color: navy;
}
@@ -615,7 +617,7 @@ toc: function (toclevels) {
function tocEntries(el, toclevels) {
var result = new Array;
var re = new RegExp('[hH]([2-'+(toclevels+1)+'])');
var re = new RegExp('[hH]([1-'+(toclevels+1)+'])');
// Function that scans the DOM tree for header elements (the DOM2
// nodeIterator API would be a better technique but not supported by all
// browsers).
@@ -644,7 +646,7 @@ toc: function (toclevels) {
var i;
for (i = 0; i < toc.childNodes.length; i++) {
var entry = toc.childNodes[i];
if (entry.nodeName == 'div'
if (entry.nodeName.toLowerCase() == 'div'
&& entry.getAttribute("class")
&& entry.getAttribute("class").match(/^toclevel/))
tocEntriesToRemove.push(entry);
@@ -690,7 +692,7 @@ footnotes: function () {
var entriesToRemove = [];
for (i = 0; i < noteholder.childNodes.length; i++) {
var entry = noteholder.childNodes[i];
if (entry.nodeName == 'div' && entry.getAttribute("class") == "footnote")
if (entry.nodeName.toLowerCase() == 'div' && entry.getAttribute("class") == "footnote")
entriesToRemove.push(entry);
}
for (i = 0; i < entriesToRemove.length; i++) {
@@ -772,8 +774,8 @@ asciidoc.install(2);
<h1>kvspool: a tool for data streams</h1>
<span id="author">Troy D. Hanson</span><br />
<span id="email"><tt>&lt;<a href="mailto:tdh@tkhanson.net">tdh@tkhanson.net</a>&gt;</tt></span><br />
<span id="revnumber">version 0.8,</span>
<span id="revdate">September 2012</span>
<span id="revnumber">version 0.9,</span>
<span id="revdate">April 2014</span>
<div id="toc">
<div id="toctitle">Table of Contents</div>
<noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
@@ -782,6 +784,10 @@ asciidoc.install(2);
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div style="float:right">
<a href="https://twitter.com/share" class="twitter-share-button" data-via="troydhanson">Tweet</a>
<script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>
</div>
<div class="dlist"><dl>
<dt class="hdlist1">
kv-spool ("key-value" spool)
@@ -812,7 +818,7 @@ when it&#8217;s caught up, waiting for more data. Like this,</p></div>
<div class="paragraph"><div class="title">Sneak peak</div><p>Here&#8217;s a sneak peak at a really simple writer and reader:</p></div>
<div class="listingblock">
<div class="title">Sample Python writer</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -822,7 +828,7 @@ d <span style="color: #990000">=</span> <span style="color: #990000">{</span><sp
kv<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">write</span></span><span style="color: #990000">(</span>d<span style="color: #990000">)</span></tt></pre></div></div>
<div class="listingblock">
<div class="title">Sample Perl reader</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -843,9 +849,9 @@ don&#8217;t have to run at the same time. They can come and go. Also, if the re
restarts, it picks up where it left off.</p></div>
<div class="sect2">
<h3 id="_space_management">Space management</h3>
<div class="paragraph"><p>In a streaming data environment, the writer and reader can run for months on end. Because
the spool backs the data to disk, space management is necessary: <em>don&#8217;t fill up the disk</em>.
When we make a spool directory, we tell kvspool what its maximum size should be:</p></div>
<div class="paragraph"><p>In a streaming data environment, the writer and reader can run for months on end. A key
implementation requirement is that we don&#8217;t fill up the disk. So, when we make a spool
directory, we tell kvspool what its maximum size should be:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>% mkdir spool
@@ -908,12 +914,48 @@ that each reader gets it&#8217;s own spool:</p></div>
<div class="paragraph"><p>This type of publish-subscribe does a "fan-out". Each subscriber gets a copy of the data.
It drops data when no subscriber is connected- it&#8217;s a blast to whoever is listening.</p></div>
<div class="sect3">
<h4 id="_s_mode">-s mode</h4>
<div class="paragraph"><p>If you use the <tt>-s</tt> switch, on both ends (kvsp-pub and kvsp-sub), two things change:
data remains queued in the spool until a subscriber connects (instead of being dropped if
no one is listening). Secondly, if more than one subscriber connects, the data gets
divided among them rather than sent to all of them. Generally the -s mode is preferred
if it fits your use case.</p></div>
<h4 id="_pub_sub_modes">Pub-sub modes</h4>
<div class="paragraph"><p>Kvspool includes three kinds of pub-sub utilities. They vary in the underlying transport
and data format.</p></div>
<div class="tableblock">
<table rules="none"
width="50%"
frame="border"
cellspacing="0" cellpadding="4">
<col width="12%" />
<col width="12%" />
<col width="62%" />
<col width="12%" />
<thead>
<tr>
<th align="left" valign="top"> publisher </th>
<th align="left" valign="top"> subscriber </th>
<th align="left" valign="top"> data format and transport </th>
<th align="left" valign="top"> notes</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-pub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-sub</tt></p></td>
<td align="left" valign="top"><p class="table">JSON over zeromq pub/sub</p></td>
<td align="left" valign="top"><p class="table">-s uses push/pull</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-bpub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-bsub</tt></p></td>
<td align="left" valign="top"><p class="table">binary over zeromq pub/sub</p></td>
<td align="left" valign="top"><p class="table">-s uses push/pull</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-tpub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>n/a</tt></p></td>
<td align="left" valign="top"><p class="table">binary over plain TCP</p></td>
<td align="left" valign="top"><p class="table"></p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_concentration">Concentration</h4>
@@ -957,9 +999,9 @@ them when the system reboots.</td>
</div>
<div class="sect2">
<h3 id="_resources_amp_help">Resources &amp; Help</h3>
<div class="paragraph"><p>News about software updates are posted to the author&#8217;s blog: <a href="http://tkhanson.net/blog">http://tkhanson.net/blog</a>.
Contact the author directly at <a href="mailto:tdh@tkhanson.net">tdh@tkhanson.net</a> if you have questions or other issues.
Additional software by the author is cataloged at <a href="http://tkhanson.net">http://tkhanson.net</a>.</p></div>
<div class="paragraph"><p>Contact the author directly at <a href="mailto:tdh@tkhanson.net">tdh@tkhanson.net</a> if you have questions or other issues.
News about software updates are posted to the author&#8217;s blog: <a href="http://tkhanson.net/blog">http://tkhanson.net/blog</a>.
Additional software by the author is cataloged at <a href="http://troydhanson.github.io/">http://troydhanson.github.io/</a>.</p></div>
</div>
</div>
</div>
@@ -1075,19 +1117,105 @@ cellspacing="0" cellpadding="4">
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-sub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-sub -d spool tcp://192.168.1.9:1110 tcp://192.168.1.10:1111</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-sub -d spool tcp://192.168.1.9:1110</tt></p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-bpub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-bpub -b cast.cfg -d spool tcp://192.168.1.9:2110</tt></p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-bsub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-bsub -b cast.cfg -d spool tcp://192.168.1.9:2110</tt></p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>kvsp-tpub</tt></p></td>
<td align="left" valign="top"><p class="table"><tt>kvsp-tpub -b cast.cfg -d spool -p 2110</tt></p></td>
</tr>
</tbody>
</table>
</div>
<div class="paragraph"><p>The network utilities keep a local spool continuously replicated to a remote spool.
These require ZeroMQ and Jansson libraries on the system in order to be built.</p></div>
<div class="paragraph"><p>The network utilities keep a local spool continuously replicated to a remote spool.</p></div>
<div class="sect3">
<h4 id="_kvsp_pub_kvsp_sub">kvsp-pub/kvsp-sub</h4>
<div class="paragraph"><p>The utilities <tt>kvsp-pub</tt> and <tt>kvsp-sub</tt> exist to publish a source spool to a remote spool.
The publisher listens on the specified TCP port. The subscriber connects to it. If more
than one subscriber connects, each receives a copy of the data. When no subscribers are
connected, the publisher drops data. However, when the <tt>-s</tt> flag is given to both
<tt>kvsp-pub</tt> and <tt>kvsp-sub</tt>, two things change: the publisher queues data when waiting for a
subscriber instead of dropping it; and each subscriber gets a "1/n" share of the data.</p></div>
This pair of utilities communicates in JSON over ZeroMQ. The publisher listens on the
specified TCP port. The subscriber connects to it. If more than one subscriber connects,
each receives a copy of the data. When no subscribers are connected, the publisher drops
data. However, when the <tt>-s</tt> flag is given to both <tt>kvsp-pub</tt> and <tt>kvsp-sub</tt>, two things
change: the publisher queues data when waiting for a subscriber instead of dropping it;
and secondly, if more than one subscriber connects, the data gets divided among them
rather than duplicated to all of them.</p></div>
<div class="paragraph"><p>A subscriber can concentrate data (that is, "fan-in" the data) from many publishers,
simply by listing multiple ZeroMQ endpoints on the command line.</p></div>
</div>
<div class="sect3">
<h4 id="_kvsp_bpub_kvsp_bsub">kvsp-bpub/kvsp-bsub</h4>
<div class="paragraph"><p>The <tt>kvsp-bpub</tt> and <tt>kvsp-bsub</tt> utilities implement binary-over-ZeroMQ replication. Their
usage is the same as <tt>kvsp-pub</tt> and <tt>kvsp-sub</tt> except the required <tt>-b cast.cfg</tt> argument.
This file lists the binary data type for each required key. Here&#8217;s an example <tt>cast.cfg</tt>:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>i32 timestamp
str sensor_name
i8 temp</tt></pre>
</div></div>
<div class="paragraph"><p>In this example, every dictionary in the source spool is expected to have three keys
(timestamp, sensor_name and temp); any other keys get ignored. Their values are parsed
to the binary types listed (i32, str and i8). The binary buffer is transmitted
without any keys. On the receiving subscriber, the binary is unpacked using the same file.
The reason for using binary publishing is speed- it&#8217;s often an order of magnitude faster.</p></div>
<div class="tableblock">
<table rules="none"
width="90%"
frame="border"
cellspacing="0" cellpadding="4">
<caption class="title">Table 1. Data types supported in <tt>cast.cfg</tt></caption>
<col width="16%" />
<col width="83%" />
<thead>
<tr>
<th align="left" valign="top">data type </th>
<th align="left" valign="top"> meaning</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left" valign="top"><p class="table"><tt>i8</tt></p></td>
<td align="left" valign="top"><p class="table">byte (8-bit int)</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>i16</tt></p></td>
<td align="left" valign="top"><p class="table">short (16-bit int)</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>i32</tt></p></td>
<td align="left" valign="top"><p class="table">integer (32-bit int)</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>ipv4</tt></p></td>
<td align="left" valign="top"><p class="table">dotted quad IP address</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>str</tt></p></td>
<td align="left" valign="top"><p class="table">string</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><tt>d64</tt></p></td>
<td align="left" valign="top"><p class="table">double (64-bit float)</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="sect3">
<h4 id="_kvsp_tpub">kvsp-tpub</h4>
<div class="paragraph"><p>Finally there is a "plain TCP" binary publisher. It has no subscriber counterpart yet, so
you have to code your own subscriber to use it. It takes a cast.cfg of the same form as above.
It listens on the specified port, and when a subscriber connects to it, the dictionaries
in the spool are transmitted as length-prefixed binary messages. The length prefix is a
32-bit (host-order endianness) integer specifying the message length that follows. The
binary data is transmitted in host-order endianness, except IP addresses in network order.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="other_utilities">Other utilities</h3>
@@ -1159,7 +1287,7 @@ convenient to locate a spool on a ramdisk for performance.</p></div>
<div class="sect2">
<h3 id="_perl">Perl</h3>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -1182,7 +1310,7 @@ To enact non-blocking mode, use <tt>$v-&gt;{blocking}=0;</tt>, then test the ref
<div class="sect2">
<h3 id="_python">Python</h3>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -1201,7 +1329,7 @@ object returned from <tt>read()</tt> is <tt>None</tt>. If so, no data is current
<div class="sect2">
<h3 id="_java">Java</h3>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -1238,7 +1366,7 @@ so that the dynamic linker can find this shared library it at runtime).</p></div
<h3 id="_c_c">C/C++</h3>
<div class="paragraph"><p>C programs must be linked with -lkvspool.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -1267,7 +1395,7 @@ set to stderr using <tt>kv_set_dump(set,stderr)</tt>.</p></div>
<div class="paragraph"><p>To open a spool for reading, call <tt>kv_spoolreader_new</tt> which takes the spool directory and
returns an opaque handle to the spool. Then call <tt>kv_spool_read</tt> to read the spool.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -1280,7 +1408,7 @@ spool and it&#8217;s been populated into the set). A zero value means that non-
was used, but no data is currently available in the spool.</p></div>
<div class="paragraph"><p>A C program can iterate through all the key-value pairs in the result set like this:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.3
<div class="content"><!-- Generator: GNU source-highlight 3.1.6
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
@@ -1320,8 +1448,8 @@ has the spool open at the time. It takes the spool directory as its only argumen
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Version 0.8<br />
Last updated 2012-10-02 22:23:09 EDT
Version 0.9<br />
Last updated 2014-04-12 10:35:20 EDT
</div>
</div>
</body>

93
doc/kvspool.txt
View File

@@ -1,7 +1,9 @@
kvspool: a tool for data streams
================================
Troy D. Hanson <tdh@tkhanson.net>
v0.8, September 2012
v0.9, April 2014
include::twitter.txt[]
kv-spool ("key-value" spool)::
a Linux-based C library, with Perl, Python and Java bindings, to stream data
@@ -53,9 +55,9 @@ restarts, it picks up where it left off.
Space management
~~~~~~~~~~~~~~~~
In a streaming data environment, the writer and reader can run for months on end. Because
the spool backs the data to disk, space management is necessary: 'don't fill up the disk'.
When we make a spool directory, we tell kvspool what its maximum size should be:
In a streaming data environment, the writer and reader can run for months on end. A key
implementation requirement is that we don't fill up the disk. So, when we make a spool
directory, we tell kvspool what its maximum size should be:
% mkdir spool
% kvsp-init -s 1G spool
@@ -113,13 +115,18 @@ Now, on the remote computers where you wish to subscribe to the spool, run:
This type of publish-subscribe does a "fan-out". Each subscriber gets a copy of the data.
It drops data when no subscriber is connected- it's a blast to whoever is listening.
-s mode
^^^^^^^
If you use the `-s` switch, on both ends (kvsp-pub and kvsp-sub), two things change:
data remains queued in the spool until a subscriber connects (instead of being dropped if
no one is listening). Secondly, if more than one subscriber connects, the data gets
divided among them rather than sent to all of them. Generally the -s mode is preferred
if it fits your use case.
Pub-sub modes
^^^^^^^^^^^^^
Kvspool includes three kinds of pub-sub utilities. They vary in the underlying transport
and data format.
[width="50%",cols="10m,10m,50,10",grid="none",options="header"]
|===============================================================================
| publisher | subscriber | data format and transport | notes
| kvsp-pub | kvsp-sub | JSON over zeromq pub/sub | -s uses push/pull
| kvsp-bpub | kvsp-bsub | binary over zeromq pub/sub | -s uses push/pull
| kvsp-tpub | n/a | binary over plain TCP |
|===============================================================================
Concentration
^^^^^^^^^^^^^
@@ -156,9 +163,9 @@ See the link:LICENSE.txt[LICENSE.txt] file. Kvspool is free and open source.
Resources & Help
~~~~~~~~~~~~~~~~
News about software updates are posted to the author's blog: http://tkhanson.net/blog.
Contact the author directly at tdh@tkhanson.net if you have questions or other issues.
Additional software by the author is cataloged at http://tkhanson.net.
News about software updates are posted to the author's blog: http://tkhanson.net/blog.
Additional software by the author is cataloged at http://troydhanson.github.io/.
Getting kvspool
---------------
@@ -229,18 +236,64 @@ Network utilities
|===============================================================================
|command | example
|kvsp-pub | kvsp-pub -d spool tcp://192.168.1.9:1110
|kvsp-sub | kvsp-sub -d spool tcp://192.168.1.9:1110 tcp://192.168.1.10:1111
|kvsp-sub | kvsp-sub -d spool tcp://192.168.1.9:1110
|kvsp-bpub | kvsp-bpub -b cast.cfg -d spool tcp://192.168.1.9:2110
|kvsp-bsub | kvsp-bsub -b cast.cfg -d spool tcp://192.168.1.9:2110
|kvsp-tpub | kvsp-tpub -b cast.cfg -d spool -p 2110
|===============================================================================
The network utilities keep a local spool continuously replicated to a remote spool.
These require ZeroMQ and Jansson libraries on the system in order to be built.
kvsp-pub/kvsp-sub
^^^^^^^^^^^^^^^^^
The utilities `kvsp-pub` and `kvsp-sub` exist to publish a source spool to a remote spool.
The publisher listens on the specified TCP port. The subscriber connects to it. If more
than one subscriber connects, each receives a copy of the data. When no subscribers are
connected, the publisher drops data. However, when the `-s` flag is given to both
`kvsp-pub` and `kvsp-sub`, two things change: the publisher queues data when waiting for a
subscriber instead of dropping it; and each subscriber gets a "1/n" share of the data.
This pair of utilities communicates in JSON over ZeroMQ. The publisher listens on the
specified TCP port. The subscriber connects to it. If more than one subscriber connects,
each receives a copy of the data. When no subscribers are connected, the publisher drops
data. However, when the `-s` flag is given to both `kvsp-pub` and `kvsp-sub`, two things
change: the publisher queues data when waiting for a subscriber instead of dropping it;
and secondly, if more than one subscriber connects, the data gets divided among them
rather than duplicated to all of them.
A subscriber can concentrate data (that is, "fan-in" the data) from many publishers,
simply by listing multiple ZeroMQ endpoints on the command line.
kvsp-bpub/kvsp-bsub
^^^^^^^^^^^^^^^^^^^
The `kvsp-bpub` and `kvsp-bsub` utilities implement binary-over-ZeroMQ replication. Their
usage is the same as `kvsp-pub` and `kvsp-sub` except the required `-b cast.cfg` argument.
This file lists the binary data type for each required key. Here's an example `cast.cfg`:
i32 timestamp
str sensor_name
i8 temp
In this example, every dictionary in the source spool is expected to have three keys
(timestamp, sensor_name and temp); any other keys get ignored. Their values are parsed
to the binary types listed (i32, str and i8). The binary buffer is transmitted
without any keys. On the receiving subscriber, the binary is unpacked using the same file.
The reason for using binary publishing is speed- it's often an order of magnitude faster.
.Data types supported in `cast.cfg`
[width="90%",cols="10m,50",grid="none",options="header"]
|===============================================================================
|data type | meaning
| i8 | byte (8-bit int)
| i16 | short (16-bit int)
| i32 | integer (32-bit int)
| ipv4 | dotted quad IP address
| str | string
| d64 | double (64-bit float)
|===============================================================================
kvsp-tpub
^^^^^^^^^
Finally there is a "plain TCP" binary publisher. It has no subscriber counterpart yet, so
you have to code your own subscriber to use it. It takes a cast.cfg of the same form as above.
It listens on the specified port, and when a subscriber connects to it, the dictionaries
in the spool are transmitted as length-prefixed binary messages. The length prefix is a
32-bit (host-order endianness) integer specifying the message length that follows. The
binary data is transmitted in host-order endianness, except IP addresses in network order.
[[other_utilities]]
Other utilities