JHUAPL/kvspool
1
0
Fork 0
mirror of https://github.com/JHUAPL/kvspool.git synced 2026-01-09 15:37:56 -05:00
Code Issues Packages Projects Releases Wiki Activity

bcat

Browse Source
This commit is contained in:
Troy D. Hanson
2015-08-16 15:13:09 -04:00
parent 1960222e00
commit 6a5e88d6b6
3 changed files with 28 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
doc/index.html
View File

@@ -770,8 +770,6 @@ asciidoc.install(2);
<h1>kvspool: a tool for data streams</h1>
<span id="author">Troy D. Hanson</span><br />
<span id="email"><code>&lt;<a href="mailto:tdh@tkhanson.net">tdh@tkhanson.net</a>&gt;</code></span><br />
<span id="revnumber">version 0.9,</span>
<span id="revdate">April 2014</span>
<div id="toc">
<div id="toctitle">Table of Contents</div>
<noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
@@ -786,9 +784,9 @@ kv-spool ("key-value" spool)
</dt>
<dd>
<p>
a Linux-based C library, with Perl, Python and Java bindings, to stream data
between programs as key-value dictionaries. The data stream is backed to disk,
supports rewind, snapshot, network replication, and bounded disk consumption.
a Linux-based C library to stream data between programs as key-value dictionaries.
It has tools to pipe the streams around, such as by network pub/sub or tee, to
concentrate streams together, and to monitor the consumption status of streams.
</p>
</dd>
</dl></div>
@@ -798,9 +796,7 @@ kv-spool ("key-value" spool)
<h2 id="_kvspool_8217_s_niche">kvspool&#8217;s niche</h2>
<div class="sectionbody">
<div class="paragraph"><p>Kvspool is a tiny API to stream dictionaries between programs. The dictionaries have
textual keys and values. Note that what we&#8217;re calling a dictionary- what Python calls a
dictionary- is known as a hash in Perl, and is manifested in the Java API as a HashMap.
It&#8217;s a set of key-value pairs.</p></div>
text keys and values. The dictionary is a set of key-value pairs.</p></div>
<div class="paragraph"><p>To use kvspool, two programs open the same spool- which is just a directory. The writer
puts dictionaries into the spool. The reader gets dictionaries from the spool. It blocks
when it&#8217;s caught up, waiting for more data. Like this,</p></div>
@@ -1005,6 +1001,10 @@ cellspacing="0" cellpadding="4">
<td align="left" valign="top"><p class="table"><code>kvsp-concen</code></p></td>
<td align="left" valign="top"><p class="table"><code>kvsp-concen -d spool1 -d spool2 spool</code></p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table"><code>kvsp-bcat</code></p></td>
<td align="left" valign="top"><p class="table"><code>kvsp-bcat -b config -d spool</code></p></td>
</tr>
</tbody>
</table>
</div>
@@ -1026,10 +1026,15 @@ readers before running this command.</p></div>
spool directory for each reader (and use <code>kvsp-init</code> to set the capacity of each one);
then use <code>kvsp-tee</code> as the reader on the source spool. It maintains a continuous copy of
the spool to the multiple destination spools. This command needs to be left running to
maintain the tee.</p></div>
maintain the tee. The <code>-W</code> flag can be used to run tee in the more efficient raw mode,
which copies the binary input to the binary output representation in the spool file.
It is possible to filter the incoming data using <code>-k key -r regex</code> options. In this
usage the tee only passes a dictionary if it has the key and its value matches regex.</p></div>
<div class="paragraph"><p>The <code>kvsp-concen</code> utility is the opposite of <code>kvsp-tee</code>. It takes multiple source
spools and makes a single output spool from them. It is a spool concentrator. The
source spools are flagged with <code>-d spool</code> and the final argument is the output spool.</p></div>
<div class="paragraph"><p>The <code>kvsp-bcat</code> command operates like <code>kvsp-bpub</code> (see below). It writes the binary
encoded spool content to standard output.</p></div>
</div>
<div class="sect2">
<h3 id="net_utilities">Network utilities</h3>
@@ -1352,8 +1357,7 @@ has the spool open at the time. It takes the spool directory as its only argumen
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Version 0.9<br />
Last updated 2014-12-02 15:45:27 EST
Last updated 2015-08-16 15:12:26 EDT
</div>
</div>
</body>

20
doc/index.txt
View File

@@ -1,19 +1,16 @@
kvspool: a tool for data streams
================================
Troy D. Hanson <tdh@tkhanson.net>
v0.9, April 2014
kv-spool ("key-value" spool)::
a Linux-based C library, with Perl, Python and Java bindings, to stream data
between programs as key-value dictionaries. The data stream is backed to disk,
supports rewind, snapshot, network replication, and bounded disk consumption.
a Linux-based C library to stream data between programs as key-value dictionaries.
It has tools to pipe the streams around, such as by network pub/sub or tee, to
concentrate streams together, and to monitor the consumption status of streams.
kvspool's niche
---------------
Kvspool is a tiny API to stream dictionaries between programs. The dictionaries have
textual keys and values. Note that what we're calling a dictionary- what Python calls a
dictionary- is known as a hash in Perl, and is manifested in the Java API as a HashMap.
It's a set of key-value pairs.
text keys and values. The dictionary is a set of key-value pairs.
To use kvspool, two programs open the same spool- which is just a directory. The writer
puts dictionaries into the spool. The reader gets dictionaries from the spool. It blocks
@@ -176,6 +173,7 @@ Basic
|kvsp-rewind | kvsp-rewind spool
|kvsp-tee | kvsp-tee -s spool copy1 copy2
|kvsp-concen | kvsp-concen -d spool1 -d spool2 spool
|kvsp-bcat | kvsp-bcat -b config -d spool
|===============================================================================
The `kvsp-init` command is used when a spool directory is first created, to set
@@ -198,12 +196,18 @@ Use `kvsp-tee` to support multiple readers from one input spool. First make a se
spool directory for each reader (and use `kvsp-init` to set the capacity of each one);
then use `kvsp-tee` as the reader on the source spool. It maintains a continuous copy of
the spool to the multiple destination spools. This command needs to be left running to
maintain the tee.
maintain the tee. The `-W` flag can be used to run tee in the more efficient raw mode,
which copies the binary input to the binary output representation in the spool file.
It is possible to filter the incoming data using `-k key -r regex` options. In this
usage the tee only passes a dictionary if it has the key and its value matches regex.
The `kvsp-concen` utility is the opposite of `kvsp-tee`. It takes multiple source
spools and makes a single output spool from them. It is a spool concentrator. The
source spools are flagged with `-d spool` and the final argument is the output spool.
The `kvsp-bcat` command operates like `kvsp-bpub` (see below). It writes the binary
encoded spool content to standard output.
[[net_utilities]]
Network utilities
~~~~~~~~~~~~~~~~~

2
utils/kvsp-tee.c
View File

@@ -113,7 +113,7 @@ int main(int argc, char * argv[]) {
/* peek into the internal set representation
* where the set is already stored serialized */
kvset_t *_set = (kvset_t*)set;
assert(_set->img); assert(_set->len);
assert(_set->img); assert(_set->len);
kv_write_raw_frame(osp->sp, _set->img, _set->len);
} else {
kv_spool_write(osp->sp,set);